2024-08-20T22:10:05.8566071Z Current runner version: '2.319.1' 2024-08-20T22:10:05.8572758Z Runner name: 'i-08584e9df90f8594b' 2024-08-20T22:10:05.8573574Z Runner group name: 'Default' 2024-08-20T22:10:05.8574415Z Machine name: 'ip-10-0-57-226' 2024-08-20T22:10:05.8591559Z Testing runner upgrade compatibility 2024-08-20T22:10:06.0709147Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:10:06.0710972Z Actions: read 2024-08-20T22:10:06.0711455Z Attestations: read 2024-08-20T22:10:06.0711891Z Checks: read 2024-08-20T22:10:06.0712275Z Contents: read 2024-08-20T22:10:06.0712726Z Deployments: read 2024-08-20T22:10:06.0713111Z Discussions: read 2024-08-20T22:10:06.0713510Z Issues: read 2024-08-20T22:10:06.0713955Z Metadata: read 2024-08-20T22:10:06.0714325Z Packages: read 2024-08-20T22:10:06.0714722Z Pages: read 2024-08-20T22:10:06.0715172Z PullRequests: read 2024-08-20T22:10:06.0715596Z RepositoryProjects: read 2024-08-20T22:10:06.0716010Z SecurityEvents: read 2024-08-20T22:10:06.0716493Z Statuses: read 2024-08-20T22:10:06.0717066Z ##[endgroup] 2024-08-20T22:10:06.0720218Z Secret source: Actions 2024-08-20T22:10:06.0720739Z Prepare workflow directory 2024-08-20T22:10:06.1638606Z Prepare all required actions 2024-08-20T22:10:06.1804155Z Getting action download info 2024-08-20T22:10:06.3629225Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:10:06.9349674Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:10.9028280Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:11.1029706Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:11.3555273Z Getting action download info 2024-08-20T22:10:11.4575128Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:11.6698101Z Getting action download info 2024-08-20T22:10:11.7628271Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:11.9088371Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:11.9090571Z ##[group] Inputs 2024-08-20T22:10:11.9091085Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:11.9100438Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:11.9110875Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:11.9112204Z sync-tag: 2024-08-20T22:10:11.9113059Z timeout-minutes: 240 2024-08-20T22:10:11.9113397Z use-gha: 2024-08-20T22:10:11.9113697Z dashboard-tag: 2024-08-20T22:10:11.9114012Z s3-bucket: gha-artifacts 2024-08-20T22:10:11.9114366Z aws-role-to-assume: 2024-08-20T22:10:11.9114695Z ##[endgroup] 2024-08-20T22:10:11.9115586Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:11.9832415Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:11.9975512Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:11.9987566Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:11.9988086Z ##[endgroup] 2024-08-20T22:10:13.2116465Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:13.2117060Z Instance Type: g5.4xlarge 2024-08-20T22:10:13.2117784Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:13.2118325Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:19.0365987Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:19.0366535Z with: 2024-08-20T22:10:19.0367551Z github-secret: *** 2024-08-20T22:10:19.0368482Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:19.0369461Z activate-with-label: false 2024-08-20T22:10:19.0369816Z label: with-ssh 2024-08-20T22:10:19.0370131Z remove-existing-keys: true 2024-08-20T22:10:19.0370493Z fail-silently: true 2024-08-20T22:10:19.0370798Z env: 2024-08-20T22:10:19.0371072Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:19.0371419Z ##[endgroup] 2024-08-20T22:10:19.1270088Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:19.1273237Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:19.3734292Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:19.4364528Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:19.4365159Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:19.5199987Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:19.5215209Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:19.5239927Z Login using: ssh ec2-user@ec2-54-242-95-76.compute-1.amazonaws.com 2024-08-20T22:10:19.5240698Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:19.5241610Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:19.5375102Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:19.5375665Z with: 2024-08-20T22:10:19.5375948Z submodules: recursive 2024-08-20T22:10:19.5376276Z fetch-depth: 0 2024-08-20T22:10:19.5376564Z env: 2024-08-20T22:10:19.5376835Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:19.5377167Z ##[endgroup] 2024-08-20T22:10:19.5577375Z ##[group]Run retry () { 2024-08-20T22:10:19.5577764Z retry () { 2024-08-20T22:10:19.5578492Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:19.5579042Z } 2024-08-20T22:10:19.5579353Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:19.5579775Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:19.5580255Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:19.5580710Z else 2024-08-20T22:10:19.5581055Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:19.5581483Z fi 2024-08-20T22:10:19.5581823Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:19.5593907Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:19.5594422Z env: 2024-08-20T22:10:19.5594709Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:19.5595055Z NO_SUDO: 2024-08-20T22:10:19.5595343Z ##[endgroup] 2024-08-20T22:10:19.5627748Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:19.8163192Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:19.8163639Z with: 2024-08-20T22:10:19.8163968Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:19.8164416Z fetch-depth: 0 2024-08-20T22:10:19.8164736Z submodules: recursive 2024-08-20T22:10:19.8166474Z quiet-checkout: true 2024-08-20T22:10:19.8166828Z repository: pytorch/pytorch 2024-08-20T22:10:19.8167730Z token: *** 2024-08-20T22:10:19.8168031Z ssh-strict: true 2024-08-20T22:10:19.8168353Z persist-credentials: true 2024-08-20T22:10:19.8168716Z clean: true 2024-08-20T22:10:19.8169054Z sparse-checkout-cone-mode: true 2024-08-20T22:10:19.8169447Z lfs: false 2024-08-20T22:10:19.8169753Z set-safe-directory: true 2024-08-20T22:10:19.8170101Z env: 2024-08-20T22:10:19.8170379Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:19.8170712Z ##[endgroup] 2024-08-20T22:10:19.9190926Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:19.9193174Z ##[group]Getting Git version info 2024-08-20T22:10:19.9194414Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:19.9195735Z [command]/usr/bin/git version 2024-08-20T22:10:19.9196296Z git version 2.40.1 2024-08-20T22:10:19.9205003Z ##[endgroup] 2024-08-20T22:10:19.9220902Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f4bb0e6f-9b5e-4619-946b-512a771f3b37' before making global git config changes 2024-08-20T22:10:19.9222161Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:19.9226338Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:19.9278238Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:19.9282279Z ##[group]Initializing the repository 2024-08-20T22:10:19.9285344Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:19.9331523Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:19.9332406Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:19.9333225Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:19.9333788Z hint: 2024-08-20T22:10:19.9334258Z hint: git config --global init.defaultBranch 2024-08-20T22:10:19.9334727Z hint: 2024-08-20T22:10:19.9335254Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:19.9336096Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:19.9337182Z hint: 2024-08-20T22:10:19.9337566Z hint: git branch -m 2024-08-20T22:10:19.9338336Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:19.9345721Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:19.9391186Z ##[endgroup] 2024-08-20T22:10:19.9391769Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:19.9394440Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:19.9436350Z ##[endgroup] 2024-08-20T22:10:19.9436884Z ##[group]Setting up auth 2024-08-20T22:10:19.9442983Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:19.9485634Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:19.9858999Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:19.9902572Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:20.0264511Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:20.0323575Z ##[endgroup] 2024-08-20T22:10:20.0324427Z ##[group]Fetching the repository 2024-08-20T22:10:20.0330887Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:23.0756406Z remote: Enumerating objects: 1008509 2024-08-20T22:10:23.0757135Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:23.0760916Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:23.0763720Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:23.0764456Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:23.0764970Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:23.0765468Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:23.0766127Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:23.0766825Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:23.0767730Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:23.0768407Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:23.0768948Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:23.0769661Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:23.0770333Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:23.0770995Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:23.0771561Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:23.0772179Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:23.0772876Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:23.0773542Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:23.0774195Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:23.0774853Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:23.0775352Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:23.0775851Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:23.0776426Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:23.0777085Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:23.0777735Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:23.0778305Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:23.0778867Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:23.0779531Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:23.0780179Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:23.0781122Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:23.0781832Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:23.0782485Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:23.0783090Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:23.0783934Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:23.0784593Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:23.0785455Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:23.0786025Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:23.0786515Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:23.0787004Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:23.0787493Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:23.0787970Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:23.0788458Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:23.0788958Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:23.0789445Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:23.0789927Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:23.0790421Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:23.0790912Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:23.0791401Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:23.0791893Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:23.0792395Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:23.0792977Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:23.0793466Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:23.0793948Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:23.0794499Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:23.0794994Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:23.0795487Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:23.0795965Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:23.0796454Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:23.0797031Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:23.0797525Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:23.0798020Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:23.0798612Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:23.0799290Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:23.0799837Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:23.0800363Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:23.0800868Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:23.0801374Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:23.0801886Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:23.0802394Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:23.0802899Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:23.0803405Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:23.0803915Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:23.0804418Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:23.0804916Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:23.0805426Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:23.0805927Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:23.0806430Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:23.0806929Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:23.0807433Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:23.0807954Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:23.0808579Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:23.0809083Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:23.0809588Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:23.0810092Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:23.0810594Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:23.0811094Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:23.0811597Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:23.0812240Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:23.0812738Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:23.0813239Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:23.0813744Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:23.0814246Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:23.0814747Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:23.0815250Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:23.0815749Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:23.0816256Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:23.0816760Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:23.0817254Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:23.0817757Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:23.0818269Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:23.0818772Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:23.0819265Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:23.0819802Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:23.1014500Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:23.1333640Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:23.1646050Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:23.2132397Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:23.2979243Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:23.3469964Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:23.4094032Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:23.4420736Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:23.4812209Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:23.5017377Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:23.5272325Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:23.5507968Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:23.5682659Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:23.5854977Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:23.5972953Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:23.6061137Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:23.6146745Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:23.6206324Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:23.6248669Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:23.6298863Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:23.6332438Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:23.6350813Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:23.6355695Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:23.6363021Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:23.6374660Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:23.6382139Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:23.6396302Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:23.6400459Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:23.6422479Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:23.6437049Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:23.6448268Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:23.6462369Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:23.6475286Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:23.6491992Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:23.6497666Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:23.6505311Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:23.6516896Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:23.6528641Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:23.6598894Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:23.6599619Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:23.6600247Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:23.6600769Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:23.6601288Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:23.6601806Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:23.6602317Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:23.6617290Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:23.6618002Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:23.6621769Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:23.6627297Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:23.6633670Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:23.6637760Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:23.6687282Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:23.6687994Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:23.6688712Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:23.6689250Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:23.6689778Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:23.6690389Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:23.6690911Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:23.6691421Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:23.6691936Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:23.6692451Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:23.6692963Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:23.6693468Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:23.6693979Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:23.6694492Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:23.6695004Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:23.6695522Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:23.6698157Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:23.6707905Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:23.6718036Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:23.6723684Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:23.6729861Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:23.6733338Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:23.6735303Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:23.6738923Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:23.6745032Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:23.6750639Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:23.6754208Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:23.6756848Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:23.6759835Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:23.6763346Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:23.6766075Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:23.6769907Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:23.6773375Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:23.6776173Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:23.6778642Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:23.6781129Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:23.6783907Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:23.6787792Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:23.6790956Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:23.6793908Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:23.6796533Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:23.6799314Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:23.6799866Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:23.6801746Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:23.6804174Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:23.6806626Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:23.6809262Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:23.6811701Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:23.6814367Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:23.6814997Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:23.6815566Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:46.2712006Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:11:10.3131225Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:10.3170627Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:10.3175737Z ##[endgroup] 2024-08-20T22:11:10.3179387Z ##[group]Determining the checkout info 2024-08-20T22:11:10.3180064Z ##[endgroup] 2024-08-20T22:11:10.3180596Z ##[group]Checking out the ref 2024-08-20T22:11:10.3181541Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:12.0486750Z ##[endgroup] 2024-08-20T22:11:12.0490599Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:12.0492421Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:12.0554583Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:12.0594600Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:12.0635660Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:12.0673357Z ##[endgroup] 2024-08-20T22:11:12.0673879Z ##[group]Fetching submodules 2024-08-20T22:11:12.0677163Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:12.1051118Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:12.1412242Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:12.1413780Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:12.1415580Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:12.1419669Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:12.1423707Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:12.1426245Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:12.1429968Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:12.1433959Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:12.1437412Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:12.1441050Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:12.1444595Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:12.1448314Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:12.1452088Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:12.1456174Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:12.1459963Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:12.1464570Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:12.1469619Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:12.1475351Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:12.1479555Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:12.1484125Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:12.1488606Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:12.1493194Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:12.1497704Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:12.1502396Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:12.1507363Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:12.1512417Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:12.1517147Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:12.1522062Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:12.1527168Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:12.1532347Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:12.1537418Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:12.1542937Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:12.1548102Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:12.1556936Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:12.1592058Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:12.4512366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:12.6245826Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:12.8011279Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:13.0258805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:14.9534340Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:26.3447464Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:26.7527639Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:27.1469948Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:27.7410510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:28.9786691Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:31.1059923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:37.0690554Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:38.4775623Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:40.2638958Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:41.6057341Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:42.0619152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:42.3997962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:43.5820212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:43.9766380Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:44.2332706Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:45.7262114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:46.4320292Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:47.1095475Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:53.1239762Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:55.1893232Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:59.4827879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:59.7218299Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:08.6185560Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:08.7859313Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:08.9854886Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:09.8878624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:10.1790218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:10.8157150Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:11.2509453Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:11.2660256Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:11.2778896Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:11.3092063Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:11.3552136Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:12.5342678Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:12.5625180Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:12.6155150Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:12.7284068Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:12.7694270Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:13.3924033Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:13.6800646Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:13.7752428Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:13.7775052Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:13.7778389Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:13.7781793Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:13.7785597Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:13.7789298Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:13.7820767Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:14.7391729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:15.3500865Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:17.4504030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:18.5162556Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:18.8931576Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:19.0037850Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:19.5118944Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:19.5825515Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:19.5976028Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:19.7490641Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:19.7950233Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:19.8406124Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:19.8735167Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:19.9272066Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:19.9428745Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:19.9448336Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:19.9479902Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:33.0210503Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:33.0428915Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:33.1428253Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:33.1450102Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:33.1453118Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:33.1456592Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:33.1488304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:33.6781289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:34.9287149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:36.0600279Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:36.0620550Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:36.0623880Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:36.0627788Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:36.0631468Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:36.0635143Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:36.0638994Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:36.0642734Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:36.0647123Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:36.0679922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:36.8981648Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:37.2698651Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:38.4532214Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:38.7215853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:39.2359095Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:40.2877518Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:47.7040827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:48.2308085Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:48.2539986Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:48.2972264Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:48.3143540Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:48.3163269Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:48.3198250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:48.6858952Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:48.7086440Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:48.7571558Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:48.8792748Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:48.8992723Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:48.9449034Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:49.0112074Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:49.0575396Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:49.0878505Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:49.2141462Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:49.7250822Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:49.7291968Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:49.7295246Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:49.7324231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:50.1743880Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:51.1523849Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:51.1939309Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:51.2846973Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:51.2870788Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:51.2874232Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:51.2877376Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:51.2880885Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:51.2884653Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:51.2888429Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:51.2892194Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:51.2895456Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:51.2926434Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:51.7209403Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:52.7674351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:53.0941581Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:59.0883149Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:59.3562118Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:59.5564969Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:59.8500629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:13:06.6223982Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:13:06.6692118Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:13:06.6882998Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:13:06.8111057Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:13:06.8267301Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:13:06.8454654Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:13:06.8659564Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:13:06.8678415Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:06.8681779Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:06.8712454Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:08.7530385Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:10.0405920Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:10.0944360Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:10.7686914Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:10.7829622Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:11.0985233Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:11.1010449Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:11.1013544Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:11.1045567Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:11.5847416Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:12.6296438Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:12.7112895Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:12.7234891Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:12.7390840Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:12.7837004Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:12.8183235Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:12.8673643Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:12.9015035Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:12.9035772Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:12.9039037Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:12.9042554Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:12.9045970Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:12.9077989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:14.0046515Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:14.2300260Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:15.4070337Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:16.4006841Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:16.4205237Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:16.4994496Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:16.5339560Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:16.5358408Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:16.5392962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:16.7446988Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:16.7497867Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:16.7857655Z Entering 'android/libs/fbjni' 2024-08-20T22:13:16.7910801Z Entering 'third_party/FP16' 2024-08-20T22:13:16.7962996Z Entering 'third_party/FXdiv' 2024-08-20T22:13:16.8015542Z Entering 'third_party/NNPACK' 2024-08-20T22:13:16.8072541Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:16.8125976Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:16.8196004Z Entering 'third_party/benchmark' 2024-08-20T22:13:16.8247773Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:16.8299915Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:16.8357173Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:16.8411029Z Entering 'third_party/cutlass' 2024-08-20T22:13:16.8471541Z Entering 'third_party/eigen' 2024-08-20T22:13:16.8525070Z Entering 'third_party/fbgemm' 2024-08-20T22:13:16.8576304Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:16.8626749Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:16.8677219Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:16.8733759Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:16.8784440Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:16.8836547Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:16.8891747Z Entering 'third_party/fmt' 2024-08-20T22:13:16.8946578Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:16.8998605Z Entering 'third_party/gloo' 2024-08-20T22:13:16.9050185Z Entering 'third_party/googletest' 2024-08-20T22:13:16.9107442Z Entering 'third_party/ideep' 2024-08-20T22:13:16.9162982Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:16.9223571Z Entering 'third_party/ittapi' 2024-08-20T22:13:16.9274915Z Entering 'third_party/kineto' 2024-08-20T22:13:16.9325402Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:16.9375553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:16.9428098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:16.9479581Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:16.9530376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:16.9579894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:16.9634990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:16.9686004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:16.9737050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:16.9793763Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:16.9846985Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:16.9897483Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:16.9950398Z Entering 'third_party/mimalloc' 2024-08-20T22:13:17.0007114Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:17.0058897Z Entering 'third_party/nlohmann' 2024-08-20T22:13:17.0112381Z Entering 'third_party/onnx' 2024-08-20T22:13:17.0180991Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:17.0232509Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:17.0288595Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:17.0343884Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:17.0395890Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:17.0445687Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:17.0495374Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:17.0546183Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:17.0596169Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:17.0646657Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:17.0694852Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:17.0747193Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:17.0800485Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:17.0871915Z Entering 'third_party/pocketfft' 2024-08-20T22:13:17.0923003Z Entering 'third_party/protobuf' 2024-08-20T22:13:17.0980155Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:17.1032452Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:17.1087694Z Entering 'third_party/psimd' 2024-08-20T22:13:17.1139098Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:17.1198364Z Entering 'third_party/pybind11' 2024-08-20T22:13:17.1249693Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:17.1307623Z Entering 'third_party/sleef' 2024-08-20T22:13:17.1358351Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:17.1410809Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:17.1461306Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:17.1511396Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:17.1561032Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:17.1613407Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:17.1690067Z ##[endgroup] 2024-08-20T22:13:17.1690872Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:17.1696588Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:17.2062232Z Entering 'android/libs/fbjni' 2024-08-20T22:13:17.2131230Z Entering 'third_party/FP16' 2024-08-20T22:13:17.2199799Z Entering 'third_party/FXdiv' 2024-08-20T22:13:17.2270841Z Entering 'third_party/NNPACK' 2024-08-20T22:13:17.2343234Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:17.2409587Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:17.2496279Z Entering 'third_party/benchmark' 2024-08-20T22:13:17.2564532Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:17.2632359Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:17.2701598Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:17.2769749Z Entering 'third_party/cutlass' 2024-08-20T22:13:17.2845987Z Entering 'third_party/eigen' 2024-08-20T22:13:17.2915408Z Entering 'third_party/fbgemm' 2024-08-20T22:13:17.2982599Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:17.3047595Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:17.3118737Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:17.3190895Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:17.3255827Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:17.3325914Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:17.3398489Z Entering 'third_party/fmt' 2024-08-20T22:13:17.3466849Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:17.3535677Z Entering 'third_party/gloo' 2024-08-20T22:13:17.3603452Z Entering 'third_party/googletest' 2024-08-20T22:13:17.3671520Z Entering 'third_party/ideep' 2024-08-20T22:13:17.3736171Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:17.3810492Z Entering 'third_party/ittapi' 2024-08-20T22:13:17.3878342Z Entering 'third_party/kineto' 2024-08-20T22:13:17.3944063Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:17.4008519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:17.4083329Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:17.4151373Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:17.4219571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:17.4285560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:17.4356546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:17.4424092Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:17.4493932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:17.4561005Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:17.4631529Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:17.4702405Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:17.4772951Z Entering 'third_party/mimalloc' 2024-08-20T22:13:17.4844275Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:17.4912656Z Entering 'third_party/nlohmann' 2024-08-20T22:13:17.4982449Z Entering 'third_party/onnx' 2024-08-20T22:13:17.5063462Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:17.5129804Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:17.5205184Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:17.5273754Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:17.5338733Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:17.5404814Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:17.5470826Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:17.5542499Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:17.5608039Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:17.5678098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:17.5741199Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:17.5808251Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:17.5877317Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:17.5963182Z Entering 'third_party/pocketfft' 2024-08-20T22:13:17.6030911Z Entering 'third_party/protobuf' 2024-08-20T22:13:17.6101294Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:17.6167858Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:17.6237176Z Entering 'third_party/psimd' 2024-08-20T22:13:17.6304670Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:17.6375053Z Entering 'third_party/pybind11' 2024-08-20T22:13:17.6442775Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:17.6512404Z Entering 'third_party/sleef' 2024-08-20T22:13:17.6579790Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:17.6645135Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:17.6712876Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:17.6777899Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:17.6844255Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:17.6908371Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:17.6998746Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:17.7360892Z Entering 'android/libs/fbjni' 2024-08-20T22:13:17.7424391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:17.7446417Z Entering 'third_party/FP16' 2024-08-20T22:13:17.7508635Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:17.7529589Z Entering 'third_party/FXdiv' 2024-08-20T22:13:17.7591079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:17.7611835Z Entering 'third_party/NNPACK' 2024-08-20T22:13:17.7675355Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:17.7695794Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:17.7757313Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:17.7780016Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:17.7842047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:17.7879029Z Entering 'third_party/benchmark' 2024-08-20T22:13:17.7940253Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:17.7961559Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:17.8022944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:17.8043815Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:17.8107968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:17.8129667Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:17.8193457Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:17.8214916Z Entering 'third_party/cutlass' 2024-08-20T22:13:17.8277769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:17.8307998Z Entering 'third_party/eigen' 2024-08-20T22:13:17.8369371Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:17.8393485Z Entering 'third_party/fbgemm' 2024-08-20T22:13:17.8456564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:17.8477045Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:17.8538155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:17.8559416Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:17.8624846Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:17.8646302Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:17.8707966Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:17.8735100Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:17.8797861Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:17.8819345Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:17.8880573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:17.8905042Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:17.8970563Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:17.8995360Z Entering 'third_party/fmt' 2024-08-20T22:13:17.9057796Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:17.9080401Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:17.9142339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:17.9164589Z Entering 'third_party/gloo' 2024-08-20T22:13:17.9229031Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:17.9251000Z Entering 'third_party/googletest' 2024-08-20T22:13:17.9314232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:17.9336059Z Entering 'third_party/ideep' 2024-08-20T22:13:17.9399055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:17.9418394Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:17.9481453Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:17.9511339Z Entering 'third_party/ittapi' 2024-08-20T22:13:17.9574010Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:17.9596117Z Entering 'third_party/kineto' 2024-08-20T22:13:17.9658609Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:17.9679845Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:17.9742240Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:17.9761094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:17.9826274Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:17.9847514Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:17.9910979Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:17.9933382Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:17.9995824Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:18.0015990Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:18.0079688Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:18.0098171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:18.0163099Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:18.0188624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:18.0250535Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:18.0271128Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:18.0333426Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:18.0353909Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:18.0417256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:18.0438507Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:18.0500640Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:18.0523802Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:18.0585586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:18.0605408Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:18.0667323Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:18.0690207Z Entering 'third_party/mimalloc' 2024-08-20T22:13:18.0752982Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:18.0776859Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:18.0839507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:18.0861664Z Entering 'third_party/nlohmann' 2024-08-20T22:13:18.0924544Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:18.0947579Z Entering 'third_party/onnx' 2024-08-20T22:13:18.1011343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:18.1048416Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:18.1110450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:18.1132017Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:18.1197985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:18.1225358Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:18.1290055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:18.1311877Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:18.1377662Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:18.1397675Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:18.1458585Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:18.1478851Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:18.1539606Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:18.1561045Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:18.1622082Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:18.1644910Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:18.1706406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:18.1726900Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:18.1788539Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:18.1808141Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:18.1870301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:18.1888101Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:18.1950281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:18.1973719Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:18.2035555Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:18.2064673Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:18.2128338Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:18.2172360Z Entering 'third_party/pocketfft' 2024-08-20T22:13:18.2234035Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:18.2256588Z Entering 'third_party/protobuf' 2024-08-20T22:13:18.2319633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:18.2342278Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:18.2403630Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:18.2423675Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:18.2485208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:18.2508828Z Entering 'third_party/psimd' 2024-08-20T22:13:18.2574557Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:18.2595244Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:18.2656813Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:18.2678745Z Entering 'third_party/pybind11' 2024-08-20T22:13:18.2741124Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:18.2761928Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:18.2823991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:18.2845565Z Entering 'third_party/sleef' 2024-08-20T22:13:18.2907406Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:18.2929712Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:18.2992503Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:18.3012355Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:18.3074491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:18.3094691Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:18.3155038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:18.3175436Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:18.3236059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:18.3256362Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:18.3317577Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:18.3341528Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:18.3402464Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:18.4408774Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:18.4774933Z Entering 'android/libs/fbjni' 2024-08-20T22:13:18.4826133Z Entering 'third_party/FP16' 2024-08-20T22:13:18.4879071Z Entering 'third_party/FXdiv' 2024-08-20T22:13:18.4930952Z Entering 'third_party/NNPACK' 2024-08-20T22:13:18.4983674Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:18.5035941Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:18.5104046Z Entering 'third_party/benchmark' 2024-08-20T22:13:18.5156082Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:18.5209235Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:18.5261671Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:18.5314246Z Entering 'third_party/cutlass' 2024-08-20T22:13:18.5372897Z Entering 'third_party/eigen' 2024-08-20T22:13:18.5426335Z Entering 'third_party/fbgemm' 2024-08-20T22:13:18.5476913Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:18.5527949Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:18.5579093Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:18.5639526Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:18.5691456Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:18.5743710Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:18.5799961Z Entering 'third_party/fmt' 2024-08-20T22:13:18.5850167Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:18.5902414Z Entering 'third_party/gloo' 2024-08-20T22:13:18.5953297Z Entering 'third_party/googletest' 2024-08-20T22:13:18.6007321Z Entering 'third_party/ideep' 2024-08-20T22:13:18.6056162Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:18.6116034Z Entering 'third_party/ittapi' 2024-08-20T22:13:18.6166300Z Entering 'third_party/kineto' 2024-08-20T22:13:18.6220607Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:18.6273347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:18.6325015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:18.6374708Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:18.6424288Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:18.6474522Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:18.6528376Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:18.6577962Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:18.6627738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:18.6678251Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:18.6731014Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:18.6787358Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:18.6840429Z Entering 'third_party/mimalloc' 2024-08-20T22:13:18.6891853Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:18.6943212Z Entering 'third_party/nlohmann' 2024-08-20T22:13:18.6995234Z Entering 'third_party/onnx' 2024-08-20T22:13:18.7061359Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:18.7112571Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:18.7167316Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:18.7221391Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:18.7277352Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:18.7327358Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:18.7382868Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:18.7434063Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:18.7483849Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:18.7533401Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:18.7581464Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:18.7634661Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:18.7689162Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:18.7761705Z Entering 'third_party/pocketfft' 2024-08-20T22:13:18.7816808Z Entering 'third_party/protobuf' 2024-08-20T22:13:18.7871413Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:18.7922785Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:18.7977082Z Entering 'third_party/psimd' 2024-08-20T22:13:18.8029687Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:18.8082298Z Entering 'third_party/pybind11' 2024-08-20T22:13:18.8136304Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:18.8189788Z Entering 'third_party/sleef' 2024-08-20T22:13:18.8242546Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:18.8294461Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:18.8344215Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:18.8393165Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:18.8442714Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:18.8491454Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:18.8569390Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:18.8928327Z Entering 'android/libs/fbjni' 2024-08-20T22:13:18.8980227Z Entering 'third_party/FP16' 2024-08-20T22:13:18.9032778Z Entering 'third_party/FXdiv' 2024-08-20T22:13:18.9086416Z Entering 'third_party/NNPACK' 2024-08-20T22:13:18.9140845Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:18.9196973Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:18.9266258Z Entering 'third_party/benchmark' 2024-08-20T22:13:18.9318094Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:18.9370249Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:18.9423631Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:18.9479008Z Entering 'third_party/cutlass' 2024-08-20T22:13:18.9538567Z Entering 'third_party/eigen' 2024-08-20T22:13:18.9593432Z Entering 'third_party/fbgemm' 2024-08-20T22:13:18.9644381Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:18.9694911Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:18.9745630Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:18.9802349Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:18.9852238Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:18.9906062Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:18.9961831Z Entering 'third_party/fmt' 2024-08-20T22:13:19.0015819Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:19.0068734Z Entering 'third_party/gloo' 2024-08-20T22:13:19.0121257Z Entering 'third_party/googletest' 2024-08-20T22:13:19.0175682Z Entering 'third_party/ideep' 2024-08-20T22:13:19.0228189Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:19.0287922Z Entering 'third_party/ittapi' 2024-08-20T22:13:19.0339985Z Entering 'third_party/kineto' 2024-08-20T22:13:19.0416887Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:19.0471480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:19.0524753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:19.0578271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:19.0629829Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:19.0680828Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:19.0736859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:19.0789293Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:19.0840250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:19.0893034Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:19.0946575Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:19.0997453Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:19.1051221Z Entering 'third_party/mimalloc' 2024-08-20T22:13:19.1104694Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:19.1156586Z Entering 'third_party/nlohmann' 2024-08-20T22:13:19.1212452Z Entering 'third_party/onnx' 2024-08-20T22:13:19.1279703Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:19.1330939Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:19.1392030Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:19.1446535Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:19.1496858Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:19.1547262Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:19.1597372Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:19.1648124Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:19.1704108Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:19.1753252Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:19.1804302Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:19.1858221Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:19.1913348Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:19.1984940Z Entering 'third_party/pocketfft' 2024-08-20T22:13:19.2037451Z Entering 'third_party/protobuf' 2024-08-20T22:13:19.2092875Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:19.2143400Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:19.2198412Z Entering 'third_party/psimd' 2024-08-20T22:13:19.2251528Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:19.2307676Z Entering 'third_party/pybind11' 2024-08-20T22:13:19.2359281Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:19.2411919Z Entering 'third_party/sleef' 2024-08-20T22:13:19.2465849Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:19.2517247Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:19.2567469Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:19.2619185Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:19.2670484Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:19.2719885Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:19.2795286Z ##[endgroup] 2024-08-20T22:13:19.2843944Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:19.2880427Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:19.3101596Z Prepare all required actions 2024-08-20T22:13:19.3102061Z Getting action download info 2024-08-20T22:13:19.4423423Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:19.4423866Z env: 2024-08-20T22:13:19.4424157Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.4424501Z ##[endgroup] 2024-08-20T22:13:19.4483247Z ##[group]Run set -euo pipefail 2024-08-20T22:13:19.4483690Z set -euo pipefail 2024-08-20T22:13:19.4484095Z function get_ec2_metadata() { 2024-08-20T22:13:19.4484644Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:19.4485540Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:19.4486318Z  category=$1 2024-08-20T22:13:19.4486879Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:19.4487532Z  runner_name_str=i-08584e9df90f8594b 2024-08-20T22:13:19.4488037Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:19.4488558Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:19.4489146Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:19.4489845Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:19.4490451Z  else 2024-08-20T22:13:19.4490962Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:19.4491548Z  fi 2024-08-20T22:13:19.4491848Z } 2024-08-20T22:13:19.4492216Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:19.4492812Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:19.4493474Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:19.4494060Z echo "system info $(uname -a)" 2024-08-20T22:13:19.4506541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:19.4507055Z env: 2024-08-20T22:13:19.4507353Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.4507718Z ##[endgroup] 2024-08-20T22:13:19.4613298Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:19.4678618Z instance-id: i-08584e9df90f8594b 2024-08-20T22:13:19.4738515Z instance-type: g5.4xlarge 2024-08-20T22:13:19.4752137Z system info Linux ip-10-0-57-226.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:19.4776317Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:19.4777208Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:19.4786367Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:19.4787041Z env: 2024-08-20T22:13:19.4787320Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.4787664Z ##[endgroup] 2024-08-20T22:13:19.4864490Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:19.4865088Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:19.4865624Z  echo "Docker daemon is running..."; 2024-08-20T22:13:19.4866073Z else 2024-08-20T22:13:19.4866563Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:19.4867499Z fi 2024-08-20T22:13:19.4875567Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:19.4876054Z env: 2024-08-20T22:13:19.4876333Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.4876677Z ##[endgroup] 2024-08-20T22:13:19.4965254Z Docker daemon is running... 2024-08-20T22:13:19.5014919Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:19.5015481Z with: 2024-08-20T22:13:19.5015766Z shell: bash 2024-08-20T22:13:19.5016063Z timeout_minutes: 5 2024-08-20T22:13:19.5016388Z max_attempts: 3 2024-08-20T22:13:19.5016707Z retry_wait_seconds: 30 2024-08-20T22:13:19.5019784Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:19.5022730Z polling_interval_seconds: 1 2024-08-20T22:13:19.5023203Z warning_on_retry: true 2024-08-20T22:13:19.5023553Z continue_on_error: false 2024-08-20T22:13:19.5023891Z env: 2024-08-20T22:13:19.5024170Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:19.5024525Z AWS_RETRY_MODE: standard 2024-08-20T22:13:19.5024872Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:19.5025215Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:19.5025599Z ##[endgroup] 2024-08-20T22:13:20.6754508Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:20.6755346Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:20.6756208Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:20.6756722Z 2024-08-20T22:13:20.6756839Z Login Succeeded 2024-08-20T22:13:21.5582073Z Command completed after 1 attempt(s). 2024-08-20T22:13:21.5650501Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:21.5651212Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:21.5651858Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:21.5662264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:21.5662767Z env: 2024-08-20T22:13:21.5663204Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:21.5663563Z ##[endgroup] 2024-08-20T22:13:21.5757846Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:21.5758608Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:21.5759195Z # shellcheck disable=SC2046 2024-08-20T22:13:21.5759655Z docker stop $(docker ps -q) || true 2024-08-20T22:13:21.5760129Z # Prune all of the docker images 2024-08-20T22:13:21.5760576Z docker system prune -af 2024-08-20T22:13:21.5769346Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:21.5769853Z env: 2024-08-20T22:13:21.5770128Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:21.5770473Z ##[endgroup] 2024-08-20T22:13:21.6085213Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:21.6086144Z See 'docker stop --help'. 2024-08-20T22:13:21.6086882Z 2024-08-20T22:13:21.6087182Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:21.6087669Z 2024-08-20T22:13:21.6087878Z Stop one or more running containers 2024-08-20T22:13:21.6507879Z Total reclaimed space: 0B 2024-08-20T22:13:21.6553902Z ##[group]Run set +e 2024-08-20T22:13:21.6554266Z set +e 2024-08-20T22:13:21.6554578Z set -x 2024-08-20T22:13:21.6554881Z  2024-08-20T22:13:21.6555206Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:21.6555989Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:21.6557056Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:21.6557806Z # one is returned at random 2024-08-20T22:13:21.6558354Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:21.6558859Z  2024-08-20T22:13:21.6559196Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:21.6559820Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:21.6560772Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:21.6561327Z  2024-08-20T22:13:21.6561656Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:21.6562209Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:21.6562715Z  exit 1 2024-08-20T22:13:21.6563026Z  fi 2024-08-20T22:13:21.6563314Z fi 2024-08-20T22:13:21.6563592Z  2024-08-20T22:13:21.6563952Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:21.6564461Z  # Clean up any old records first 2024-08-20T22:13:21.6564972Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:21.6565431Z fi 2024-08-20T22:13:21.6565706Z  2024-08-20T22:13:21.6566140Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:21.6566689Z cat /etc/hosts 2024-08-20T22:13:21.6575882Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:21.6576383Z env: 2024-08-20T22:13:21.6576685Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:21.6577046Z ##[endgroup] 2024-08-20T22:13:21.6607040Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:21.6613795Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:21.6614272Z ++ tail -n1 2024-08-20T22:13:21.7048076Z + RESOLVED_IP=18.160.10.76 2024-08-20T22:13:21.7048612Z + '[' -z 18.160.10.76 ']' 2024-08-20T22:13:21.7049063Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:21.7067548Z + echo '18.160.10.76 download.pytorch.org' 2024-08-20T22:13:21.7068356Z + sudo tee -a /etc/hosts 2024-08-20T22:13:21.9839451Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:21.9860975Z + cat /etc/hosts 2024-08-20T22:13:21.9872185Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:21.9884890Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:21.9885623Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:22.0039574Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:22.0040173Z with: 2024-08-20T22:13:22.0041266Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0042480Z docker-build-dir: .ci/docker 2024-08-20T22:13:22.0042858Z working-directory: . 2024-08-20T22:13:22.0043324Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.0043843Z force-push: false 2024-08-20T22:13:22.0044147Z env: 2024-08-20T22:13:22.0044422Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.0044755Z ##[endgroup] 2024-08-20T22:13:22.0063411Z ##[group]Run set -ex 2024-08-20T22:13:22.0063782Z set -ex 2024-08-20T22:13:22.0064086Z  2024-08-20T22:13:22.0064803Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:22.0065769Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:22.0066559Z # job could then download the pre-built image as usual 2024-08-20T22:13:22.0067720Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:22.0068395Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0069037Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0069659Z  2024-08-20T22:13:22.0070178Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:22.0070810Z  exit 0 2024-08-20T22:13:22.0071118Z else 2024-08-20T22:13:22.0071491Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0071953Z fi 2024-08-20T22:13:22.0072249Z  2024-08-20T22:13:22.0072729Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:22.0073579Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:22.0074371Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:22.0075085Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:22.0075815Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0076519Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0077079Z else 2024-08-20T22:13:22.0077523Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:22.0078194Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0079093Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0079845Z fi 2024-08-20T22:13:22.0089727Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.0090223Z env: 2024-08-20T22:13:22.0090503Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.0090853Z REPO_NAME: pytorch 2024-08-20T22:13:22.0091987Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0093197Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:22.0093694Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.0094202Z ##[endgroup] 2024-08-20T22:13:22.0125742Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:22.0126155Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:22.0126591Z + echo skip=false 2024-08-20T22:13:22.0128690Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:22.0133190Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0134639Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:22.0159666Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0160535Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0162291Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0196068Z ##[group]Run set +e 2024-08-20T22:13:22.0196557Z set +e 2024-08-20T22:13:22.0196936Z set -x 2024-08-20T22:13:22.0197370Z  2024-08-20T22:13:22.0197992Z login() { 2024-08-20T22:13:22.0198743Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:22.0199571Z } 2024-08-20T22:13:22.0199964Z  2024-08-20T22:13:22.0200331Z retry () { 2024-08-20T22:13:22.0200886Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:22.0201390Z } 2024-08-20T22:13:22.0201754Z  2024-08-20T22:13:22.0202230Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:22.0202737Z  2024-08-20T22:13:22.0203267Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:22.0204209Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:22.0204801Z  exit 0 2024-08-20T22:13:22.0205244Z fi 2024-08-20T22:13:22.0205603Z  2024-08-20T22:13:22.0206190Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:22.0207176Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:22.0208007Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:22.0208729Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:22.0209485Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:22.0210003Z else 2024-08-20T22:13:22.0210569Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:22.0211449Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:22.0212048Z fi 2024-08-20T22:13:22.0212421Z  2024-08-20T22:13:22.0212894Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:22.0213505Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0214076Z  2024-08-20T22:13:22.0214822Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:22.0215673Z  exit 0 2024-08-20T22:13:22.0216094Z fi 2024-08-20T22:13:22.0216518Z  2024-08-20T22:13:22.0217025Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:22.0218114Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:22.0219124Z  exit 1 2024-08-20T22:13:22.0219569Z fi 2024-08-20T22:13:22.0219934Z  2024-08-20T22:13:22.0220574Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:22.0221644Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:22.0222658Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:22.0223804Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:22.0225018Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:22.0225970Z fi 2024-08-20T22:13:22.0226378Z  2024-08-20T22:13:22.0226797Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:22.0235429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.0236019Z env: 2024-08-20T22:13:22.0236478Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.0236917Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:22.0237505Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:22.0238989Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0240387Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.0241077Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.0241822Z ##[endgroup] 2024-08-20T22:13:22.0272427Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.0273221Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.0274547Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:22.0276766Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.5683551Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:22.5684571Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:22.5685613Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:22.5686238Z 2024-08-20T22:13:22.5686411Z Login Succeeded 2024-08-20T22:13:22.5708516Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.7905897Z { 2024-08-20T22:13:22.7906373Z "schemaVersion": 2, 2024-08-20T22:13:22.7907572Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:22.7908228Z "config": { 2024-08-20T22:13:22.7909143Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:22.7910132Z "size": 48711, 2024-08-20T22:13:22.7910783Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:22.7911483Z }, 2024-08-20T22:13:22.7912050Z "layers": [ 2024-08-20T22:13:22.7912468Z { 2024-08-20T22:13:22.7913338Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7913998Z "size": 28580681, 2024-08-20T22:13:22.7914994Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:22.7915856Z }, 2024-08-20T22:13:22.7916439Z { 2024-08-20T22:13:22.7917022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7917928Z "size": 7943451, 2024-08-20T22:13:22.7919278Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:22.7920109Z }, 2024-08-20T22:13:22.7920772Z { 2024-08-20T22:13:22.7921412Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7922306Z "size": 55728572, 2024-08-20T22:13:22.7922973Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:22.7923656Z }, 2024-08-20T22:13:22.7924431Z { 2024-08-20T22:13:22.7925876Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7927002Z "size": 186, 2024-08-20T22:13:22.7928237Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:22.7929032Z }, 2024-08-20T22:13:22.7929448Z { 2024-08-20T22:13:22.7930174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7930837Z "size": 6886, 2024-08-20T22:13:22.7931568Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:22.7932773Z }, 2024-08-20T22:13:22.7933326Z { 2024-08-20T22:13:22.7934152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7935020Z "size": 1329001756, 2024-08-20T22:13:22.7936222Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:22.7937215Z }, 2024-08-20T22:13:22.7937631Z { 2024-08-20T22:13:22.7938431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7939280Z "size": 62501, 2024-08-20T22:13:22.7939897Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:22.7940621Z }, 2024-08-20T22:13:22.7941007Z { 2024-08-20T22:13:22.7941538Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7942131Z "size": 1684, 2024-08-20T22:13:22.7942786Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:22.7943884Z }, 2024-08-20T22:13:22.7944192Z { 2024-08-20T22:13:22.7944787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7945452Z "size": 1523, 2024-08-20T22:13:22.7946097Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:22.7946826Z }, 2024-08-20T22:13:22.7947198Z { 2024-08-20T22:13:22.7947725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7948442Z "size": 2528295403, 2024-08-20T22:13:22.7949058Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:22.7949907Z }, 2024-08-20T22:13:22.7950375Z { 2024-08-20T22:13:22.7950922Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7951521Z "size": 86016, 2024-08-20T22:13:22.7952235Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:22.7952954Z }, 2024-08-20T22:13:22.7953264Z { 2024-08-20T22:13:22.7953877Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7954522Z "size": 1822, 2024-08-20T22:13:22.7955223Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:22.7955894Z }, 2024-08-20T22:13:22.7956242Z { 2024-08-20T22:13:22.7956868Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7957586Z "size": 246761563, 2024-08-20T22:13:22.7958216Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:22.7959003Z }, 2024-08-20T22:13:22.7959342Z { 2024-08-20T22:13:22.7959826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7960542Z "size": 544, 2024-08-20T22:13:22.7961160Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:22.7961875Z }, 2024-08-20T22:13:22.7962241Z { 2024-08-20T22:13:22.7962768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7963431Z "size": 1258, 2024-08-20T22:13:22.7964093Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:22.7964777Z }, 2024-08-20T22:13:22.7965138Z { 2024-08-20T22:13:22.7965706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7966366Z "size": 484, 2024-08-20T22:13:22.7967342Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:22.7968120Z }, 2024-08-20T22:13:22.7968487Z { 2024-08-20T22:13:22.7968977Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7969729Z "size": 91711766, 2024-08-20T22:13:22.7970408Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:22.7971122Z }, 2024-08-20T22:13:22.7971538Z { 2024-08-20T22:13:22.7972093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7972736Z "size": 3390, 2024-08-20T22:13:22.7973610Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:22.7974272Z }, 2024-08-20T22:13:22.7974614Z { 2024-08-20T22:13:22.7975219Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7975823Z "size": 1909, 2024-08-20T22:13:22.7976443Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:22.7977212Z }, 2024-08-20T22:13:22.7977553Z { 2024-08-20T22:13:22.7978060Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7978793Z "size": 701, 2024-08-20T22:13:22.7979417Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:22.7980200Z }, 2024-08-20T22:13:22.7980504Z { 2024-08-20T22:13:22.7981022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7983159Z "size": 2864254879, 2024-08-20T22:13:22.7983847Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:22.7984542Z }, 2024-08-20T22:13:22.7984933Z { 2024-08-20T22:13:22.7985453Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7986067Z "size": 380, 2024-08-20T22:13:22.7986768Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:22.7987470Z }, 2024-08-20T22:13:22.7987831Z { 2024-08-20T22:13:22.7988366Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7989034Z "size": 12900, 2024-08-20T22:13:22.7989677Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:22.7990423Z }, 2024-08-20T22:13:22.7990751Z { 2024-08-20T22:13:22.7991270Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7991971Z "size": 803, 2024-08-20T22:13:22.7992613Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:22.7993290Z }, 2024-08-20T22:13:22.7993682Z { 2024-08-20T22:13:22.7994224Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7994830Z "size": 106, 2024-08-20T22:13:22.7995524Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:22.7996221Z }, 2024-08-20T22:13:22.7996566Z { 2024-08-20T22:13:22.7997129Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.7997778Z "size": 503, 2024-08-20T22:13:22.7998405Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:22.7999181Z }, 2024-08-20T22:13:22.7999500Z { 2024-08-20T22:13:22.8000098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8000829Z "size": 121477267, 2024-08-20T22:13:22.8001432Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:22.8002154Z }, 2024-08-20T22:13:22.8002551Z { 2024-08-20T22:13:22.8003069Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8003696Z "size": 109, 2024-08-20T22:13:22.8004396Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:22.8005098Z }, 2024-08-20T22:13:22.8005520Z { 2024-08-20T22:13:22.8006001Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8006668Z "size": 490, 2024-08-20T22:13:22.8007354Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:22.8008042Z }, 2024-08-20T22:13:22.8008370Z { 2024-08-20T22:13:22.8008944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8009595Z "size": 297, 2024-08-20T22:13:22.8010260Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:22.8011005Z }, 2024-08-20T22:13:22.8011483Z { 2024-08-20T22:13:22.8012028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8012694Z "size": 103, 2024-08-20T22:13:22.8013389Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:22.8014096Z }, 2024-08-20T22:13:22.8014487Z { 2024-08-20T22:13:22.8014987Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8015625Z "size": 1473, 2024-08-20T22:13:22.8016295Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:22.8016974Z }, 2024-08-20T22:13:22.8017310Z { 2024-08-20T22:13:22.8017901Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8018549Z "size": 594729516, 2024-08-20T22:13:22.8019149Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:22.8020013Z }, 2024-08-20T22:13:22.8020384Z { 2024-08-20T22:13:22.8020944Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8021596Z "size": 164, 2024-08-20T22:13:22.8022214Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:22.8022927Z }, 2024-08-20T22:13:22.8023422Z { 2024-08-20T22:13:22.8023908Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8024644Z "size": 564, 2024-08-20T22:13:22.8025334Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:22.8025985Z }, 2024-08-20T22:13:22.8026349Z { 2024-08-20T22:13:22.8026958Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8027624Z "size": 35874203, 2024-08-20T22:13:22.8028245Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:22.8029030Z }, 2024-08-20T22:13:22.8029397Z { 2024-08-20T22:13:22.8030025Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8030626Z "size": 104, 2024-08-20T22:13:22.8031268Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:22.8032005Z }, 2024-08-20T22:13:22.8032313Z { 2024-08-20T22:13:22.8032856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8033565Z "size": 425, 2024-08-20T22:13:22.8034225Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:22.8034902Z }, 2024-08-20T22:13:22.8035289Z { 2024-08-20T22:13:22.8035828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8036476Z "size": 20262183, 2024-08-20T22:13:22.8037130Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:22.8037862Z }, 2024-08-20T22:13:22.8038207Z { 2024-08-20T22:13:22.8038732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8039389Z "size": 438, 2024-08-20T22:13:22.8040008Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:22.8040796Z }, 2024-08-20T22:13:22.8041098Z { 2024-08-20T22:13:22.8041611Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8042318Z "size": 701, 2024-08-20T22:13:22.8042949Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:22.8043612Z }, 2024-08-20T22:13:22.8044026Z { 2024-08-20T22:13:22.8044542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8045205Z "size": 141, 2024-08-20T22:13:22.8045849Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:22.8046561Z }, 2024-08-20T22:13:22.8046954Z { 2024-08-20T22:13:22.8047514Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8048150Z "size": 135, 2024-08-20T22:13:22.8048893Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:22.8049651Z }, 2024-08-20T22:13:22.8049977Z { 2024-08-20T22:13:22.8050495Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8051188Z "size": 4838068375, 2024-08-20T22:13:22.8051855Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:22.8052518Z }, 2024-08-20T22:13:22.8052916Z { 2024-08-20T22:13:22.8053489Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8054178Z "size": 194, 2024-08-20T22:13:22.8054767Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:22.8055482Z }, 2024-08-20T22:13:22.8055868Z { 2024-08-20T22:13:22.8056372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8057101Z "size": 564, 2024-08-20T22:13:22.8057785Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:22.8058515Z }, 2024-08-20T22:13:22.8058819Z { 2024-08-20T22:13:22.8059411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8060056Z "size": 43162921, 2024-08-20T22:13:22.8060726Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:22.8061453Z }, 2024-08-20T22:13:22.8061803Z { 2024-08-20T22:13:22.8062327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8062993Z "size": 106, 2024-08-20T22:13:22.8063747Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:22.8064451Z }, 2024-08-20T22:13:22.8064871Z { 2024-08-20T22:13:22.8065359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8066035Z "size": 1380, 2024-08-20T22:13:22.8066719Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:22.8067667Z }, 2024-08-20T22:13:22.8068000Z { 2024-08-20T22:13:22.8068597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8069243Z "size": 701, 2024-08-20T22:13:22.8070066Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:22.8070739Z }, 2024-08-20T22:13:22.8071102Z { 2024-08-20T22:13:22.8071694Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8072347Z "size": 139, 2024-08-20T22:13:22.8072996Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:22.8073736Z }, 2024-08-20T22:13:22.8074118Z { 2024-08-20T22:13:22.8074626Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8075330Z "size": 118, 2024-08-20T22:13:22.8075976Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:22.8076673Z }, 2024-08-20T22:13:22.8077030Z { 2024-08-20T22:13:22.8077577Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8078224Z "size": 4324672728, 2024-08-20T22:13:22.8078928Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:22.8079615Z }, 2024-08-20T22:13:22.8079959Z { 2024-08-20T22:13:22.8080523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8081182Z "size": 174, 2024-08-20T22:13:22.8081802Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:22.8082566Z }, 2024-08-20T22:13:22.8082913Z { 2024-08-20T22:13:22.8083399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8084121Z "size": 907, 2024-08-20T22:13:22.8084764Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:22.8085483Z }, 2024-08-20T22:13:22.8086005Z { 2024-08-20T22:13:22.8086539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8087203Z "size": 701, 2024-08-20T22:13:22.8087879Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:22.8088581Z }, 2024-08-20T22:13:22.8088945Z { 2024-08-20T22:13:22.8089519Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8090117Z "size": 134, 2024-08-20T22:13:22.8090772Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:22.8091534Z }, 2024-08-20T22:13:22.8091899Z { 2024-08-20T22:13:22.8092388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8093098Z "size": 32, 2024-08-20T22:13:22.8093750Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:22.8094675Z }, 2024-08-20T22:13:22.8094988Z { 2024-08-20T22:13:22.8095544Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8096236Z "size": 158, 2024-08-20T22:13:22.8096822Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:22.8097534Z }, 2024-08-20T22:13:22.8097921Z { 2024-08-20T22:13:22.8098465Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8099066Z "size": 1841, 2024-08-20T22:13:22.8099745Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:22.8100477Z }, 2024-08-20T22:13:22.8100821Z { 2024-08-20T22:13:22.8101399Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8102058Z "size": 7529776, 2024-08-20T22:13:22.8102696Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:22.8103530Z }, 2024-08-20T22:13:22.8103889Z { 2024-08-20T22:13:22.8104415Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8105136Z "size": 163, 2024-08-20T22:13:22.8105726Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:22.8106432Z }, 2024-08-20T22:13:22.8106847Z { 2024-08-20T22:13:22.8107362Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8107992Z "size": 7943, 2024-08-20T22:13:22.8108689Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:22.8109399Z }, 2024-08-20T22:13:22.8109764Z { 2024-08-20T22:13:22.8110292Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8110924Z "size": 8068, 2024-08-20T22:13:22.8111579Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:22.8112291Z }, 2024-08-20T22:13:22.8112632Z { 2024-08-20T22:13:22.8113176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8113875Z "size": 302, 2024-08-20T22:13:22.8114461Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:22.8115210Z }, 2024-08-20T22:13:22.8115598Z { 2024-08-20T22:13:22.8116134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8116728Z "size": 7629638, 2024-08-20T22:13:22.8117401Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:22.8118102Z }, 2024-08-20T22:13:22.8118401Z { 2024-08-20T22:13:22.8118961Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8119616Z "size": 108, 2024-08-20T22:13:22.8120297Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:22.8120964Z }, 2024-08-20T22:13:22.8121379Z { 2024-08-20T22:13:22.8121966Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8122627Z "size": 54145661, 2024-08-20T22:13:22.8123341Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:22.8124104Z }, 2024-08-20T22:13:22.8124462Z { 2024-08-20T22:13:22.8124938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8125654Z "size": 473, 2024-08-20T22:13:22.8126303Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:22.8127008Z }, 2024-08-20T22:13:22.8127379Z { 2024-08-20T22:13:22.8127905Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8128587Z "size": 1374858766, 2024-08-20T22:13:22.8129299Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:22.8130008Z }, 2024-08-20T22:13:22.8130344Z { 2024-08-20T22:13:22.8130924Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8131648Z "size": 106, 2024-08-20T22:13:22.8132273Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:22.8133030Z }, 2024-08-20T22:13:22.8133363Z { 2024-08-20T22:13:22.8133866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8134557Z "size": 559, 2024-08-20T22:13:22.8135208Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:22.8135916Z }, 2024-08-20T22:13:22.8136274Z { 2024-08-20T22:13:22.8136798Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8137429Z "size": 46248488, 2024-08-20T22:13:22.8138122Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:22.8138808Z }, 2024-08-20T22:13:22.8139143Z { 2024-08-20T22:13:22.8139702Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8140355Z "size": 111, 2024-08-20T22:13:22.8140939Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:22.8141718Z }, 2024-08-20T22:13:22.8142065Z { 2024-08-20T22:13:22.8142543Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8143337Z "size": 32, 2024-08-20T22:13:22.8143955Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:22.8144698Z }, 2024-08-20T22:13:22.8145017Z { 2024-08-20T22:13:22.8145528Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8146209Z "size": 32, 2024-08-20T22:13:22.8146847Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:22.8147507Z }, 2024-08-20T22:13:22.8147888Z { 2024-08-20T22:13:22.8148477Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8149065Z "size": 32, 2024-08-20T22:13:22.8149780Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:22.8150480Z }, 2024-08-20T22:13:22.8150808Z { 2024-08-20T22:13:22.8151353Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:22.8151983Z "size": 32, 2024-08-20T22:13:22.8152598Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:22.8153353Z } 2024-08-20T22:13:22.8153647Z ] 2024-08-20T22:13:22.8153974Z } 2024-08-20T22:13:22.8349709Z + exit 0 2024-08-20T22:13:22.8456068Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:22.8456578Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:22.8457138Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:22.8466473Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.8467218Z env: 2024-08-20T22:13:22.8467545Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.8468723Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.8469930Z ##[endgroup] 2024-08-20T22:13:22.8501418Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.8555505Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:22.8556104Z with: 2024-08-20T22:13:22.8557323Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.8558820Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.8559380Z env: 2024-08-20T22:13:22.8559667Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.8560032Z ##[endgroup] 2024-08-20T22:13:22.8577634Z ##[group]Run set -x 2024-08-20T22:13:22.8578141Z set -x 2024-08-20T22:13:22.8578443Z set +e 2024-08-20T22:13:22.8578736Z  2024-08-20T22:13:22.8579005Z login() { 2024-08-20T22:13:22.8579659Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:22.8580365Z } 2024-08-20T22:13:22.8580641Z  2024-08-20T22:13:22.8580941Z retry () { 2024-08-20T22:13:22.8581326Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:22.8581763Z } 2024-08-20T22:13:22.8582037Z  2024-08-20T22:13:22.8582351Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:22.8582753Z  2024-08-20T22:13:22.8583108Z set -e 2024-08-20T22:13:22.8583577Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:22.8584271Z # only pull docker image if it's not available locally 2024-08-20T22:13:22.8585025Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:22.8585726Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:22.8586168Z fi 2024-08-20T22:13:22.8595147Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:22.8595650Z env: 2024-08-20T22:13:22.8595936Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:22.8597089Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:22.8598398Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.8598911Z ##[endgroup] 2024-08-20T22:13:22.8628038Z + set +e 2024-08-20T22:13:22.8628575Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.8629217Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:22.8632087Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:22.8633280Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:23.3949298Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:23.3950125Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:23.3950962Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:23.3951496Z 2024-08-20T22:13:23.3951633Z Login Succeeded 2024-08-20T22:13:23.3972500Z + set -e 2024-08-20T22:13:23.3973831Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:23.4133845Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:23.4135980Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:23.6504170Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:23.6505511Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:23.6506027Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:23.6506884Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:23.6507425Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:23.6507916Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:23.6508409Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:23.6508896Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:23.6509375Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:23.6509859Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:23.6510348Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:23.6510850Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:23.6511344Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:23.6511843Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:23.6512577Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:23.6513063Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:23.6513552Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:23.6514050Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:23.6514599Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:23.6515102Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:23.6515619Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:23.6516115Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:23.6516625Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:23.6517119Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:23.6517610Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:23.6518094Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:23.6518583Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:23.6519075Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:23.6519548Z ec1e7978c1fe: Waiting 2024-08-20T22:13:23.6520051Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:23.6520549Z 75b2abb16c68: Waiting 2024-08-20T22:13:23.6520978Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:23.6521384Z 6efb96fc87ea: Waiting 2024-08-20T22:13:23.6521794Z d527cbbb87e3: Waiting 2024-08-20T22:13:23.6522211Z ca426ffc9d2b: Waiting 2024-08-20T22:13:23.6522619Z b119c4fc6c58: Waiting 2024-08-20T22:13:23.6523032Z 34c47d63c5de: Waiting 2024-08-20T22:13:23.6523446Z b57676e46aee: Waiting 2024-08-20T22:13:23.6523892Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:23.6524348Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:23.6524789Z 4373aa743caf: Waiting 2024-08-20T22:13:23.6525183Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:23.6525597Z 7ed32bc8e469: Waiting 2024-08-20T22:13:23.6525935Z 33b639226eee: Waiting 2024-08-20T22:13:23.6526247Z d3547bd43146: Waiting 2024-08-20T22:13:23.6526557Z 1fef4bd19792: Waiting 2024-08-20T22:13:23.6526945Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:23.6527346Z a8c1e85b5e14: Waiting 2024-08-20T22:13:23.6527675Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:23.6528098Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:23.6528519Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:23.6528877Z 0c1227890755: Waiting 2024-08-20T22:13:23.6529206Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:23.6529579Z 019620471c25: Pulling fs layer 2024-08-20T22:13:23.6529954Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:23.6530330Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:23.6530706Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:23.6531081Z 2df849f72ba4: Waiting 2024-08-20T22:13:23.6531463Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:23.6531830Z 829ed7d91e38: Waiting 2024-08-20T22:13:23.6532161Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:23.6532585Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:23.6532944Z f1d6874dee9c: Waiting 2024-08-20T22:13:23.6533261Z 6658e558ebe4: Waiting 2024-08-20T22:13:23.6533586Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:23.6533998Z 33444a5a7e01: Waiting 2024-08-20T22:13:23.6534329Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:23.6534705Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:23.6535131Z 1811ff4d3556: Waiting 2024-08-20T22:13:23.6535449Z 385d0421d67c: Waiting 2024-08-20T22:13:23.6535780Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:23.6536190Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:23.6536546Z d8d1234baab3: Waiting 2024-08-20T22:13:23.6536861Z afc429e87b59: Waiting 2024-08-20T22:13:23.6537177Z 0479fbac3361: Waiting 2024-08-20T22:13:23.6537655Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:23.6538016Z 7754ee068b33: Waiting 2024-08-20T22:13:23.6538351Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:23.6538724Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:23.6539080Z 5a4eb19d7219: Waiting 2024-08-20T22:13:23.6539401Z 829c0e19f4b7: Waiting 2024-08-20T22:13:23.6539713Z 0f085d4fa279: Waiting 2024-08-20T22:13:23.6540032Z 7cbfce999d76: Waiting 2024-08-20T22:13:23.6540363Z f957c1624481: Pulling fs layer 2024-08-20T22:13:23.6540732Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:23.6541092Z 302c8c177944: Waiting 2024-08-20T22:13:23.6541403Z f957c1624481: Waiting 2024-08-20T22:13:23.6541825Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:23.6542261Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:23.6542629Z 02970347dff6: Waiting 2024-08-20T22:13:23.6542959Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:23.6543585Z c42b3602a1fb: Waiting 2024-08-20T22:13:23.6543898Z 916c5e858016: Waiting 2024-08-20T22:13:23.6544209Z 11fad972ef6c: Waiting 2024-08-20T22:13:23.6544545Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:23.6544926Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:23.6545288Z 4f4fb700ef54: Waiting 2024-08-20T22:13:23.6545600Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:23.6545908Z 560cd6122a38: Waiting 2024-08-20T22:13:23.6546232Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:23.6546613Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:23.6546988Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:23.6547362Z b10de02255c0: Waiting 2024-08-20T22:13:23.6547678Z 37513e9b43d7: Waiting 2024-08-20T22:13:23.6548009Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:23.6548381Z ee1273153675: Pulling fs layer 2024-08-20T22:13:23.6548765Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:23.6549141Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:23.6549504Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:23.6549822Z e56eeff2db93: Waiting 2024-08-20T22:13:23.6550151Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:23.6550510Z eb09cbf3e416: Waiting 2024-08-20T22:13:23.6550824Z e22979f9e0ee: Waiting 2024-08-20T22:13:23.6551135Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:23.6551469Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:23.6551822Z 768c31dca016: Waiting 2024-08-20T22:13:23.6552135Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:23.6552461Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:23.6552844Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:23.6553196Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:23.6553516Z d79fd7d52698: Waiting 2024-08-20T22:13:23.6553829Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:23.6554168Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:23.6554545Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:23.6554910Z da6a838a66c7: Waiting 2024-08-20T22:13:23.6555230Z ee1273153675: Waiting 2024-08-20T22:13:23.6555546Z 9752d6fbbcab: Waiting 2024-08-20T22:13:23.7858866Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:23.7859727Z 224fe954d725: Download complete 2024-08-20T22:13:23.8613758Z d527cbbb87e3: Download complete 2024-08-20T22:13:23.9316146Z b57676e46aee: Download complete 2024-08-20T22:13:23.9939989Z 7a2c55901189: Download complete 2024-08-20T22:13:24.0677401Z a41a8d1c11c8: Download complete 2024-08-20T22:13:24.1405021Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:24.1405594Z 0c1227890755: Download complete 2024-08-20T22:13:24.2270938Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:24.2271424Z d8d1234baab3: Download complete 2024-08-20T22:13:24.2877520Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:24.2878041Z 75722010b82e: Download complete 2024-08-20T22:13:24.3587793Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:24.3588422Z ec1e7978c1fe: Download complete 2024-08-20T22:13:24.4346800Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:24.4347405Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:25.1648412Z 7a2c55901189: Pull complete 2024-08-20T22:13:25.4728646Z 224fe954d725: Pull complete 2024-08-20T22:13:26.4079084Z 75722010b82e: Pull complete 2024-08-20T22:13:26.4271520Z d527cbbb87e3: Pull complete 2024-08-20T22:13:26.4455532Z b57676e46aee: Pull complete 2024-08-20T22:13:26.9803366Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:26.9804374Z ca426ffc9d2b: Download complete 2024-08-20T22:13:27.2906778Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:27.2907303Z 4373aa743caf: Download complete 2024-08-20T22:13:27.3571583Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:27.3572054Z 75b2abb16c68: Download complete 2024-08-20T22:13:27.4354054Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:27.4354563Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:28.3959722Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:28.3960194Z 0bf61114d37b: Download complete 2024-08-20T22:13:28.4709490Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:28.4709944Z 2df849f72ba4: Download complete 2024-08-20T22:13:28.5598104Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:28.5598563Z 34c47d63c5de: Download complete 2024-08-20T22:13:28.6318308Z 1fef4bd19792: Download complete 2024-08-20T22:13:37.2690608Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:37.2691246Z a8c1e85b5e14: Download complete 2024-08-20T22:13:37.3454938Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:37.3456773Z b119c4fc6c58: Download complete 2024-08-20T22:13:37.4170955Z 33b639226eee: Download complete 2024-08-20T22:13:37.4827282Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:37.4827745Z d3547bd43146: Download complete 2024-08-20T22:13:37.5576176Z 5645520256fa: Download complete 2024-08-20T22:13:37.6511039Z 829ed7d91e38: Verifying Checksum 2024-08-20T22:13:37.6511515Z 829ed7d91e38: Download complete 2024-08-20T22:13:38.9230571Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:38.9231270Z 385d0421d67c: Download complete 2024-08-20T22:13:38.9853789Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:38.9854286Z 7cbfce999d76: Download complete 2024-08-20T22:13:39.0738904Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:39.0739467Z 6658e558ebe4: Download complete 2024-08-20T22:13:39.1566794Z 33444a5a7e01: Verifying Checksum 2024-08-20T22:13:39.1567555Z 33444a5a7e01: Download complete 2024-08-20T22:13:39.2360035Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:39.2360609Z 7c47122f4e09: Download complete 2024-08-20T22:13:39.3023766Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:39.3024317Z 1811ff4d3556: Download complete 2024-08-20T22:13:45.3006947Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:45.3007517Z 7754ee068b33: Download complete 2024-08-20T22:13:45.3747300Z 0479fbac3361: Verifying Checksum 2024-08-20T22:13:45.3747965Z 0479fbac3361: Download complete 2024-08-20T22:13:45.4437284Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:45.4437854Z f1d6874dee9c: Download complete 2024-08-20T22:13:45.8457967Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:45.8458914Z afc429e87b59: Download complete 2024-08-20T22:13:45.9296928Z 019620471c25: Verifying Checksum 2024-08-20T22:13:45.9297493Z 019620471c25: Download complete 2024-08-20T22:13:46.0167502Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:46.0168050Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:46.2754706Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:46.2755380Z 829c0e19f4b7: Download complete 2024-08-20T22:13:46.3439113Z 560cd6122a38: Download complete 2024-08-20T22:13:46.4196571Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:46.4197221Z 5a4eb19d7219: Download complete 2024-08-20T22:13:46.4959239Z da6a838a66c7: Verifying Checksum 2024-08-20T22:13:46.4960136Z da6a838a66c7: Download complete 2024-08-20T22:13:49.5631512Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:49.5632475Z 7ed32bc8e469: Download complete 2024-08-20T22:13:49.6540636Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:49.6541279Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:49.7336639Z 0f085d4fa279: Download complete 2024-08-20T22:13:50.2084133Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:50.2084755Z 4a0412373305: Download complete 2024-08-20T22:13:50.2712810Z 302c8c177944: Download complete 2024-08-20T22:13:50.3346229Z 02970347dff6: Verifying Checksum 2024-08-20T22:13:50.3346684Z 02970347dff6: Download complete 2024-08-20T22:13:50.3855919Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:50.4009602Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:50.4010048Z 916c5e858016: Download complete 2024-08-20T22:13:50.4863355Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:50.4863825Z c42b3602a1fb: Download complete 2024-08-20T22:13:50.5425825Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:50.6878402Z 0c1227890755: Pull complete 2024-08-20T22:13:50.8741704Z d8d1234baab3: Pull complete 2024-08-20T22:13:57.3149445Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:57.3149949Z 6efb96fc87ea: Download complete 2024-08-20T22:13:57.4025222Z f957c1624481: Verifying Checksum 2024-08-20T22:13:57.4025947Z f957c1624481: Download complete 2024-08-20T22:13:57.4892290Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:57.4892873Z 768c31dca016: Download complete 2024-08-20T22:13:57.5815574Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:57.5816055Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:57.5919922Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:13:57.5920395Z 4f4fb700ef54: Download complete 2024-08-20T22:13:57.6646133Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:13:57.6646870Z 78ee77eddef2: Download complete 2024-08-20T22:13:57.7359053Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:57.7359656Z 37513e9b43d7: Download complete 2024-08-20T22:13:57.8654113Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:57.8654605Z a7a5392e2bf6: Download complete 2024-08-20T22:13:57.9439364Z b10de02255c0: Download complete 2024-08-20T22:13:58.0170830Z e56eeff2db93: Verifying Checksum 2024-08-20T22:13:58.0171280Z e56eeff2db93: Download complete 2024-08-20T22:13:58.0921807Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:13:58.0922343Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:58.1665062Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:58.1665711Z eb09cbf3e416: Download complete 2024-08-20T22:13:58.2923713Z ee1273153675: Verifying Checksum 2024-08-20T22:13:58.2924181Z ee1273153675: Download complete 2024-08-20T22:13:58.3693477Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:13:58.3693944Z e22979f9e0ee: Download complete 2024-08-20T22:13:58.9618610Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:58.9619103Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:59.0438229Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:13:59.0438699Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:26.3682227Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:26.3682948Z 5d115b29b5d5: Download complete 2024-08-20T22:14:26.4400775Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:26.4401229Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:26.5257664Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:26.5258261Z d79fd7d52698: Download complete 2024-08-20T22:14:27.8689807Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:27.8690280Z 9752d6fbbcab: Download complete 2024-08-20T22:14:27.9404691Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:27.9405157Z 83808b2eff4d: Download complete 2024-08-20T22:15:03.1270319Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:15:03.1272494Z 11fad972ef6c: Download complete 2024-08-20T22:15:06.3733051Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:06.3733538Z cbb79a705353: Download complete 2024-08-20T22:15:20.3587315Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:20.5935885Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:20.8112362Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:29.1936069Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:29.4254142Z 4373aa743caf: Pull complete 2024-08-20T22:15:29.6601802Z 75b2abb16c68: Pull complete 2024-08-20T22:15:29.8758466Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:32.4804407Z 0bf61114d37b: Pull complete 2024-08-20T22:15:32.5843545Z 2df849f72ba4: Pull complete 2024-08-20T22:15:32.6855744Z 34c47d63c5de: Pull complete 2024-08-20T22:15:32.8040064Z 1fef4bd19792: Pull complete 2024-08-20T22:16:31.2217814Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:31.4381012Z b119c4fc6c58: Pull complete 2024-08-20T22:16:31.6776826Z 33b639226eee: Pull complete 2024-08-20T22:16:31.8282080Z d3547bd43146: Pull complete 2024-08-20T22:16:31.9544760Z 5645520256fa: Pull complete 2024-08-20T22:16:32.1773022Z 829ed7d91e38: Pull complete 2024-08-20T22:16:35.8931652Z 385d0421d67c: Pull complete 2024-08-20T22:16:36.0857178Z 7cbfce999d76: Pull complete 2024-08-20T22:16:36.2984263Z 6658e558ebe4: Pull complete 2024-08-20T22:16:36.5065587Z 33444a5a7e01: Pull complete 2024-08-20T22:16:36.6386513Z 7c47122f4e09: Pull complete 2024-08-20T22:16:36.8568534Z 1811ff4d3556: Pull complete 2024-08-20T22:16:47.6968675Z 7754ee068b33: Pull complete 2024-08-20T22:16:47.8885093Z 0479fbac3361: Pull complete 2024-08-20T22:16:48.0603065Z f1d6874dee9c: Pull complete 2024-08-20T22:16:48.9567627Z afc429e87b59: Pull complete 2024-08-20T22:16:49.0452261Z 019620471c25: Pull complete 2024-08-20T22:16:49.1027503Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:49.5813264Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:49.7516604Z 560cd6122a38: Pull complete 2024-08-20T22:16:50.1538493Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:50.3534557Z da6a838a66c7: Pull complete 2024-08-20T22:17:29.4072216Z cbb79a705353: Pull complete 2024-08-20T22:17:29.6395260Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:29.6758837Z 0f085d4fa279: Pull complete 2024-08-20T22:17:31.9972349Z 4a0412373305: Pull complete 2024-08-20T22:17:32.0176227Z 302c8c177944: Pull complete 2024-08-20T22:17:32.0340313Z 02970347dff6: Pull complete 2024-08-20T22:17:32.0686581Z 916c5e858016: Pull complete 2024-08-20T22:17:32.0866008Z c42b3602a1fb: Pull complete 2024-08-20T22:18:53.8140091Z 11fad972ef6c: Pull complete 2024-08-20T22:18:54.0364384Z f957c1624481: Pull complete 2024-08-20T22:18:54.2717850Z 768c31dca016: Pull complete 2024-08-20T22:18:54.6697684Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:54.8993409Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:55.1314538Z 78ee77eddef2: Pull complete 2024-08-20T22:18:55.3233115Z 37513e9b43d7: Pull complete 2024-08-20T22:18:55.7037051Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:55.9288101Z b10de02255c0: Pull complete 2024-08-20T22:18:56.1580715Z e56eeff2db93: Pull complete 2024-08-20T22:18:56.3386165Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:56.5610871Z eb09cbf3e416: Pull complete 2024-08-20T22:18:57.9152931Z ee1273153675: Pull complete 2024-08-20T22:18:58.1494117Z e22979f9e0ee: Pull complete 2024-08-20T22:19:00.3348398Z 6c1f0aabce7c: Pull complete 2024-08-20T22:19:00.5616001Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:13.9141581Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:14.1354401Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:14.3574398Z d79fd7d52698: Pull complete 2024-08-20T22:19:15.2269526Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:15.4522765Z 83808b2eff4d: Pull complete 2024-08-20T22:19:16.4299943Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:16.4758325Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:16.5041509Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:16.5143207Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:16.5144146Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:16.5155880Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:16.5156368Z env: 2024-08-20T22:19:16.5156646Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:16.5156989Z ##[endgroup] 2024-08-20T22:19:16.5375505Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:16.5376074Z with: 2024-08-20T22:19:16.5376362Z driver-version: 550.54.15 2024-08-20T22:19:16.5376707Z env: 2024-08-20T22:19:16.5376981Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:16.5377331Z ##[endgroup] 2024-08-20T22:19:16.5458857Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:16.5459409Z with: 2024-08-20T22:19:16.5459680Z timeout_minutes: 10 2024-08-20T22:19:16.5459997Z max_attempts: 3 2024-08-20T22:19:16.5492830Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:16.5526038Z retry_wait_seconds: 10 2024-08-20T22:19:16.5526402Z polling_interval_seconds: 1 2024-08-20T22:19:16.5526778Z warning_on_retry: true 2024-08-20T22:19:16.5527126Z continue_on_error: false 2024-08-20T22:19:16.5527458Z env: 2024-08-20T22:19:16.5527726Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:16.5528067Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:16.5528408Z ##[endgroup] 2024-08-20T22:19:16.6340897Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:16.6341683Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:16.6343832Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:17.1008517Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:17.1009055Z No packages marked for removal. 2024-08-20T22:19:17.1071665Z Dependencies resolved. 2024-08-20T22:19:17.1081341Z Nothing to do. 2024-08-20T22:19:17.1082038Z Complete! 2024-08-20T22:19:17.1959603Z + install_nvidia_driver_common 2024-08-20T22:19:17.1962281Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:17.1962750Z Before installing NVIDIA driver 2024-08-20T22:19:17.1963511Z + lspci 2024-08-20T22:19:17.2807138Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:17.2808116Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:17.2809386Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:17.2810396Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:17.2811484Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:17.2812550Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:17.2813367Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:17.2814208Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:17.2814789Z + lsmod 2024-08-20T22:19:17.2856300Z Module Size Used by 2024-08-20T22:19:17.2856889Z xt_conntrack 16384 1 2024-08-20T22:19:17.2857357Z nft_chain_nat 16384 3 2024-08-20T22:19:17.2857730Z xt_MASQUERADE 20480 1 2024-08-20T22:19:17.2858184Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:17.2858673Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:17.2859255Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:17.2859883Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:17.2860355Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:17.2860787Z xfrm_user 57344 1 2024-08-20T22:19:17.2861210Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:17.2861753Z xt_addrtype 16384 2 2024-08-20T22:19:17.2862242Z nft_compat 20480 4 2024-08-20T22:19:17.2862924Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:17.2863735Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:17.2864421Z br_netfilter 36864 0 2024-08-20T22:19:17.2865217Z bridge 307200 1 br_netfilter 2024-08-20T22:19:17.2865810Z stp 16384 1 bridge 2024-08-20T22:19:17.2866333Z llc 16384 2 bridge,stp 2024-08-20T22:19:17.2866751Z overlay 167936 0 2024-08-20T22:19:17.2867410Z tls 114688 0 2024-08-20T22:19:17.2867787Z nls_ascii 16384 1 2024-08-20T22:19:17.2868168Z nls_cp437 20480 1 2024-08-20T22:19:17.2868649Z vfat 24576 1 2024-08-20T22:19:17.2869137Z fat 86016 1 vfat 2024-08-20T22:19:17.2869656Z ena 167936 0 2024-08-20T22:19:17.2870143Z ptp 36864 1 ena 2024-08-20T22:19:17.2870655Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:17.2871151Z sunrpc 692224 1 2024-08-20T22:19:17.2871625Z pps_core 24576 1 ptp 2024-08-20T22:19:17.2872023Z aesni_intel 393216 0 2024-08-20T22:19:17.2872391Z i8042 45056 0 2024-08-20T22:19:17.2872774Z serio 28672 3 i8042 2024-08-20T22:19:17.2873243Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:17.2873752Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:17.2874245Z button 24576 0 2024-08-20T22:19:17.2874617Z sch_fq_codel 20480 17 2024-08-20T22:19:17.2874989Z dm_mod 188416 0 2024-08-20T22:19:17.2875421Z configfs 57344 1 2024-08-20T22:19:17.2875942Z fuse 163840 1 2024-08-20T22:19:17.2876443Z dax 45056 1 dm_mod 2024-08-20T22:19:17.2876944Z loop 36864 0 2024-08-20T22:19:17.2877446Z dmi_sysfs 20480 0 2024-08-20T22:19:17.2877944Z crc32_pclmul 16384 0 2024-08-20T22:19:17.2878420Z crc32c_intel 24576 0 2024-08-20T22:19:17.2878887Z efivarfs 24576 1 2024-08-20T22:19:17.2879367Z + modinfo nvidia 2024-08-20T22:19:17.2880283Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:17.2881172Z alias: char-major-195-* 2024-08-20T22:19:17.2881668Z version: 550.54.15 2024-08-20T22:19:17.2882120Z supported: external 2024-08-20T22:19:17.2882542Z license: NVIDIA 2024-08-20T22:19:17.2882927Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:17.2883405Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:17.2883862Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:17.2884412Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:17.2885054Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:17.2885597Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:17.2886113Z depends: i2c-core,drm 2024-08-20T22:19:17.2886479Z retpoline: Y 2024-08-20T22:19:17.2886789Z name: nvidia 2024-08-20T22:19:17.2887588Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:17.2888498Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:17.2889247Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:17.2889847Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:17.2890294Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:17.2890738Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:17.2891202Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:17.2891640Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:17.2892089Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:17.2892625Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:17.2893191Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:17.2893673Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:17.2894117Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:17.2894555Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:17.2895019Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:17.2895697Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:17.2896286Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:17.2896852Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:17.2897468Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:17.2898066Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:17.2898687Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:17.2899285Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:17.2899785Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:17.2900320Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:17.2900872Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:17.2901371Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:17.2901852Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:17.2902349Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:17.2903023Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:17.2903484Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:17.2903992Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:17.2904523Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:17.2905010Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:17.2905511Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:17.2906011Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:17.2906508Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:17.2907022Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:17.2907510Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:17.2907925Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:17.2908404Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:17.2908887Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:17.2909358Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:17.2909842Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:17.2910328Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:17.2910791Z parm: rm_firmware_active:charp 2024-08-20T22:19:17.2911214Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:17.2911627Z ++ command -v nvidia-smi 2024-08-20T22:19:17.2912041Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:17.2912422Z + set +e 2024-08-20T22:19:17.2912962Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:19.5258642Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:19.5259257Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:19.5259989Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:19.5260369Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:19.5260751Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:19.5261930Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:19.5262658Z + set -e 2024-08-20T22:19:19.5263282Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:19.5263856Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:19.5264540Z + post_install_nvidia_driver_common 2024-08-20T22:19:19.5265313Z + sudo modprobe nvidia 2024-08-20T22:19:19.6965944Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:19.6966497Z + lspci 2024-08-20T22:19:19.6966795Z After installing NVIDIA driver 2024-08-20T22:19:19.7076633Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:19.7077659Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:19.7078614Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:19.7079553Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:19.7080621Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:19.7081413Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:19.7082394Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:19.7083209Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:19.7083796Z + lsmod 2024-08-20T22:19:19.7111816Z Module Size Used by 2024-08-20T22:19:19.7112395Z nvidia_uvm 4706304 0 2024-08-20T22:19:19.7112928Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:19:19.7113492Z drm 602112 1 nvidia 2024-08-20T22:19:19.7114074Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:19.7114682Z backlight 24576 1 drm 2024-08-20T22:19:19.7115246Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:19.7115816Z xt_conntrack 16384 1 2024-08-20T22:19:19.7116317Z nft_chain_nat 16384 3 2024-08-20T22:19:19.7116809Z xt_MASQUERADE 20480 1 2024-08-20T22:19:19.7117365Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:19.7117873Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:19.7118451Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:19.7119090Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:19.7119558Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:19.7119997Z xfrm_user 57344 1 2024-08-20T22:19:19.7120388Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:19.7120811Z xt_addrtype 16384 2 2024-08-20T22:19:19.7121190Z nft_compat 20480 4 2024-08-20T22:19:19.7121649Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:19.7122257Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:19.7122798Z br_netfilter 36864 0 2024-08-20T22:19:19.7123214Z bridge 307200 1 br_netfilter 2024-08-20T22:19:19.7123663Z stp 16384 1 bridge 2024-08-20T22:19:19.7124087Z llc 16384 2 bridge,stp 2024-08-20T22:19:19.7124509Z overlay 167936 0 2024-08-20T22:19:19.7124879Z tls 114688 0 2024-08-20T22:19:19.7125240Z nls_ascii 16384 1 2024-08-20T22:19:19.7125614Z nls_cp437 20480 1 2024-08-20T22:19:19.7125981Z vfat 24576 1 2024-08-20T22:19:19.7126350Z fat 86016 1 vfat 2024-08-20T22:19:19.7126733Z ena 167936 0 2024-08-20T22:19:19.7127120Z ptp 36864 1 ena 2024-08-20T22:19:19.7127516Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:19.7127894Z sunrpc 692224 1 2024-08-20T22:19:19.7128266Z pps_core 24576 1 ptp 2024-08-20T22:19:19.7128663Z aesni_intel 393216 0 2024-08-20T22:19:19.7129027Z i8042 45056 0 2024-08-20T22:19:19.7129400Z serio 28672 3 i8042 2024-08-20T22:19:19.7130043Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:19.7130752Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:19.7131417Z button 24576 0 2024-08-20T22:19:19.7131904Z sch_fq_codel 20480 17 2024-08-20T22:19:19.7132382Z dm_mod 188416 0 2024-08-20T22:19:19.7132857Z configfs 57344 1 2024-08-20T22:19:19.7133333Z fuse 163840 1 2024-08-20T22:19:19.7133807Z dax 45056 1 dm_mod 2024-08-20T22:19:19.7134320Z loop 36864 0 2024-08-20T22:19:19.7134792Z dmi_sysfs 20480 0 2024-08-20T22:19:19.7135268Z crc32_pclmul 16384 0 2024-08-20T22:19:19.7135632Z crc32c_intel 24576 0 2024-08-20T22:19:19.7136054Z efivarfs 24576 1 2024-08-20T22:19:19.7136538Z + modinfo nvidia 2024-08-20T22:19:19.7137420Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:19.7138127Z alias: char-major-195-* 2024-08-20T22:19:19.7138506Z version: 550.54.15 2024-08-20T22:19:19.7138990Z supported: external 2024-08-20T22:19:19.7139330Z license: NVIDIA 2024-08-20T22:19:19.7139701Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:19.7140191Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:19.7140647Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:19.7141110Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:19.7141600Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:19.7142090Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:19.7142582Z depends: i2c-core,drm 2024-08-20T22:19:19.7143091Z retpoline: Y 2024-08-20T22:19:19.7143400Z name: nvidia 2024-08-20T22:19:19.7144013Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:19.7144714Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:19.7145351Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:19.7146108Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:19.7146556Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:19.7146991Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:19.7147441Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:19.7147873Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:19.7148306Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:19.7148830Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:19.7149386Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:19.7149863Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:19.7150294Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:19.7150705Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:19.7151161Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:19.7151686Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:19.7152273Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:19.7152846Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:19.7153463Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:19.7154067Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:19.7154686Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:19.7155285Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:19.7155780Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:19.7156316Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:19.7156871Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:19.7157466Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:19.7157938Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:19.7158431Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:19.7158906Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:19.7159465Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:19.7159984Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:19.7160516Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:19.7161001Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:19.7161495Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:19.7162002Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:19.7162495Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:19.7163000Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:19.7163519Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:19.7163954Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:19.7164435Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:19.7165003Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:19.7165462Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:19.7165957Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:19.7166431Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:19.7167279Z parm: rm_firmware_active:charp 2024-08-20T22:19:19.7167702Z + set +e 2024-08-20T22:19:19.7168027Z + nvidia-smi 2024-08-20T22:19:21.2673210Z Tue Aug 20 22:19:21 2024 2024-08-20T22:19:21.2674243Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:21.2675102Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:21.2675972Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:21.2676795Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:21.2677685Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:21.2678373Z | | | MIG M. | 2024-08-20T22:19:21.2678947Z |=========================================+========================+======================| 2024-08-20T22:19:21.2769509Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:21.2770377Z | 0% 28C P0 57W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:21.2771047Z | | | N/A | 2024-08-20T22:19:21.2771831Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:21.2772416Z 2024-08-20T22:19:21.2773093Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:21.2773714Z | Processes: | 2024-08-20T22:19:21.2774425Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:21.2775116Z | ID ID Usage | 2024-08-20T22:19:21.2775680Z |=========================================================================================| 2024-08-20T22:19:21.2776325Z | No running processes found | 2024-08-20T22:19:21.2777069Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:21.8922784Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:23.4418597Z NVIDIA A10G 2024-08-20T22:19:23.8834772Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:23.8835460Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:23.8835946Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:23.8836386Z + set -e 2024-08-20T22:19:23.8836691Z INFO: Ignoring allowed status 0 2024-08-20T22:19:23.8845924Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:23.8849790Z + sudo yum install -y yum-utils 2024-08-20T22:19:24.3047781Z Last metadata expiration check: 0:09:35 ago on Tue Aug 20 22:09:49 2024. 2024-08-20T22:19:24.3276120Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:24.3580967Z Dependencies resolved. 2024-08-20T22:19:24.3707794Z Nothing to do. 2024-08-20T22:19:24.3708222Z Complete! 2024-08-20T22:19:24.4537745Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:24.4538986Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:24.4540346Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:24.7470783Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:24.8198256Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:25.2841187Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:25.3064182Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:25.3366115Z Dependencies resolved. 2024-08-20T22:19:25.3489523Z Nothing to do. 2024-08-20T22:19:25.3490472Z Complete! 2024-08-20T22:19:25.4452854Z + sudo systemctl restart docker 2024-08-20T22:20:22.0843335Z Tue Aug 20 22:20:22 2024 2024-08-20T22:20:22.0844217Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:22.0845050Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:22.0845833Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:22.0846674Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:22.0847565Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:22.0848291Z | | | MIG M. | 2024-08-20T22:20:22.0848848Z |=========================================+========================+======================| 2024-08-20T22:20:22.0960779Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:22.0961535Z | 0% 29C P0 57W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:20:22.0962176Z | | | N/A | 2024-08-20T22:20:22.0963053Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:22.0963651Z 2024-08-20T22:20:22.0964328Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:22.0964970Z | Processes: | 2024-08-20T22:20:22.0965684Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:22.0966383Z | ID ID Usage | 2024-08-20T22:20:22.0967196Z |=========================================================================================| 2024-08-20T22:20:22.0967893Z | No running processes found | 2024-08-20T22:20:22.0968661Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:22.6789680Z Command completed after 1 attempt(s). 2024-08-20T22:20:22.6873484Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:22.6874255Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:22.6874943Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:22.6875595Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:22.6889595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:22.6890096Z env: 2024-08-20T22:20:22.6890378Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:22.6890837Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:22.6891315Z ##[endgroup] 2024-08-20T22:20:22.9747515Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:23.4000708Z Collecting psutil==5.9.1 2024-08-20T22:20:25.0312251Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:25.0859885Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:25.0908068Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:25.1764742Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:25.3342228Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:25.5327744Z Prepare all required actions 2024-08-20T22:20:25.5328307Z Getting action download info 2024-08-20T22:20:25.6363366Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:25.8100507Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:25.9258873Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:25.9259353Z with: 2024-08-20T22:20:25.9259689Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:25.9260137Z s3-bucket: gha-artifacts 2024-08-20T22:20:25.9260471Z env: 2024-08-20T22:20:25.9260739Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.9261187Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.9261670Z ##[endgroup] 2024-08-20T22:20:25.9295574Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:25.9296008Z with: 2024-08-20T22:20:25.9296382Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:25.9296825Z s3-bucket: gha-artifacts 2024-08-20T22:20:25.9297191Z region: us-east-1 2024-08-20T22:20:25.9297490Z env: 2024-08-20T22:20:25.9297763Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:25.9298205Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:25.9298680Z ##[endgroup] 2024-08-20T22:20:26.4100354Z (node:52493) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:26.4101216Z 2024-08-20T22:20:26.4101528Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:26.4102297Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:26.4103186Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:26.4917672Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:26.4918781Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:34.0213509Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:34.0221085Z Artifact download has finished successfully 2024-08-20T22:20:34.0575501Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:34.0575942Z unzip -o artifacts.zip 2024-08-20T22:20:34.0585376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:34.0585879Z env: 2024-08-20T22:20:34.0586155Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:34.0586611Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:34.0587097Z ##[endgroup] 2024-08-20T22:20:34.0659025Z Archive: artifacts.zip 2024-08-20T22:20:34.0660081Z creating: dist/ 2024-08-20T22:20:36.1974614Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:36.1975610Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:36.1976426Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:36.1977375Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:36.1978385Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.1984147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.1985415Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.1986600Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.1987598Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.1988763Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.1990051Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.1991813Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.1992848Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.1994248Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.1995645Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.1997048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.1998568Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.1999773Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.2001236Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.2002796Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.2003895Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.2004926Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.2043840Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.2083999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.2085579Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.2130810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.2132685Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.2134594Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.2136547Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.2138283Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.2140123Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.2141904Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.2143850Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.2145501Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.2146825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.2148063Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.2149288Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.2150521Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.2151706Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.2152918Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.2206303Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.2266395Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.2268331Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.2269675Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.2270837Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.2272063Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.2273308Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:36.2274618Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:36.2276184Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:36.2277649Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:36.2278753Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:36.2279825Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:36.2280902Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:36.2281982Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:36.2283059Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:36.2284116Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:36.2300574Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:36.2444896Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:36.2446268Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:36.2447710Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:36.2449304Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:36.2450861Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:36.2452326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:36.2453610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:36.2454746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:36.2455893Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:36.2457025Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:36.2458142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:36.2474762Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:36.2557726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:36.2559260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.2560644Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.2562114Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.2563373Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.2564510Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.2565414Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:36.2566480Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:36.2567732Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:36.2568835Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:36.2688740Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:36.2752149Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:36.2753116Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:36.2753903Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:36.2754718Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.2761257Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.2762493Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.2763702Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.2764695Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.2765842Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.2767274Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.2768760Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.2769905Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.2770900Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.2772371Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.2773785Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.2775278Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.2776498Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.2778039Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.2779347Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.2780567Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.2781571Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.2822330Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.2860995Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.2862538Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.2909194Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.2910887Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.2912682Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.2914508Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.2915975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.2917415Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.2918901Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.2920394Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.2921821Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.2923118Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.2924392Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.2925596Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.2926808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.2927988Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.2929185Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.2984368Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.3044780Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.3046134Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.3047345Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.3048509Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.3049691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.3050960Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:36.3052310Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:36.3053887Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:36.3055386Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:36.3056568Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:36.3057674Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:36.3058778Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:36.3059890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:36.3060984Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:36.3062068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:36.3077344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:36.3142020Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:36.3143857Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.3145268Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.3146484Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.3147598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.3148483Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.3149425Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:36.3150556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:36.3151579Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:36.3152519Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:36.3202336Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:36.3203356Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:36.3204136Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:36.3205019Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:36.3211161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:36.3212389Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:36.3213696Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:36.3214774Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:36.3215969Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:36.3217357Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:36.3218916Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:36.3219979Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:36.3221181Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:36.3222615Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:36.3224176Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:36.3225759Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:36.3227040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:36.3228592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:36.3230206Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:36.3231304Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:36.3232363Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:36.3271622Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:36.3312777Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:36.3314576Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:36.3359688Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:36.3361631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:36.3363588Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:36.3365637Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:36.3367630Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:36.3369545Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:36.3371396Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:36.3373191Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:36.3374903Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:36.3376236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:36.3377527Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:36.3378803Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:36.3380093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:36.3381353Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:36.3382623Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:36.3434843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:36.3496464Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:36.3498088Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:36.3499502Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:36.3500737Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:36.3502028Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:36.3503503Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:36.3505014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:36.3506712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:36.3508097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:36.3509243Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:36.3510411Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:36.3512056Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:36.3513260Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:36.3514567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:36.3515732Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:36.3516986Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:36.3637907Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:36.3639532Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:36.3641147Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:36.3642798Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:36.3644493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:36.3646073Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:36.3647319Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:36.3648546Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:36.3649836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:36.3651085Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:36.3652314Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:36.3667750Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:36.3723349Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:36.3725053Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:36.3726533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:36.3727852Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:36.3729065Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:36.3730027Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:36.3730999Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:36.3732217Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:36.3733303Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:36.3734202Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:36.3837437Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:36.3881074Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:36.3881733Z creating: build/lib/ 2024-08-20T22:20:36.3970512Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:36.4426294Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:36.4435563Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:36.4445583Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:36.4455592Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:36.4456223Z inflating: build/lib/libclog.a 2024-08-20T22:20:36.4474756Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:36.4477463Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:36.4542331Z inflating: build/lib/libgtest.a 2024-08-20T22:20:36.4617535Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:36.4681069Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:36.4688748Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:36.4716551Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:36.4843568Z inflating: build/lib/libgloo.a 2024-08-20T22:20:36.4864413Z inflating: build/lib/libfmt.a 2024-08-20T22:20:36.4959435Z inflating: build/lib/libc10.so 2024-08-20T22:20:36.4961281Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:36.4962664Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:36.4982405Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:36.5486361Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:36.5506301Z inflating: build/lib/libgmock.a 2024-08-20T22:20:36.5506942Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:36.5507956Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:36.5897494Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:37.6100343Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:37.6674242Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:37.6731886Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:37.6732801Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:37.7997874Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:37.8253081Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:37.8759860Z inflating: build/lib/libkineto.a 2024-08-20T22:20:37.8944275Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:37.8986952Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:37.9685315Z inflating: build/lib/libonnx.a 2024-08-20T22:20:40.4230494Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:40.4235193Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:40.4239478Z inflating: build/lib/libshm.so 2024-08-20T22:20:42.4953822Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:42.4955482Z inflating: build/lib/libtorch.so 2024-08-20T22:20:42.4957609Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:43.3170904Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:43.5172104Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:43.5244735Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:43.5265864Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:43.5295349Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:43.5320650Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:43.5355539Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:43.5356016Z creating: build/bin/ 2024-08-20T22:20:43.5406817Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:43.5458998Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:43.5510143Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:43.5572373Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:43.5626070Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:43.5675493Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:43.5727837Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:43.5782301Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:43.5838238Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:43.5896222Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:43.5967169Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:43.6020256Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:43.6069932Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:43.6120332Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:43.6171383Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:43.6228438Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:43.6283609Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:43.6334020Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:43.6390302Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:43.6442506Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:43.6494104Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:43.6542862Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:43.6596579Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:43.6652717Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:43.6703715Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:43.6760199Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:43.6816610Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:43.6869741Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:43.6920316Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:43.6970770Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:43.7136149Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:43.7187543Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:43.7241087Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:43.7298787Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:43.7362051Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:43.7436885Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:43.7588867Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:43.7643041Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:43.7696404Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:43.7752330Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:43.7803076Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:43.7862418Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:43.7911451Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:43.7967886Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:43.8416449Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:43.8867371Z inflating: build/bin/protoc 2024-08-20T22:20:43.8921179Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:43.8974549Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:43.9028430Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:43.9080277Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:43.9133890Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:43.9187396Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:43.9236697Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:43.9289768Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:43.9621687Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:43.9974042Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:44.0335809Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:44.0388399Z inflating: build/bin/BackoffTest 2024-08-20T22:20:44.0442004Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:44.0497979Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:44.0551543Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:44.0565329Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:44.0620122Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:44.0624603Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:44.0627681Z inflating: build/bin/example_allreduce 2024-08-20T22:20:44.0683446Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:44.0752641Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:44.0755397Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:44.0822917Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:44.0832157Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:44.1181908Z inflating: build/bin/test_lazy 2024-08-20T22:20:44.2370377Z inflating: build/bin/test_api 2024-08-20T22:20:44.2443888Z inflating: build/bin/Dict_test 2024-08-20T22:20:44.2497030Z inflating: build/bin/Dimname_test 2024-08-20T22:20:44.2561767Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:44.2619391Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:44.2679897Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:44.2739727Z inflating: build/bin/atest 2024-08-20T22:20:44.2804189Z inflating: build/bin/basic 2024-08-20T22:20:44.2859376Z inflating: build/bin/broadcast_test 2024-08-20T22:20:44.2910808Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:44.2969609Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:44.3022428Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:44.3119805Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:44.3170266Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:44.3221691Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:44.3280486Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:44.3335849Z inflating: build/bin/half_test 2024-08-20T22:20:44.3432049Z inflating: build/bin/ivalue_test 2024-08-20T22:20:44.3482117Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:44.3536834Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:44.3591286Z inflating: build/bin/memory_format_test 2024-08-20T22:20:44.3644475Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:44.3698211Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:44.3754245Z inflating: build/bin/native_test 2024-08-20T22:20:44.3805249Z inflating: build/bin/operator_name_test 2024-08-20T22:20:44.3857009Z inflating: build/bin/operators_test 2024-08-20T22:20:44.3909192Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:44.3976549Z inflating: build/bin/pow_test 2024-08-20T22:20:44.4034118Z inflating: build/bin/quantized_test 2024-08-20T22:20:44.4084437Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:44.4137323Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:44.4194322Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:44.4254337Z inflating: build/bin/scalar_test 2024-08-20T22:20:44.4306932Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:44.4359244Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:44.4438886Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:44.4493633Z inflating: build/bin/test_parallel 2024-08-20T22:20:44.4496766Z inflating: build/bin/thread_init_test 2024-08-20T22:20:44.4552533Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:44.4613449Z inflating: build/bin/type_test 2024-08-20T22:20:44.4665184Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:44.4667047Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:44.4737281Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:44.4789117Z inflating: build/bin/weakref_test 2024-08-20T22:20:44.4840780Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:44.4892922Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:44.4953156Z inflating: build/bin/IListRef_test 2024-08-20T22:20:44.5060139Z inflating: build/bin/List_test 2024-08-20T22:20:44.5127765Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:44.5252734Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:44.5350566Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:44.5479566Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:44.5584066Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:44.5646580Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:44.5744147Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:44.5794818Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:44.5851397Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:44.5900744Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:44.6204800Z inflating: build/bin/op_registration_test 2024-08-20T22:20:44.6270610Z inflating: build/bin/inline_container_test 2024-08-20T22:20:44.6323932Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:44.6375959Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:44.6435203Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:44.6490355Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:44.6560708Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:44.6620728Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:44.6670969Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:44.6728822Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:44.6779594Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:44.6844623Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:44.6902103Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:44.6951448Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:44.7003575Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:44.7052385Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:44.7104651Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:44.7157417Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:44.7207729Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:44.7272173Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:44.7321978Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:44.7374826Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:44.7389463Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:44.7455548Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:44.7513039Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:44.7576854Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:44.7639284Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:44.8471335Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:44.9054088Z inflating: build/bin/test_jit 2024-08-20T22:20:44.9054958Z creating: .additional_ci_files/ 2024-08-20T22:20:44.9115424Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:44.9356565Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:44.9393393Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:44.9393793Z rm artifacts.zip 2024-08-20T22:20:44.9404033Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:44.9404540Z env: 2024-08-20T22:20:44.9404816Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:44.9405286Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:44.9405774Z ##[endgroup] 2024-08-20T22:20:45.0505421Z ##[group]Run df -H 2024-08-20T22:20:45.0505744Z df -H 2024-08-20T22:20:45.0514671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:45.0515180Z env: 2024-08-20T22:20:45.0515462Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.0515922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.0516412Z ##[endgroup] 2024-08-20T22:20:45.0568338Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:45.0569058Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:45.0569548Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:45.0570020Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:45.0570471Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:45.0571219Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:45.0571720Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:45.0572444Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:45.0611831Z Prepare all required actions 2024-08-20T22:20:45.0612288Z Getting action download info 2024-08-20T22:20:45.1776963Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:45.1777426Z with: 2024-08-20T22:20:45.1777682Z env: 2024-08-20T22:20:45.1777957Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.1778403Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.1778891Z ##[endgroup] 2024-08-20T22:20:45.1811796Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:45.1812238Z with: 2024-08-20T22:20:45.1812507Z name: td_results 2024-08-20T22:20:45.1812830Z s3-bucket: gha-artifacts 2024-08-20T22:20:45.1813181Z region: us-east-1 2024-08-20T22:20:45.1813485Z env: 2024-08-20T22:20:45.1813758Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.1814219Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.1814719Z ##[endgroup] 2024-08-20T22:20:45.6576891Z (node:52513) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:45.6577646Z 2024-08-20T22:20:45.6577913Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:45.6578624Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:45.6579491Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:45.7272813Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:45.7279803Z Artifact download has finished successfully 2024-08-20T22:20:45.7601964Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:45.7602459Z mkdir -p .additional_ci_files 2024-08-20T22:20:45.7603030Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:45.7611960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:45.7612478Z env: 2024-08-20T22:20:45.7612764Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.7613238Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.7613727Z ##[endgroup] 2024-08-20T22:20:45.7671902Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:45.7686922Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:45.7720866Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:45.7721375Z .github/scripts/parse_ref.py 2024-08-20T22:20:45.7729478Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:45.7729831Z env: 2024-08-20T22:20:45.7730111Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.7730568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.7731046Z ##[endgroup] 2024-08-20T22:20:45.8017469Z Prepare all required actions 2024-08-20T22:20:45.8056380Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:45.8056843Z with: 2024-08-20T22:20:45.8057408Z github-token: *** 2024-08-20T22:20:45.8057745Z env: 2024-08-20T22:20:45.8058027Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.8058488Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.8058979Z ##[endgroup] 2024-08-20T22:20:45.8076360Z ##[group]Run set -eux 2024-08-20T22:20:45.8076704Z set -eux 2024-08-20T22:20:45.8077288Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:45.8086309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:45.8086815Z env: 2024-08-20T22:20:45.8087102Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.8087558Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.8088210Z GITHUB_TOKEN: *** 2024-08-20T22:20:45.8088522Z ##[endgroup] 2024-08-20T22:20:45.8120616Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-08584e9df90f8594b 2024-08-20T22:20:47.3467428Z setting job-id=29026528326 2024-08-20T22:20:47.3468577Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.3687834Z Prepare all required actions 2024-08-20T22:20:47.3688284Z Getting action download info 2024-08-20T22:20:47.4755696Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:47.4756160Z with: 2024-08-20T22:20:47.4756622Z github-token: *** 2024-08-20T22:20:47.4765611Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:47.4775447Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.4776229Z env: 2024-08-20T22:20:47.4776508Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.4776969Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.4777443Z ##[endgroup] 2024-08-20T22:20:47.4820300Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:47.4820967Z with: 2024-08-20T22:20:47.4821251Z shell: bash 2024-08-20T22:20:47.4821557Z timeout_minutes: 10 2024-08-20T22:20:47.4821882Z max_attempts: 5 2024-08-20T22:20:47.4822205Z retry_wait_seconds: 30 2024-08-20T22:20:47.4823449Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:47.4824602Z polling_interval_seconds: 1 2024-08-20T22:20:47.4824976Z warning_on_retry: true 2024-08-20T22:20:47.4825322Z continue_on_error: false 2024-08-20T22:20:47.4825660Z env: 2024-08-20T22:20:47.4825935Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.4826379Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.4827020Z GITHUB_TOKEN: *** 2024-08-20T22:20:47.4827333Z ##[endgroup] 2024-08-20T22:20:47.5591324Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:47.7972212Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:47.9447833Z Collecting requests==2.27.1 2024-08-20T22:20:47.9785656Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:48.1714207Z Collecting pyyaml==6.0.1 2024-08-20T22:20:48.1751504Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:48.2031736Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:48.2485949Z Collecting certifi>=2017.4.17 2024-08-20T22:20:48.2522073Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:48.2577623Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:48.5550240Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:48.5589726Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:48.6392756Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:48.7579355Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:49.5379521Z Command completed after 1 attempt(s). 2024-08-20T22:20:49.5441566Z ##[group]Run set -x 2024-08-20T22:20:49.5441900Z set -x 2024-08-20T22:20:49.5442201Z  2024-08-20T22:20:49.5442780Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:49.5443460Z # in runner workspace 2024-08-20T22:20:49.5443975Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:49.5453392Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.5453897Z env: 2024-08-20T22:20:49.5454172Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.5454628Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.5455117Z ##[endgroup] 2024-08-20T22:20:49.5485789Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:49.5726809Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:49.5727357Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:49.5727823Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:49.5728220Z  2024-08-20T22:20:49.5728768Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:49.5729449Z # in runner workspace 2024-08-20T22:20:49.5730028Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:49.5730660Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:49.5731120Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:49.5740307Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:49.5749931Z  --selected-test-configs "" \ 2024-08-20T22:20:49.5750408Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:49.5750836Z  --tag "${TAG}" \ 2024-08-20T22:20:49.5751232Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:49.5751668Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:49.5752102Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:49.5760997Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.5761504Z env: 2024-08-20T22:20:49.5761778Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.5762242Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.5762925Z GITHUB_TOKEN: *** 2024-08-20T22:20:49.5763622Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:49.5764369Z PR_NUMBER: 2024-08-20T22:20:49.5764677Z TAG: ciflow/inductor/133712 2024-08-20T22:20:49.5765050Z EVENT_NAME: push 2024-08-20T22:20:49.5765366Z SCHEDULE: 2024-08-20T22:20:49.5765650Z HEAD_BRANCH: 2024-08-20T22:20:49.5765959Z ##[endgroup] 2024-08-20T22:20:49.5796372Z Workflow: inductor 2024-08-20T22:20:49.5797239Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:49.8374397Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:49.9724010Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:49.9724469Z echo "Filtered matrix:" 2024-08-20T22:20:49.9733557Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:49.9742875Z  2024-08-20T22:20:49.9743167Z echo 2024-08-20T22:20:49.9743699Z echo "Is the current job unstable? False" 2024-08-20T22:20:49.9744167Z  2024-08-20T22:20:49.9744453Z echo 2024-08-20T22:20:49.9744824Z echo "Is keep-going label set? False" 2024-08-20T22:20:49.9745277Z  2024-08-20T22:20:49.9745562Z echo 2024-08-20T22:20:49.9745894Z echo "Renabled issues? " 2024-08-20T22:20:49.9754679Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.9755176Z env: 2024-08-20T22:20:49.9755461Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.9755922Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.9756403Z ##[endgroup] 2024-08-20T22:20:49.9786582Z Filtered matrix: 2024-08-20T22:20:49.9797528Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:49.9806403Z 2024-08-20T22:20:49.9806552Z Is the current job unstable? False 2024-08-20T22:20:49.9806832Z 2024-08-20T22:20:49.9807215Z Is keep-going label set? False 2024-08-20T22:20:49.9807469Z 2024-08-20T22:20:49.9807602Z Renabled issues? 2024-08-20T22:20:49.9851825Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:49.9852547Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:49.9860737Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:49.9861230Z env: 2024-08-20T22:20:49.9861515Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:49.9862120Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:49.9862607Z JOB_TIMEOUT: 240 2024-08-20T22:20:49.9862909Z ##[endgroup] 2024-08-20T22:20:49.9949252Z ##[group]Run set -x 2024-08-20T22:20:49.9949811Z set -x 2024-08-20T22:20:49.9950207Z  2024-08-20T22:20:49.9950682Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:49.9951347Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:49.9952005Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:49.9952656Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:49.9953143Z else 2024-08-20T22:20:49.9953581Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:49.9954137Z fi 2024-08-20T22:20:49.9954493Z  2024-08-20T22:20:49.9955046Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:49.9984370Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:49.9985083Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:49.9985689Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:49.9986155Z container_name=$(docker run \ 2024-08-20T22:20:49.9986570Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:49.9986944Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:49.9987345Z  -e PR_NUMBER \ 2024-08-20T22:20:49.9987703Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:49.9988098Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:49.9988506Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:49.9988894Z  -e GITHUB_JOB \ 2024-08-20T22:20:49.9989253Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:49.9989642Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:49.9990049Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:49.9990448Z  -e JOB_ID \ 2024-08-20T22:20:49.9990785Z  -e JOB_NAME \ 2024-08-20T22:20:49.9991138Z  -e BASE_SHA \ 2024-08-20T22:20:49.9991485Z  -e BRANCH \ 2024-08-20T22:20:49.9991820Z  -e SHA1 \ 2024-08-20T22:20:49.9992162Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:49.9992570Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:49.9992954Z  -e SHARD_NUMBER \ 2024-08-20T22:20:49.9993320Z  -e TEST_CONFIG \ 2024-08-20T22:20:49.9993694Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:49.9994088Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:49.9994504Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:49.9994925Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:49.9995331Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:49.9995723Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:49.9996097Z  -e NO_TD \ 2024-08-20T22:20:49.9996434Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:49.9996811Z  -e PR_LABELS \ 2024-08-20T22:20:49.9997223Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:49.9997683Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:49.9998070Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:49.9998472Z  -e XLA_CUDA \ 2024-08-20T22:20:49.9998876Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:49.9999385Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:49.9999899Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:50.0000410Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:50.0000873Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:50.0001323Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:50.0001759Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:50.0002261Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:50.0002835Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:50.0003295Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:50.0003676Z  --ipc=host \ 2024-08-20T22:20:50.0004300Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:50.0004694Z  --tty \ 2024-08-20T22:20:50.0005002Z  --detach \ 2024-08-20T22:20:50.0005547Z  --name="${container_name}" \ 2024-08-20T22:20:50.0005969Z  --user jenkins \ 2024-08-20T22:20:50.0006452Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:50.0007005Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:50.0007434Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:50.0007780Z ) 2024-08-20T22:20:50.0008182Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:50.0009094Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:50.0010053Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:50.0010952Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:50.0020216Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:50.0020566Z env: 2024-08-20T22:20:50.0020834Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:50.0021293Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:50.0021889Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:50.0022385Z PR_NUMBER: 2024-08-20T22:20:50.0022698Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:50.0023109Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:50.0023566Z GITHUB_JOB: test 2024-08-20T22:20:50.0023874Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:50.0024232Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:50.0024581Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:50.0024910Z JOB_ID: 29026528326 2024-08-20T22:20:50.0025588Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:50.0026332Z BRANCH: 2024-08-20T22:20:50.0026673Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:50.0027197Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:50.0027668Z TEST_CONFIG: inductor_torchbench 2024-08-20T22:20:50.0028056Z SHARD_NUMBER: 1 2024-08-20T22:20:50.0028364Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:50.0028685Z REENABLED_ISSUES: 2024-08-20T22:20:50.0029016Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:50.0029404Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:50.0029755Z TEST_SHOWLOCALS: False 2024-08-20T22:20:50.0030098Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:50.0030422Z NO_TD: False 2024-08-20T22:20:50.0030720Z TD_DISTRIBUTED: False 2024-08-20T22:20:50.0031131Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:50.0031612Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:50.0031990Z SHM_SIZE: 2g 2024-08-20T22:20:50.0033070Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:50.0034235Z XLA_CUDA: 2024-08-20T22:20:50.0034707Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:50.0035308Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:50.0035738Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:50.0036141Z DASHBOARD_TAG: 2024-08-20T22:20:50.0036659Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:50.0037030Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:50.0037401Z ##[endgroup] 2024-08-20T22:20:50.0068367Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:50.0069413Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:50.0069955Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:50.0077431Z +++ nproc --ignore=2 2024-08-20T22:20:50.0106290Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:21:01.3688724Z + container_name=f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-20T22:21:01.3694269Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:21:01.3695748Z + docker exec -i f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:21:01.4632324Z + echo DOCKER_CONTAINER_ID=f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-20T22:21:01.4637001Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:21:01.4639838Z + docker exec -t f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:21:01.8605400Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:21:02.1866880Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:21:02.1870539Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:21:02.1872886Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:21:02.1876224Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:21:02.1879364Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:21:02.1884489Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:21:02.1902197Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:21:02.1915953Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:21:02.1933294Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:21:02.2316346Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:21:02.8070852Z Installing collected packages: torch 2024-08-20T22:21:12.6618739Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:12.6620530Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:12.6621444Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:12.7469082Z + export TERM=vt100 2024-08-20T22:21:12.7469479Z + TERM=vt100 2024-08-20T22:21:12.7472373Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:12.7483300Z + source .ci/pytorch/common.sh 2024-08-20T22:21:12.7487447Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:12.7498293Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:12.7500401Z +++ declare -f -t trap_add 2024-08-20T22:21:12.7505779Z ++ set -ex 2024-08-20T22:21:12.7506461Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:12.7506992Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:12.7508168Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:12.7510646Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:12.7531408Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:12.7531940Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:12.7532355Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:12.7532738Z + shift 2024-08-20T22:21:12.7535354Z + for trap_add_name in "$@" 2024-08-20T22:21:12.7539671Z +++ trap -p EXIT 2024-08-20T22:21:12.7542935Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:12.7543430Z +++ extract_trap_cmd 2024-08-20T22:21:12.7543821Z +++ printf '%s\n' '' 2024-08-20T22:21:12.7544235Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:12.7546168Z + trap -- ' 2024-08-20T22:21:12.7546599Z cleanup_workspace' EXIT 2024-08-20T22:21:12.7547084Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:13.4732508Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:13.4754112Z + echo 'Environment variables:' 2024-08-20T22:21:13.4754533Z Environment variables: 2024-08-20T22:21:13.4754867Z + env 2024-08-20T22:21:13.4764747Z INSTALLED_DB=yes 2024-08-20T22:21:13.4766144Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:13.4766724Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:13.4767477Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.4768123Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:13.4768726Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:13.4769430Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:13.4770027Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.4770611Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:13.4771110Z HOSTNAME=f46ac5459924 2024-08-20T22:21:13.4771964Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.4772844Z GITHUB_ACTION=__self 2024-08-20T22:21:13.4773322Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:13.4777980Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:13.4781850Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:13.4782364Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:13.4782860Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:13.4783289Z TEST_CONFIG=inductor_torchbench 2024-08-20T22:21:13.4783689Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:13.4784165Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:13.4784977Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.4785420Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:13.4785823Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:13.4786364Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:13.4786822Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.4787316Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:13.4787771Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:13.4788098Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:13.4788476Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:13.4788868Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.4789304Z XLA_CUDA= 2024-08-20T22:21:13.4789804Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:13.4792205Z *** 2024-08-20T22:21:13.4792508Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:13.4792904Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:13.4793270Z GITHUB_ACTIONS=true 2024-08-20T22:21:13.4793604Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:13.4794127Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:13.4794668Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:13.4795139Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.4795664Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:13.4796148Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.4796925Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.4797624Z UCC_HOME=/usr 2024-08-20T22:21:13.4797965Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:13.4798363Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:13.4798701Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:13.4799171Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:13.4799664Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.4800124Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:13.4800500Z SHARD_NUMBER=1 2024-08-20T22:21:13.4800808Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:13.4801157Z HOME=/var/lib/jenkins 2024-08-20T22:21:13.4801518Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:13.4801957Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:13.4802434Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:13.4802919Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:13.4803283Z CUDA_VERSION=12.1.1 2024-08-20T22:21:13.4803714Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:13.4804154Z NUM_TEST_SHARDS=2 2024-08-20T22:21:13.4804451Z UCX_HOME=/usr 2024-08-20T22:21:13.4804974Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:13.4806044Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.4807358Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:13.4808594Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.4809700Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:13.4810359Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:13.4810692Z DASHBOARD_TAG= 2024-08-20T22:21:13.4810998Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:13.4811455Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:13.4811999Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:13.4812985Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.4813884Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.4814286Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:13.4814661Z PR_NUMBER= 2024-08-20T22:21:13.4814945Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:13.4815283Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:13.4815732Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:13.4816197Z TERM=vt100 2024-08-20T22:21:13.4816541Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:13.4817071Z INSTALLED_VISION=yes 2024-08-20T22:21:13.4817391Z BRANCH= 2024-08-20T22:21:13.4817681Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:13.4818170Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:13.4818604Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:13.4819370Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:13.4820118Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:13.4820604Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:13.4821076Z REENABLED_ISSUES= 2024-08-20T22:21:13.4821382Z SHLVL=1 2024-08-20T22:21:13.4821650Z MAX_JOBS=14 2024-08-20T22:21:13.4821970Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:13.4822327Z NVARCH=x86_64 2024-08-20T22:21:13.4822633Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:13.4823247Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.4823790Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:13.4824297Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:13.4824969Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:13.4825545Z GITHUB_JOB=test 2024-08-20T22:21:13.4825955Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:13.4826507Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:13.4827027Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:13.4827358Z TD_DISTRIBUTED=False 2024-08-20T22:21:13.4827742Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:13.4828178Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:13.4828597Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:13.4828952Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:13.4829299Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:13.4829647Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:13.4830661Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:13.4831718Z GITHUB_BASE_REF= 2024-08-20T22:21:13.4832045Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:13.4832411Z CI=true 2024-08-20T22:21:13.4832729Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.4833143Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:13.4833514Z JOB_ID=29026528326 2024-08-20T22:21:13.4833826Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:13.4834151Z GITHUB_HEAD_REF= 2024-08-20T22:21:13.4834455Z GITHUB_ACTION_REF= 2024-08-20T22:21:13.4834895Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:13.4835349Z TEST_SHOWLOCALS=False 2024-08-20T22:21:13.4835682Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:13.4836040Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:13.4836942Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.4837734Z NO_TD=False 2024-08-20T22:21:13.4838033Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:13.4838397Z _=/usr/bin/env 2024-08-20T22:21:13.4838881Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:13.4998162Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:13.4999407Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:13.5000378Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:13.5001248Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:13.5001842Z + BUILD_DIR=build 2024-08-20T22:21:13.5002161Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:13.5002549Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:13.5002898Z + SHARD_NUMBER=1 2024-08-20T22:21:13.5003299Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:13.5003711Z + export VALGRIND=ON 2024-08-20T22:21:13.5004060Z + VALGRIND=ON 2024-08-20T22:21:13.5004610Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:13.5005095Z + [[ 0 == \1 ]] 2024-08-20T22:21:13.5005388Z + [[ False == \1 ]] 2024-08-20T22:21:13.5006059Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:13.5006579Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:13.5016482Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:13.5017209Z + [[ -n '' ]] 2024-08-20T22:21:13.5017575Z + echo 'Environment variables' 2024-08-20T22:21:13.5017970Z Environment variables 2024-08-20T22:21:13.5018289Z + env 2024-08-20T22:21:13.5025897Z INSTALLED_DB=yes 2024-08-20T22:21:13.5026445Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:13.5026980Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:13.5027581Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.5028401Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:13.5029180Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:13.5029895Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:13.5030608Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.5031385Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:13.5032062Z HOSTNAME=f46ac5459924 2024-08-20T22:21:13.5033100Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.5033900Z GITHUB_ACTION=__self 2024-08-20T22:21:13.5034252Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:13.5038605Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:13.5042743Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:13.5043266Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:13.5043624Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:13.5043973Z TEST_CONFIG=inductor_torchbench 2024-08-20T22:21:13.5044374Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:13.5044856Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:13.5045327Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:13.5045779Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:13.5046187Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:13.5046587Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:13.5047034Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.5047535Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:13.5047999Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:13.5048340Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:13.5048766Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:13.5049152Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.5049605Z XLA_CUDA= 2024-08-20T22:21:13.5050062Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:13.5050450Z *** 2024-08-20T22:21:13.5050724Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:13.5051130Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:13.5051499Z GITHUB_ACTIONS=true 2024-08-20T22:21:13.5051833Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:13.5052325Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:13.5052862Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:13.5053333Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.5053842Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:13.5054327Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.5055116Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.5055955Z UCC_HOME=/usr 2024-08-20T22:21:13.5056301Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:13.5056709Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:13.5057295Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:13.5057790Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:13.5058290Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:13.5058768Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:13.5059169Z SHARD_NUMBER=1 2024-08-20T22:21:13.5059490Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:13.5059843Z HOME=/var/lib/jenkins 2024-08-20T22:21:13.5060214Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:13.5060652Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:13.5061120Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:13.5061613Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:13.5061988Z CUDA_VERSION=12.1.1 2024-08-20T22:21:13.5062425Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:13.5063018Z NUM_TEST_SHARDS=2 2024-08-20T22:21:13.5063335Z UCX_HOME=/usr 2024-08-20T22:21:13.5063870Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:13.5064961Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.5066231Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:13.5067846Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.5068951Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:13.5069604Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:13.5069930Z DASHBOARD_TAG= 2024-08-20T22:21:13.5070238Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:13.5070703Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:13.5071243Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:13.5072228Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.5073126Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:13.5073536Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:13.5073907Z PR_NUMBER= 2024-08-20T22:21:13.5074185Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:13.5074502Z VALGRIND=ON 2024-08-20T22:21:13.5074800Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:13.5075233Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:13.5075693Z TERM=vt100 2024-08-20T22:21:13.5076038Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:13.5076459Z INSTALLED_VISION=yes 2024-08-20T22:21:13.5076774Z BRANCH= 2024-08-20T22:21:13.5077064Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:13.5077469Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:13.5077901Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:13.5078653Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:13.5079406Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:13.5079885Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:13.5080361Z REENABLED_ISSUES= 2024-08-20T22:21:13.5080650Z SHLVL=1 2024-08-20T22:21:13.5080912Z MAX_JOBS=14 2024-08-20T22:21:13.5081227Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:13.5081573Z NVARCH=x86_64 2024-08-20T22:21:13.5081867Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:13.5082330Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:13.5082860Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:13.5083340Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:13.5084004Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:13.5084575Z GITHUB_JOB=test 2024-08-20T22:21:13.5084974Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:13.5085518Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:13.5086027Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:13.5086517Z TD_DISTRIBUTED=False 2024-08-20T22:21:13.5086899Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:13.5087448Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:13.5087924Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:13.5088287Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:13.5088630Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:13.5088983Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:13.5090005Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:13.5091057Z GITHUB_BASE_REF= 2024-08-20T22:21:13.5091372Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:13.5091748Z CI=true 2024-08-20T22:21:13.5092067Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:13.5092484Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:13.5092849Z JOB_ID=29026528326 2024-08-20T22:21:13.5093166Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:13.5093496Z GITHUB_HEAD_REF= 2024-08-20T22:21:13.5093804Z GITHUB_ACTION_REF= 2024-08-20T22:21:13.5094243Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:13.5094704Z TEST_SHOWLOCALS=False 2024-08-20T22:21:13.5095045Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:13.5095395Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:13.5096298Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_801689e8-35ea-4b71-98a6-2e7807ac99d9 2024-08-20T22:21:13.5097118Z NO_TD=False 2024-08-20T22:21:13.5097451Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:13.5097809Z _=/usr/bin/env 2024-08-20T22:21:13.5098154Z + echo 'Testing pytorch' 2024-08-20T22:21:13.5098495Z Testing pytorch 2024-08-20T22:21:13.5098814Z + export LANG=C.UTF-8 2024-08-20T22:21:13.5099156Z + LANG=C.UTF-8 2024-08-20T22:21:13.5099447Z + PR_NUMBER= 2024-08-20T22:21:13.5099789Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:13.5100293Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:13.5100787Z + [[ inductor_torchbench == \s\l\o\w ]] 2024-08-20T22:21:13.5101422Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:13.5102115Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:13.5102648Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:13.5103235Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:13.5103680Z + [[ inductor_torchbench == *crossref* ]] 2024-08-20T22:21:13.5104252Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:13.5104889Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:13.5105540Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:13.5106102Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:13.5106637Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:14.6817442Z Collecting ninja==1.10.2 2024-08-20T22:21:14.6976808Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:14.7499645Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:15.3479579Z Installing collected packages: ninja 2024-08-20T22:21:15.3558315Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:15.3559767Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:15.4040827Z Successfully installed ninja-1.10.2 2024-08-20T22:21:15.4868693Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:15.4872318Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:15.4873755Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:15.4874522Z + install_tlparse 2024-08-20T22:21:15.4874927Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:15.4875641Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:15.9219982Z Collecting tlparse==0.3.25 2024-08-20T22:21:15.9388853Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:15.9915175Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:16.6489196Z Installing collected packages: tlparse 2024-08-20T22:21:16.7242996Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:16.8056927Z ++ python -m site --user-base 2024-08-20T22:21:16.8300856Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:16.8302612Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:16.8303624Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:16.8304425Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:16.8305501Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:16.8306664Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:16.8307350Z + cd test 2024-08-20T22:21:16.8307957Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:18.4647770Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:18.4648860Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:18.4649771Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:18.4650382Z + [[ inductor_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:21:18.4651078Z + [[ inductor_torchbench == *dynamo_eager* ]] 2024-08-20T22:21:18.4651746Z + [[ inductor_torchbench == *aot_eager* ]] 2024-08-20T22:21:18.4652279Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T22:21:18.4652755Z + [[ inductor_torchbench == *inductor* ]] 2024-08-20T22:21:18.4653195Z + [[ inductor_torchbench != *perf* ]] 2024-08-20T22:21:18.4653892Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:18.4654336Z + [[ inductor_torchbench == *dynamic* ]] 2024-08-20T22:21:18.4654769Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T22:21:18.4655266Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:18.4684619Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:18.4685486Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:18.4687525Z + cd test 2024-08-20T22:21:18.4688185Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:19.9308723Z PyTorch built with: 2024-08-20T22:21:19.9309507Z - GCC 9.4 2024-08-20T22:21:19.9309947Z - C++ Version: 201703 2024-08-20T22:21:19.9310853Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:19.9311942Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:19.9312610Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:19.9313124Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:19.9313605Z - NNPACK is enabled 2024-08-20T22:21:19.9313997Z - CPU capability usage: AVX2 2024-08-20T22:21:19.9314406Z - CUDA Runtime 12.1 2024-08-20T22:21:19.9314922Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:19.9315531Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:19.9315974Z - Magma 2.6.1 2024-08-20T22:21:19.9323002Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:19.9329452Z 2024-08-20T22:21:20.2597498Z + cd test 2024-08-20T22:21:20.2598174Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:21.6071688Z ATen/Parallel: 2024-08-20T22:21:21.6072236Z at::get_num_threads() : 8 2024-08-20T22:21:21.6072691Z at::get_num_interop_threads() : 16 2024-08-20T22:21:21.6073192Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:21.6073720Z omp_get_max_threads() : 8 2024-08-20T22:21:21.6074938Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:21.6075734Z mkl_get_max_threads() : 8 2024-08-20T22:21:21.6076366Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:21.6077003Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:21.6077434Z Environment variables: 2024-08-20T22:21:21.6077787Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:21.6078160Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:21.6078531Z ATen parallel backend: OpenMP 2024-08-20T22:21:21.6078819Z 2024-08-20T22:21:21.8900177Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:21.8900979Z + [[ inductor_torchbench == *backward* ]] 2024-08-20T22:21:21.8901465Z + [[ inductor_torchbench == *xla* ]] 2024-08-20T22:21:21.8901912Z + [[ inductor_torchbench == *executorch* ]] 2024-08-20T22:21:21.8902408Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:21.8903382Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:21.8903940Z + [[ inductor_torchbench == distributed ]] 2024-08-20T22:21:21.8904548Z + [[ inductor_torchbench == *inductor_distributed* ]] 2024-08-20T22:21:21.8905320Z + [[ inductor_torchbench == *inductor-halide* ]] 2024-08-20T22:21:21.8905934Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:21:21.8906471Z + [[ inductor_torchbench == *huggingface* ]] 2024-08-20T22:21:21.8906932Z + [[ inductor_torchbench == *timm* ]] 2024-08-20T22:21:21.8907371Z + [[ inductor_torchbench == *torchbench* ]] 2024-08-20T22:21:21.8907839Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T22:21:21.8908248Z + install_torchaudio cuda 2024-08-20T22:21:21.8908597Z + local commit 2024-08-20T22:21:21.8908912Z ++ get_pinned_commit audio 2024-08-20T22:21:21.8909293Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:21:21.8922341Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:21.8922974Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:21:21.8923516Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:21:21.8924453Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:21.8925897Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.2281468Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:22.2286059Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-h7o4j1js 2024-08-20T22:21:22.2316943Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-h7o4j1js 2024-08-20T22:21:23.0221243Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:21:23.0248490Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:23.4357937Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:23.4359114Z Running command git submodule update --init --recursive -q 2024-08-20T22:21:25.8650881Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:21:25.8681242Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:25.8709115Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:21:25.8713644Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:21:25.8716036Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:21:25.8719469Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:21:25.8722713Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:21:25.8727810Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:21:25.8742364Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:21:25.9217486Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:21:25.9282238Z Building wheels for collected packages: torchaudio 2024-08-20T22:23:59.9251964Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:23:59.9288510Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310393 sha256=aade3d1d430d28da21c9928e9102c718c4ae2074c16de6444d85aa692c675956 2024-08-20T22:23:59.9290206Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:23:59.9319394Z Successfully built torchaudio 2024-08-20T22:24:00.4205898Z Installing collected packages: torchaudio 2024-08-20T22:24:00.6372923Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:24:00.8179592Z + install_torchtext 2024-08-20T22:24:00.8179992Z + local data_commit 2024-08-20T22:24:00.8182681Z + local text_commit 2024-08-20T22:24:00.8184873Z ++ get_pinned_commit data 2024-08-20T22:24:00.8185273Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:24:00.8214248Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:00.8217559Z ++ get_pinned_commit text 2024-08-20T22:24:00.8217950Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:24:00.8235360Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:24:00.8237512Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:00.8238989Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:01.1645421Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:01.1652529Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-kd7lnm11 2024-08-20T22:24:01.1698081Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-kd7lnm11 2024-08-20T22:24:01.6455800Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:24:01.6485276Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:01.9036740Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:02.0532824Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:24:02.0533937Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:21.8869009Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:21.8907088Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:25:21.8909859Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:25:21.8913793Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:21.8942856Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:25:21.8947095Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:25:21.8950468Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:25:21.8953330Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:25:21.8956317Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:25:21.8961100Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:25:21.8978219Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:25:21.9035135Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:25:21.9038473Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:25:21.9043086Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:25:21.9537422Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:25:21.9601453Z Building wheels for collected packages: torchdata 2024-08-20T22:25:23.9417574Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:25:23.9429004Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=f02eb21b8b67912e329886d3faaffe002140066e85f11a93a18db41ed6cfeb85 2024-08-20T22:25:23.9431324Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:25:23.9458197Z Successfully built torchdata 2024-08-20T22:25:24.4235567Z Installing collected packages: torchdata 2024-08-20T22:25:24.5220676Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:25:29.4605293Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:29.4606794Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:29.7978916Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:29.7983121Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-48zsz4ig 2024-08-20T22:25:29.8013464Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-48zsz4ig 2024-08-20T22:25:30.6524156Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:25:30.6551251Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:31.1465135Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:31.3146580Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:31.3147717Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:38.9878417Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:38.9912580Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:25:38.9916001Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:25:38.9919459Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:38.9922889Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:25:38.9926144Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:25:38.9952611Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:25:38.9956348Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:25:38.9960761Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:25:38.9964748Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:25:38.9992097Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:25:38.9996154Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:25:38.9998905Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:25:39.0003455Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:25:39.0006554Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:25:39.0011892Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:25:39.0026802Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:25:39.0607408Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:25:39.0674057Z Building wheels for collected packages: torchtext 2024-08-20T22:26:17.6607558Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T22:26:17.6641091Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045465 sha256=d5b40865ce27c500c83865b313640009194ea9bf1b6e025e9bd27cebd706cd57 2024-08-20T22:26:17.6642914Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:26:17.6670361Z Successfully built torchtext 2024-08-20T22:26:18.1503968Z Installing collected packages: torchtext 2024-08-20T22:26:18.2797199Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:26:18.4417451Z + install_torchvision 2024-08-20T22:26:18.4417864Z + local orig_preload 2024-08-20T22:26:18.4418192Z + local commit 2024-08-20T22:26:18.4421137Z ++ get_pinned_commit vision 2024-08-20T22:26:18.4421675Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:26:18.4440653Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:18.4441254Z + orig_preload= 2024-08-20T22:26:18.4441792Z + '[' -n '' ']' 2024-08-20T22:26:18.4442650Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:18.4445190Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:18.7815504Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:18.7818966Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-_s9kw_ic 2024-08-20T22:26:18.7849849Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-_s9kw_ic 2024-08-20T22:26:20.2542133Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:26:20.2569030Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:21.6153001Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:21.9107204Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:24.3909727Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:26:24.3941804Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:26:24.3945793Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:26:24.3950663Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:26:24.4021565Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:26:24.4026213Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:26:24.4029133Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:26:24.4032104Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:26:24.4035209Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:26:24.4040269Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:26:24.4054226Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:26:24.4534045Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:26:24.4596516Z Building wheels for collected packages: torchvision 2024-08-20T22:27:43.0083503Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:27:43.0116895Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116136 sha256=7987baf979a88006452ce080b35a0d7c58a6ad2d121e92447efaa5ea7de80035 2024-08-20T22:27:43.0120295Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:27:43.0153763Z Successfully built torchvision 2024-08-20T22:27:43.5003104Z Installing collected packages: torchvision 2024-08-20T22:27:43.9155735Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:27:44.0604498Z + '[' -n '' ']' 2024-08-20T22:27:44.0604979Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:27:44.0605485Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:27:44.0608294Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:27:44.3986599Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:27:44.3990891Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-y9ioj9gf 2024-08-20T22:27:44.4021103Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-y9ioj9gf 2024-08-20T22:27:44.9631133Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:27:47.3953898Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:27:47.3989732Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:29:06.6781898Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - done 2024-08-20T22:29:06.6799914Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928016 sha256=ea766eb5fef2f3798c2103d1e7f5c4ac1018483673df45277a83fed4abd59d47 2024-08-20T22:29:06.6802671Z Stored in directory: /tmp/pip-ephem-wheel-cache-jxvi8d4r/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:29:06.6832062Z Successfully built torchao 2024-08-20T22:29:07.2320630Z Installing collected packages: torchao 2024-08-20T22:29:07.4753714Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:29:07.5837638Z + id=0 2024-08-20T22:29:07.5838960Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:29:07.5839627Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:29:08.0514816Z Collecting opencv-python==4.8.0.74 2024-08-20T22:29:08.0688425Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:29:08.0777271Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:29:08.0847627Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:29:09.1647201Z Installing collected packages: opencv-python 2024-08-20T22:29:10.0853706Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:29:10.1860711Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:29:10.1861473Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:29:10.1863169Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:29:10.1863735Z + checkout_install_torchbench 2024-08-20T22:29:10.1864182Z + local commit 2024-08-20T22:29:10.1864724Z ++ get_pinned_commit torchbench 2024-08-20T22:29:10.1865205Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:29:10.1883023Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:10.1883638Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:29:10.1899813Z Cloning into 'torchbench'... 2024-08-20T22:29:10.3397191Z remote: Enumerating objects: 31006, done. 2024-08-20T22:29:10.3398094Z remote: Counting objects: 0% (1/1746) 2024-08-20T22:29:10.3399472Z remote: Counting objects: 1% (18/1746) 2024-08-20T22:29:10.3403367Z remote: Counting objects: 2% (35/1746) 2024-08-20T22:29:10.3404039Z remote: Counting objects: 3% (53/1746) 2024-08-20T22:29:10.3404654Z remote: Counting objects: 4% (70/1746) 2024-08-20T22:29:10.3405206Z remote: Counting objects: 5% (88/1746) 2024-08-20T22:29:10.3405829Z remote: Counting objects: 6% (105/1746) 2024-08-20T22:29:10.3406430Z remote: Counting objects: 7% (123/1746) 2024-08-20T22:29:10.3406975Z remote: Counting objects: 8% (140/1746) 2024-08-20T22:29:10.3408339Z remote: Counting objects: 9% (158/1746) 2024-08-20T22:29:10.3409000Z remote: Counting objects: 10% (175/1746) 2024-08-20T22:29:10.3409723Z remote: Counting objects: 11% (193/1746) 2024-08-20T22:29:10.3410378Z remote: Counting objects: 12% (210/1746) 2024-08-20T22:29:10.3410918Z remote: Counting objects: 13% (227/1746) 2024-08-20T22:29:10.3411456Z remote: Counting objects: 14% (245/1746) 2024-08-20T22:29:10.3411980Z remote: Counting objects: 15% (262/1746) 2024-08-20T22:29:10.3412591Z remote: Counting objects: 16% (280/1746) 2024-08-20T22:29:10.3413286Z remote: Counting objects: 17% (297/1746) 2024-08-20T22:29:10.3413970Z remote: Counting objects: 18% (315/1746) 2024-08-20T22:29:10.3414639Z remote: Counting objects: 19% (332/1746) 2024-08-20T22:29:10.3415329Z remote: Counting objects: 20% (350/1746) 2024-08-20T22:29:10.3416000Z remote: Counting objects: 21% (367/1746) 2024-08-20T22:29:10.3416690Z remote: Counting objects: 22% (385/1746) 2024-08-20T22:29:10.3417274Z remote: Counting objects: 23% (402/1746) 2024-08-20T22:29:10.3417851Z remote: Counting objects: 24% (420/1746) 2024-08-20T22:29:10.3418556Z remote: Counting objects: 25% (437/1746) 2024-08-20T22:29:10.3419198Z remote: Counting objects: 26% (454/1746) 2024-08-20T22:29:10.3419724Z remote: Counting objects: 27% (472/1746) 2024-08-20T22:29:10.3420248Z remote: Counting objects: 28% (489/1746) 2024-08-20T22:29:10.3420766Z remote: Counting objects: 29% (507/1746) 2024-08-20T22:29:10.3421284Z remote: Counting objects: 30% (524/1746) 2024-08-20T22:29:10.3422390Z remote: Counting objects: 31% (542/1746) 2024-08-20T22:29:10.3423184Z remote: Counting objects: 32% (559/1746) 2024-08-20T22:29:10.3423922Z remote: Counting objects: 33% (577/1746) 2024-08-20T22:29:10.3424446Z remote: Counting objects: 34% (594/1746) 2024-08-20T22:29:10.3424960Z remote: Counting objects: 35% (612/1746) 2024-08-20T22:29:10.3425483Z remote: Counting objects: 36% (629/1746) 2024-08-20T22:29:10.3426028Z remote: Counting objects: 37% (647/1746) 2024-08-20T22:29:10.3426540Z remote: Counting objects: 38% (664/1746) 2024-08-20T22:29:10.3427124Z remote: Counting objects: 39% (681/1746) 2024-08-20T22:29:10.3427745Z remote: Counting objects: 40% (699/1746) 2024-08-20T22:29:10.3428354Z remote: Counting objects: 41% (716/1746) 2024-08-20T22:29:10.3428871Z remote: Counting objects: 42% (734/1746) 2024-08-20T22:29:10.3429391Z remote: Counting objects: 43% (751/1746) 2024-08-20T22:29:10.3429914Z remote: Counting objects: 44% (769/1746) 2024-08-20T22:29:10.3430456Z remote: Counting objects: 45% (786/1746) 2024-08-20T22:29:10.3430973Z remote: Counting objects: 46% (804/1746) 2024-08-20T22:29:10.3431501Z remote: Counting objects: 47% (821/1746) 2024-08-20T22:29:10.3432032Z remote: Counting objects: 48% (839/1746) 2024-08-20T22:29:10.3432550Z remote: Counting objects: 49% (856/1746) 2024-08-20T22:29:10.3433100Z remote: Counting objects: 50% (873/1746) 2024-08-20T22:29:10.3433627Z remote: Counting objects: 51% (891/1746) 2024-08-20T22:29:10.3434146Z remote: Counting objects: 52% (908/1746) 2024-08-20T22:29:10.3434671Z remote: Counting objects: 53% (926/1746) 2024-08-20T22:29:10.3435193Z remote: Counting objects: 54% (943/1746) 2024-08-20T22:29:10.3435718Z remote: Counting objects: 55% (961/1746) 2024-08-20T22:29:10.3436242Z remote: Counting objects: 56% (978/1746) 2024-08-20T22:29:10.3436764Z remote: Counting objects: 57% (996/1746) 2024-08-20T22:29:10.3437288Z remote: Counting objects: 58% (1013/1746) 2024-08-20T22:29:10.3437835Z remote: Counting objects: 59% (1031/1746) 2024-08-20T22:29:10.3438370Z remote: Counting objects: 60% (1048/1746) 2024-08-20T22:29:10.3438904Z remote: Counting objects: 61% (1066/1746) 2024-08-20T22:29:10.3439449Z remote: Counting objects: 62% (1083/1746) 2024-08-20T22:29:10.3439982Z remote: Counting objects: 63% (1100/1746) 2024-08-20T22:29:10.3440515Z remote: Counting objects: 64% (1118/1746) 2024-08-20T22:29:10.3441037Z remote: Counting objects: 65% (1135/1746) 2024-08-20T22:29:10.3441568Z remote: Counting objects: 66% (1153/1746) 2024-08-20T22:29:10.3442097Z remote: Counting objects: 67% (1170/1746) 2024-08-20T22:29:10.3442632Z remote: Counting objects: 68% (1188/1746) 2024-08-20T22:29:10.3443168Z remote: Counting objects: 69% (1205/1746) 2024-08-20T22:29:10.3443700Z remote: Counting objects: 70% (1223/1746) 2024-08-20T22:29:10.3444231Z remote: Counting objects: 71% (1240/1746) 2024-08-20T22:29:10.3444763Z remote: Counting objects: 72% (1258/1746) 2024-08-20T22:29:10.3445293Z remote: Counting objects: 73% (1275/1746) 2024-08-20T22:29:10.3445824Z remote: Counting objects: 74% (1293/1746) 2024-08-20T22:29:10.3446368Z remote: Counting objects: 75% (1310/1746) 2024-08-20T22:29:10.3446901Z remote: Counting objects: 76% (1327/1746) 2024-08-20T22:29:10.3447426Z remote: Counting objects: 77% (1345/1746) 2024-08-20T22:29:10.3447965Z remote: Counting objects: 78% (1362/1746) 2024-08-20T22:29:10.3448496Z remote: Counting objects: 79% (1380/1746) 2024-08-20T22:29:10.3449024Z remote: Counting objects: 80% (1397/1746) 2024-08-20T22:29:10.3449551Z remote: Counting objects: 81% (1415/1746) 2024-08-20T22:29:10.3450080Z remote: Counting objects: 82% (1432/1746) 2024-08-20T22:29:10.3450606Z remote: Counting objects: 83% (1450/1746) 2024-08-20T22:29:10.3451131Z remote: Counting objects: 84% (1467/1746) 2024-08-20T22:29:10.3451659Z remote: Counting objects: 85% (1485/1746) 2024-08-20T22:29:10.3452308Z remote: Counting objects: 86% (1502/1746) 2024-08-20T22:29:10.3452848Z remote: Counting objects: 87% (1520/1746) 2024-08-20T22:29:10.3453451Z remote: Counting objects: 88% (1537/1746) 2024-08-20T22:29:10.3453991Z remote: Counting objects: 89% (1554/1746) 2024-08-20T22:29:10.3454523Z remote: Counting objects: 90% (1572/1746) 2024-08-20T22:29:10.3455055Z remote: Counting objects: 91% (1589/1746) 2024-08-20T22:29:10.3455584Z remote: Counting objects: 92% (1607/1746) 2024-08-20T22:29:10.3456123Z remote: Counting objects: 93% (1624/1746) 2024-08-20T22:29:10.3456654Z remote: Counting objects: 94% (1642/1746) 2024-08-20T22:29:10.3457204Z remote: Counting objects: 95% (1659/1746) 2024-08-20T22:29:10.3457760Z remote: Counting objects: 96% (1677/1746) 2024-08-20T22:29:10.3458290Z remote: Counting objects: 97% (1694/1746) 2024-08-20T22:29:10.3458820Z remote: Counting objects: 98% (1712/1746) 2024-08-20T22:29:10.3459349Z remote: Counting objects: 99% (1729/1746) 2024-08-20T22:29:10.3459881Z remote: Counting objects: 100% (1746/1746) 2024-08-20T22:29:10.3460460Z remote: Counting objects: 100% (1746/1746), done. 2024-08-20T22:29:10.3461037Z remote: Compressing objects: 0% (1/904) 2024-08-20T22:29:10.3506710Z remote: Compressing objects: 1% (10/904) 2024-08-20T22:29:10.3620368Z remote: Compressing objects: 2% (19/904) 2024-08-20T22:29:10.3703071Z remote: Compressing objects: 3% (28/904) 2024-08-20T22:29:10.3751765Z remote: Compressing objects: 4% (37/904) 2024-08-20T22:29:10.3799651Z remote: Compressing objects: 5% (46/904) 2024-08-20T22:29:10.3834369Z remote: Compressing objects: 6% (55/904) 2024-08-20T22:29:10.3863342Z remote: Compressing objects: 7% (64/904) 2024-08-20T22:29:10.3894784Z remote: Compressing objects: 8% (73/904) 2024-08-20T22:29:10.3934211Z remote: Compressing objects: 9% (82/904) 2024-08-20T22:29:10.3957812Z remote: Compressing objects: 10% (91/904) 2024-08-20T22:29:10.3974655Z remote: Compressing objects: 11% (100/904) 2024-08-20T22:29:10.3985468Z remote: Compressing objects: 12% (109/904) 2024-08-20T22:29:10.4000981Z remote: Compressing objects: 13% (118/904) 2024-08-20T22:29:10.4016424Z remote: Compressing objects: 14% (127/904) 2024-08-20T22:29:10.4026362Z remote: Compressing objects: 15% (136/904) 2024-08-20T22:29:10.4033645Z remote: Compressing objects: 16% (145/904) 2024-08-20T22:29:10.4041009Z remote: Compressing objects: 17% (154/904) 2024-08-20T22:29:10.4044114Z remote: Compressing objects: 18% (163/904) 2024-08-20T22:29:10.4046877Z remote: Compressing objects: 19% (172/904) 2024-08-20T22:29:10.4048836Z remote: Compressing objects: 20% (181/904) 2024-08-20T22:29:10.4050545Z remote: Compressing objects: 21% (190/904) 2024-08-20T22:29:10.4051105Z remote: Compressing objects: 22% (199/904) 2024-08-20T22:29:10.4053502Z remote: Compressing objects: 23% (208/904) 2024-08-20T22:29:10.4054988Z remote: Compressing objects: 24% (217/904) 2024-08-20T22:29:10.4055901Z remote: Compressing objects: 25% (226/904) 2024-08-20T22:29:10.4057600Z remote: Compressing objects: 26% (236/904) 2024-08-20T22:29:10.4069723Z remote: Compressing objects: 27% (245/904) 2024-08-20T22:29:10.4070328Z remote: Compressing objects: 28% (254/904) 2024-08-20T22:29:10.4072588Z remote: Compressing objects: 29% (263/904) 2024-08-20T22:29:10.4075022Z remote: Compressing objects: 30% (272/904) 2024-08-20T22:29:10.4077477Z remote: Compressing objects: 31% (281/904) 2024-08-20T22:29:10.4080064Z remote: Compressing objects: 32% (290/904) 2024-08-20T22:29:10.4083232Z remote: Compressing objects: 33% (299/904) 2024-08-20T22:29:10.4086590Z remote: Compressing objects: 34% (308/904) 2024-08-20T22:29:10.4089210Z remote: Compressing objects: 35% (317/904) 2024-08-20T22:29:10.4091596Z remote: Compressing objects: 36% (326/904) 2024-08-20T22:29:10.4093689Z remote: Compressing objects: 37% (335/904) 2024-08-20T22:29:10.4095627Z remote: Compressing objects: 38% (344/904) 2024-08-20T22:29:10.4097458Z remote: Compressing objects: 39% (353/904) 2024-08-20T22:29:10.4098522Z remote: Compressing objects: 40% (362/904) 2024-08-20T22:29:10.4100523Z remote: Compressing objects: 41% (371/904) 2024-08-20T22:29:10.4101217Z remote: Compressing objects: 42% (380/904) 2024-08-20T22:29:10.4101902Z remote: Compressing objects: 43% (389/904) 2024-08-20T22:29:10.4102615Z remote: Compressing objects: 44% (398/904) 2024-08-20T22:29:10.4103846Z remote: Compressing objects: 45% (407/904) 2024-08-20T22:29:10.4104595Z remote: Compressing objects: 46% (416/904) 2024-08-20T22:29:10.4105332Z remote: Compressing objects: 47% (425/904) 2024-08-20T22:29:10.4106049Z remote: Compressing objects: 48% (434/904) 2024-08-20T22:29:10.4106767Z remote: Compressing objects: 49% (443/904) 2024-08-20T22:29:10.4107322Z remote: Compressing objects: 50% (452/904) 2024-08-20T22:29:10.4108041Z remote: Compressing objects: 51% (462/904) 2024-08-20T22:29:10.4108701Z remote: Compressing objects: 52% (471/904) 2024-08-20T22:29:10.4109352Z remote: Compressing objects: 53% (480/904) 2024-08-20T22:29:10.4110082Z remote: Compressing objects: 54% (489/904) 2024-08-20T22:29:10.4110725Z remote: Compressing objects: 55% (498/904) 2024-08-20T22:29:10.4111434Z remote: Compressing objects: 56% (507/904) 2024-08-20T22:29:10.4112007Z remote: Compressing objects: 57% (516/904) 2024-08-20T22:29:10.4112667Z remote: Compressing objects: 58% (525/904) 2024-08-20T22:29:10.4113355Z remote: Compressing objects: 59% (534/904) 2024-08-20T22:29:10.4113898Z remote: Compressing objects: 60% (543/904) 2024-08-20T22:29:10.4114439Z remote: Compressing objects: 61% (552/904) 2024-08-20T22:29:10.4114983Z remote: Compressing objects: 62% (561/904) 2024-08-20T22:29:10.4115528Z remote: Compressing objects: 63% (570/904) 2024-08-20T22:29:10.4116067Z remote: Compressing objects: 64% (579/904) 2024-08-20T22:29:10.4116620Z remote: Compressing objects: 65% (588/904) 2024-08-20T22:29:10.4117172Z remote: Compressing objects: 66% (597/904) 2024-08-20T22:29:10.4117721Z remote: Compressing objects: 67% (606/904) 2024-08-20T22:29:10.4118269Z remote: Compressing objects: 68% (615/904) 2024-08-20T22:29:10.4118820Z remote: Compressing objects: 69% (624/904) 2024-08-20T22:29:10.4119365Z remote: Compressing objects: 70% (633/904) 2024-08-20T22:29:10.4119905Z remote: Compressing objects: 71% (642/904) 2024-08-20T22:29:10.4120443Z remote: Compressing objects: 72% (651/904) 2024-08-20T22:29:10.4120993Z remote: Compressing objects: 73% (660/904) 2024-08-20T22:29:10.4121536Z remote: Compressing objects: 74% (669/904) 2024-08-20T22:29:10.4170734Z remote: Compressing objects: 75% (678/904) 2024-08-20T22:29:10.4171396Z remote: Compressing objects: 76% (688/904) 2024-08-20T22:29:10.4172037Z remote: Compressing objects: 77% (697/904) 2024-08-20T22:29:10.4172585Z remote: Compressing objects: 78% (706/904) 2024-08-20T22:29:10.4173138Z remote: Compressing objects: 79% (715/904) 2024-08-20T22:29:10.4173747Z remote: Compressing objects: 80% (724/904) 2024-08-20T22:29:10.4174374Z remote: Compressing objects: 81% (733/904) 2024-08-20T22:29:10.4174954Z remote: Compressing objects: 82% (742/904) 2024-08-20T22:29:10.4175543Z remote: Compressing objects: 83% (751/904) 2024-08-20T22:29:10.4176090Z remote: Compressing objects: 84% (760/904) 2024-08-20T22:29:10.4178365Z remote: Compressing objects: 85% (769/904) 2024-08-20T22:29:10.4182656Z remote: Compressing objects: 86% (778/904) 2024-08-20T22:29:10.4186778Z remote: Compressing objects: 87% (787/904) 2024-08-20T22:29:10.4190614Z remote: Compressing objects: 88% (796/904) 2024-08-20T22:29:10.4194153Z remote: Compressing objects: 89% (805/904) 2024-08-20T22:29:10.4197393Z remote: Compressing objects: 90% (814/904) 2024-08-20T22:29:10.4200273Z remote: Compressing objects: 91% (823/904) 2024-08-20T22:29:10.4203837Z remote: Compressing objects: 92% (832/904) 2024-08-20T22:29:10.4207368Z remote: Compressing objects: 93% (841/904) 2024-08-20T22:29:10.4209369Z remote: Compressing objects: 94% (850/904) 2024-08-20T22:29:10.4211626Z remote: Compressing objects: 95% (859/904) 2024-08-20T22:29:10.4214385Z remote: Compressing objects: 96% (868/904) 2024-08-20T22:29:10.4216970Z remote: Compressing objects: 97% (877/904) 2024-08-20T22:29:10.4218000Z remote: Compressing objects: 98% (886/904) 2024-08-20T22:29:10.4219397Z remote: Compressing objects: 99% (895/904) 2024-08-20T22:29:10.4220425Z remote: Compressing objects: 100% (904/904) 2024-08-20T22:29:10.4221115Z remote: Compressing objects: 100% (904/904), done. 2024-08-20T22:29:10.4295858Z Receiving objects: 0% (1/31006) 2024-08-20T22:29:10.4339581Z Receiving objects: 1% (311/31006) 2024-08-20T22:29:10.4382753Z Receiving objects: 2% (621/31006) 2024-08-20T22:29:10.4416427Z Receiving objects: 3% (931/31006) 2024-08-20T22:29:10.4447844Z Receiving objects: 4% (1241/31006) 2024-08-20T22:29:10.4479839Z Receiving objects: 5% (1551/31006) 2024-08-20T22:29:10.4516566Z Receiving objects: 6% (1861/31006) 2024-08-20T22:29:10.4555585Z Receiving objects: 7% (2171/31006) 2024-08-20T22:29:10.4586407Z Receiving objects: 8% (2481/31006) 2024-08-20T22:29:10.4620692Z Receiving objects: 9% (2791/31006) 2024-08-20T22:29:10.4690441Z Receiving objects: 10% (3101/31006) 2024-08-20T22:29:10.4739040Z Receiving objects: 11% (3411/31006) 2024-08-20T22:29:10.4805012Z Receiving objects: 12% (3721/31006) 2024-08-20T22:29:10.4935531Z Receiving objects: 13% (4031/31006) 2024-08-20T22:29:10.5080354Z Receiving objects: 14% (4341/31006) 2024-08-20T22:29:10.5151662Z Receiving objects: 15% (4651/31006) 2024-08-20T22:29:10.5225489Z Receiving objects: 16% (4961/31006) 2024-08-20T22:29:10.5284215Z Receiving objects: 17% (5272/31006) 2024-08-20T22:29:10.5312202Z Receiving objects: 18% (5582/31006) 2024-08-20T22:29:10.5370893Z Receiving objects: 19% (5892/31006) 2024-08-20T22:29:10.5403479Z Receiving objects: 20% (6202/31006) 2024-08-20T22:29:10.5418073Z Receiving objects: 21% (6512/31006) 2024-08-20T22:29:10.5618587Z Receiving objects: 22% (6822/31006) 2024-08-20T22:29:11.4245321Z Receiving objects: 23% (7132/31006) 2024-08-20T22:29:11.5489862Z Receiving objects: 23% (7375/31006), 75.06 MiB | 75.06 MiB/s 2024-08-20T22:29:12.1005131Z Receiving objects: 24% (7442/31006), 75.06 MiB | 75.06 MiB/s 2024-08-20T22:29:12.1728731Z Receiving objects: 25% (7752/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.2457432Z Receiving objects: 26% (8062/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.3169681Z Receiving objects: 27% (8372/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.4039827Z Receiving objects: 28% (8682/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.4244769Z Receiving objects: 29% (8992/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.4916660Z Receiving objects: 29% (9063/31006), 112.80 MiB | 75.25 MiB/s 2024-08-20T22:29:12.5686281Z Receiving objects: 30% (9302/31006), 149.35 MiB | 74.71 MiB/s 2024-08-20T22:29:12.6402272Z Receiving objects: 31% (9612/31006), 149.35 MiB | 74.71 MiB/s 2024-08-20T22:29:12.8738720Z Receiving objects: 32% (9922/31006), 149.35 MiB | 74.71 MiB/s 2024-08-20T22:29:12.9484381Z Receiving objects: 33% (10232/31006), 149.35 MiB | 74.71 MiB/s 2024-08-20T22:29:12.9610535Z Receiving objects: 34% (10543/31006), 183.42 MiB | 73.40 MiB/s 2024-08-20T22:29:13.2217031Z Receiving objects: 35% (10853/31006), 183.42 MiB | 73.40 MiB/s 2024-08-20T22:29:13.4247088Z Receiving objects: 36% (11163/31006), 183.42 MiB | 73.40 MiB/s 2024-08-20T22:29:13.4668203Z Receiving objects: 36% (11293/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4685927Z Receiving objects: 37% (11473/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4708304Z Receiving objects: 38% (11783/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4726193Z Receiving objects: 39% (12093/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4740882Z Receiving objects: 40% (12403/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4752220Z Receiving objects: 41% (12713/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4764593Z Receiving objects: 42% (13023/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4780550Z Receiving objects: 43% (13333/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4796824Z Receiving objects: 44% (13643/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4804694Z Receiving objects: 45% (13953/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4809476Z Receiving objects: 46% (14263/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4823698Z Receiving objects: 47% (14573/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4844898Z Receiving objects: 48% (14883/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.4857800Z Receiving objects: 49% (15193/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5061203Z Receiving objects: 50% (15503/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5067513Z Receiving objects: 51% (15814/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5074232Z Receiving objects: 52% (16124/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5080785Z Receiving objects: 53% (16434/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5086060Z Receiving objects: 54% (16744/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5103316Z Receiving objects: 55% (17054/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5114110Z Receiving objects: 56% (17364/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5120189Z Receiving objects: 57% (17674/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5133914Z Receiving objects: 58% (17984/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5146540Z Receiving objects: 59% (18294/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5156624Z Receiving objects: 60% (18604/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5297946Z Receiving objects: 61% (18914/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5303556Z Receiving objects: 62% (19224/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5309534Z Receiving objects: 63% (19534/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5317446Z Receiving objects: 64% (19844/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5325449Z Receiving objects: 65% (20154/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5334226Z Receiving objects: 66% (20464/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5589996Z Receiving objects: 67% (20775/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5988604Z Receiving objects: 68% (21085/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.5996576Z Receiving objects: 69% (21395/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6009524Z Receiving objects: 70% (21705/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6020526Z Receiving objects: 71% (22015/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6035225Z Receiving objects: 72% (22325/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6044654Z Receiving objects: 73% (22635/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6060875Z Receiving objects: 74% (22945/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6071166Z Receiving objects: 75% (23255/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6078371Z Receiving objects: 76% (23565/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6086196Z Receiving objects: 77% (23875/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6303637Z Receiving objects: 78% (24185/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6335212Z Receiving objects: 79% (24495/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6344917Z Receiving objects: 80% (24805/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6356764Z Receiving objects: 81% (25115/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6369483Z Receiving objects: 82% (25425/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6417009Z Receiving objects: 83% (25735/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6554998Z Receiving objects: 84% (26046/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6560567Z Receiving objects: 85% (26356/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6568371Z Receiving objects: 86% (26666/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6573646Z Receiving objects: 87% (26976/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6581118Z Receiving objects: 88% (27286/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6587384Z Receiving objects: 89% (27596/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6699346Z Receiving objects: 90% (27906/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6713662Z Receiving objects: 91% (28216/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6725312Z Receiving objects: 92% (28526/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6742070Z Receiving objects: 93% (28836/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6777038Z Receiving objects: 94% (29146/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6787971Z Receiving objects: 95% (29456/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6795792Z Receiving objects: 96% (29766/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6801028Z Receiving objects: 97% (30076/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6872232Z Receiving objects: 98% (30386/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6896006Z Receiving objects: 99% (30696/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6897053Z remote: Total 31006 (delta 1013), reused 1476 (delta 823), pack-reused 29260 (from 1) 2024-08-20T22:29:13.6913377Z Receiving objects: 100% (31006/31006), 218.47 MiB | 72.84 MiB/s 2024-08-20T22:29:13.6914050Z Receiving objects: 100% (31006/31006), 232.50 MiB | 71.17 MiB/s, done. 2024-08-20T22:29:13.6945120Z Resolving deltas: 0% (0/16726) 2024-08-20T22:29:13.6958918Z Resolving deltas: 1% (168/16726) 2024-08-20T22:29:13.6969446Z Resolving deltas: 2% (400/16726) 2024-08-20T22:29:13.6993174Z Resolving deltas: 3% (516/16726) 2024-08-20T22:29:13.7008862Z Resolving deltas: 4% (673/16726) 2024-08-20T22:29:13.7018435Z Resolving deltas: 5% (901/16726) 2024-08-20T22:29:13.7028357Z Resolving deltas: 6% (1019/16726) 2024-08-20T22:29:13.7040523Z Resolving deltas: 7% (1172/16726) 2024-08-20T22:29:13.7049649Z Resolving deltas: 8% (1355/16726) 2024-08-20T22:29:13.7061167Z Resolving deltas: 9% (1516/16726) 2024-08-20T22:29:13.7072737Z Resolving deltas: 10% (1677/16726) 2024-08-20T22:29:13.7085631Z Resolving deltas: 11% (1841/16726) 2024-08-20T22:29:13.7097181Z Resolving deltas: 12% (2013/16726) 2024-08-20T22:29:13.7111883Z Resolving deltas: 13% (2175/16726) 2024-08-20T22:29:13.7125403Z Resolving deltas: 14% (2392/16726) 2024-08-20T22:29:13.7141762Z Resolving deltas: 15% (2509/16726) 2024-08-20T22:29:13.7157815Z Resolving deltas: 16% (2681/16726) 2024-08-20T22:29:13.7170740Z Resolving deltas: 17% (2864/16726) 2024-08-20T22:29:13.7219948Z Resolving deltas: 18% (3019/16726) 2024-08-20T22:29:13.7243537Z Resolving deltas: 19% (3197/16726) 2024-08-20T22:29:13.7246373Z Resolving deltas: 20% (3494/16726) 2024-08-20T22:29:13.7259721Z Resolving deltas: 21% (3527/16726) 2024-08-20T22:29:13.7281302Z Resolving deltas: 22% (3681/16726) 2024-08-20T22:29:13.7293107Z Resolving deltas: 23% (3922/16726) 2024-08-20T22:29:13.7303783Z Resolving deltas: 24% (4074/16726) 2024-08-20T22:29:13.7344220Z Resolving deltas: 25% (4192/16726) 2024-08-20T22:29:13.7363316Z Resolving deltas: 26% (4360/16726) 2024-08-20T22:29:13.7379412Z Resolving deltas: 27% (4532/16726) 2024-08-20T22:29:13.7397093Z Resolving deltas: 28% (4690/16726) 2024-08-20T22:29:13.7414678Z Resolving deltas: 29% (4851/16726) 2024-08-20T22:29:13.7427244Z Resolving deltas: 30% (5043/16726) 2024-08-20T22:29:13.7438218Z Resolving deltas: 31% (5186/16726) 2024-08-20T22:29:13.7448348Z Resolving deltas: 32% (5358/16726) 2024-08-20T22:29:13.7464195Z Resolving deltas: 33% (5523/16726) 2024-08-20T22:29:13.7482521Z Resolving deltas: 34% (5705/16726) 2024-08-20T22:29:13.7502092Z Resolving deltas: 35% (5857/16726) 2024-08-20T22:29:13.7516976Z Resolving deltas: 36% (6038/16726) 2024-08-20T22:29:13.7527130Z Resolving deltas: 37% (6234/16726) 2024-08-20T22:29:13.7546495Z Resolving deltas: 38% (6357/16726) 2024-08-20T22:29:13.7559326Z Resolving deltas: 39% (6528/16726) 2024-08-20T22:29:13.7577900Z Resolving deltas: 40% (6693/16726) 2024-08-20T22:29:13.7591143Z Resolving deltas: 41% (6891/16726) 2024-08-20T22:29:13.7600616Z Resolving deltas: 42% (7025/16726) 2024-08-20T22:29:13.7611023Z Resolving deltas: 43% (7228/16726) 2024-08-20T22:29:13.7626825Z Resolving deltas: 44% (7368/16726) 2024-08-20T22:29:13.7650919Z Resolving deltas: 45% (7529/16726) 2024-08-20T22:29:13.7659252Z Resolving deltas: 46% (7703/16726) 2024-08-20T22:29:13.7679928Z Resolving deltas: 47% (7864/16726) 2024-08-20T22:29:13.7688432Z Resolving deltas: 48% (8030/16726) 2024-08-20T22:29:13.7696372Z Resolving deltas: 49% (8197/16726) 2024-08-20T22:29:13.7706980Z Resolving deltas: 50% (8366/16726) 2024-08-20T22:29:13.7735515Z Resolving deltas: 51% (8543/16726) 2024-08-20T22:29:13.7743174Z Resolving deltas: 53% (8883/16726) 2024-08-20T22:29:13.7752550Z Resolving deltas: 54% (9036/16726) 2024-08-20T22:29:13.7769065Z Resolving deltas: 55% (9201/16726) 2024-08-20T22:29:13.7777307Z Resolving deltas: 56% (9371/16726) 2024-08-20T22:29:13.7787438Z Resolving deltas: 57% (9570/16726) 2024-08-20T22:29:13.7791407Z Resolving deltas: 58% (9806/16726) 2024-08-20T22:29:13.7803519Z Resolving deltas: 59% (9879/16726) 2024-08-20T22:29:13.7819155Z Resolving deltas: 60% (10072/16726) 2024-08-20T22:29:13.7829439Z Resolving deltas: 61% (10328/16726) 2024-08-20T22:29:13.7831948Z Resolving deltas: 62% (10494/16726) 2024-08-20T22:29:13.7844073Z Resolving deltas: 63% (10538/16726) 2024-08-20T22:29:13.7857981Z Resolving deltas: 64% (10709/16726) 2024-08-20T22:29:13.7878042Z Resolving deltas: 65% (10912/16726) 2024-08-20T22:29:13.7882860Z Resolving deltas: 66% (11193/16726) 2024-08-20T22:29:13.7904628Z Resolving deltas: 67% (11237/16726) 2024-08-20T22:29:13.7913778Z Resolving deltas: 68% (11416/16726) 2024-08-20T22:29:13.7925653Z Resolving deltas: 69% (11588/16726) 2024-08-20T22:29:13.7940306Z Resolving deltas: 70% (11749/16726) 2024-08-20T22:29:13.7953516Z Resolving deltas: 71% (11876/16726) 2024-08-20T22:29:13.7964513Z Resolving deltas: 72% (12044/16726) 2024-08-20T22:29:13.7975257Z Resolving deltas: 73% (12213/16726) 2024-08-20T22:29:13.7996265Z Resolving deltas: 74% (12380/16726) 2024-08-20T22:29:13.8001393Z Resolving deltas: 75% (12677/16726) 2024-08-20T22:29:13.8007298Z Resolving deltas: 76% (12775/16726) 2024-08-20T22:29:13.8015475Z Resolving deltas: 77% (12897/16726) 2024-08-20T22:29:13.8034927Z Resolving deltas: 78% (13047/16726) 2024-08-20T22:29:13.8044166Z Resolving deltas: 79% (13223/16726) 2024-08-20T22:29:13.8057045Z Resolving deltas: 80% (13381/16726) 2024-08-20T22:29:13.8068489Z Resolving deltas: 81% (13572/16726) 2024-08-20T22:29:13.8081277Z Resolving deltas: 82% (13739/16726) 2024-08-20T22:29:13.8092777Z Resolving deltas: 83% (13883/16726) 2024-08-20T22:29:13.8100843Z Resolving deltas: 84% (14099/16726) 2024-08-20T22:29:13.8110115Z Resolving deltas: 85% (14244/16726) 2024-08-20T22:29:13.8119353Z Resolving deltas: 86% (14418/16726) 2024-08-20T22:29:13.8128607Z Resolving deltas: 87% (14553/16726) 2024-08-20T22:29:13.8139700Z Resolving deltas: 88% (14722/16726) 2024-08-20T22:29:13.8149651Z Resolving deltas: 89% (14895/16726) 2024-08-20T22:29:13.8160336Z Resolving deltas: 90% (15062/16726) 2024-08-20T22:29:13.8168956Z Resolving deltas: 91% (15221/16726) 2024-08-20T22:29:13.8175965Z Resolving deltas: 92% (15388/16726) 2024-08-20T22:29:13.8183938Z Resolving deltas: 93% (15557/16726) 2024-08-20T22:29:13.8193988Z Resolving deltas: 94% (15723/16726) 2024-08-20T22:29:13.8201810Z Resolving deltas: 95% (15932/16726) 2024-08-20T22:29:13.8208103Z Resolving deltas: 96% (16102/16726) 2024-08-20T22:29:13.8218019Z Resolving deltas: 97% (16235/16726) 2024-08-20T22:29:13.8263189Z Resolving deltas: 98% (16394/16726) 2024-08-20T22:29:13.8338513Z Resolving deltas: 99% (16561/16726) 2024-08-20T22:29:13.8339000Z Resolving deltas: 100% (16726/16726) 2024-08-20T22:29:13.8339621Z Resolving deltas: 100% (16726/16726), done. 2024-08-20T22:29:14.8355715Z + pushd torchbench 2024-08-20T22:29:14.8356210Z ~/workspace/torchbench ~/workspace 2024-08-20T22:29:14.8356860Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:14.9142336Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:29:14.9142951Z 2024-08-20T22:29:14.9143449Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:29:14.9144243Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:29:14.9145000Z state without impacting any branches by switching back to a branch. 2024-08-20T22:29:14.9145445Z 2024-08-20T22:29:14.9145758Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:29:14.9146568Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:29:14.9147014Z 2024-08-20T22:29:14.9147217Z git switch -c 2024-08-20T22:29:14.9147492Z 2024-08-20T22:29:14.9147646Z Or undo this operation with: 2024-08-20T22:29:14.9147898Z 2024-08-20T22:29:14.9148019Z git switch - 2024-08-20T22:29:14.9148207Z 2024-08-20T22:29:14.9148546Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:29:14.9149038Z 2024-08-20T22:29:14.9149205Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:29:14.9153634Z + '[' '' ']' 2024-08-20T22:29:14.9154023Z + python install.py --continue_on_fail 2024-08-20T22:29:17.2080041Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:29:35.5338798Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:29:38.6414378Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:29:40.5497234Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:29:42.2215584Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:29:42.2452168Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:29:49.7904643Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:29:52.5203043Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:29:55.2363578Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:29:57.9763818Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:29:57.9765989Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:29:59.0268463Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:30:01.7279076Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:30:01.7515703Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:30:59.4363086Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:31:08.0711267Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:31:15.2334550Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:31:22.0652769Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:31:30.9033404Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:31:38.1446278Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:31:44.4798610Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:31:51.4532019Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:31:58.7805609Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:32:06.0332140Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:32:12.8940901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:32:19.8085251Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:32:21.0068198Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:32:56.9314299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:33:01.4942601Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:33:06.7565291Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:33:17.8057286Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:33:18.9914984Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:33:20.4631661Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:33:27.1127328Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:33:34.1975440Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:33:39.6637697Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:33:47.3535146Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:33:53.2653707Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:33:58.2731930Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:34:04.5018446Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:34:21.0059829Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:34:27.1780765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:34:31.0888539Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:34:35.8215047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:34:42.6280721Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:34:42.6283227Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:34:55.6569053Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:34:59.5985366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:34:59.5987580Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:35:07.1965068Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:35:08.4277601Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:35:09.6717625Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:35:54.5420540Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:37:23.7536884Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:37:23.7821111Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:37:25.3228457Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:37:25.3469643Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:37:25.3712523Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:37:25.3954927Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:37:25.4199192Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:37:25.4443435Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:37:46.6506894Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:37:46.6508993Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:37:47.9016649Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:37:50.3348725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:37:50.3590774Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:37:50.3834837Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:37:50.4080657Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:37:50.4329352Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:37:50.4574413Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:37:57.1025135Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:37:58.5582622Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:38:02.0805156Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:38:02.1049020Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:38:02.1286324Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:38:02.1530503Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:38:02.1770284Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:38:02.2011087Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:38:17.5341715Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:38:35.1836261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:38:35.2082951Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:38:35.2085132Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:38:35.2087567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:38:37.7964091Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:38:39.6030266Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:38:39.6275016Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:39:04.8293688Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:39:11.4884889Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:39:15.5982002Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:39:22.1480077Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:39:22.1727984Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:39:22.1976461Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:39:22.2219462Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:39:22.2465379Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:39:22.2709977Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:39:22.2951315Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:39:22.3194033Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:39:29.2435275Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:39:40.5107868Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:39:40.5360477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:39:42.6323143Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:39:44.6050055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:39:45.0945672Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:39:45.0946508Z Print all dependencies after TorchBench is installed 2024-08-20T22:39:45.0949398Z + python -mpip freeze 2024-08-20T22:39:45.6102090Z absl-py==2.1.0 2024-08-20T22:39:45.6102538Z accelerate==0.33.0 2024-08-20T22:39:45.6102929Z aiohappyeyeballs==2.4.0 2024-08-20T22:39:45.6106629Z aiohttp==3.10.5 2024-08-20T22:39:45.6107141Z aiosignal==1.3.1 2024-08-20T22:39:45.6107871Z annotated-types==0.7.0 2024-08-20T22:39:45.6108503Z antlr4-python3-runtime==4.9.3 2024-08-20T22:39:45.6108960Z anyascii==0.3.2 2024-08-20T22:39:45.6109341Z astroid==3.2.4 2024-08-20T22:39:45.6109776Z astunparse==1.6.3 2024-08-20T22:39:45.6110333Z async-timeout==4.0.3 2024-08-20T22:39:45.6110820Z attrs==23.1.0 2024-08-20T22:39:45.6111211Z audioread==3.0.1 2024-08-20T22:39:45.6111628Z babel==2.16.0 2024-08-20T22:39:45.6112043Z beautifulsoup4==4.12.3 2024-08-20T22:39:45.6113491Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:39:45.6114735Z bibtexparser==2.0.0b7 2024-08-20T22:39:45.6115179Z black==24.8.0 2024-08-20T22:39:45.6115682Z blinker==1.8.2 2024-08-20T22:39:45.6116073Z blis==0.7.11 2024-08-20T22:39:45.6116442Z blobfile==2.1.1 2024-08-20T22:39:45.6116833Z bokeh==3.5.1 2024-08-20T22:39:45.6117234Z boto3==1.19.12 2024-08-20T22:39:45.6117641Z botocore==1.22.12 2024-08-20T22:39:45.6118053Z bs4==0.0.2 2024-08-20T22:39:45.6118868Z cachetools==5.5.0 2024-08-20T22:39:45.6119285Z cardboardlint==1.3.1 2024-08-20T22:39:45.6119727Z catalogue==2.0.10 2024-08-20T22:39:45.6120116Z certifi==2024.7.4 2024-08-20T22:39:45.6120768Z cffi==1.17.0 2024-08-20T22:39:45.6121279Z charset-normalizer==3.3.2 2024-08-20T22:39:45.6121755Z click==8.1.7 2024-08-20T22:39:45.6122134Z clldutils==3.22.2 2024-08-20T22:39:45.6122530Z cloudpathlib==0.18.1 2024-08-20T22:39:45.6122951Z cloudpickle==3.0.0 2024-08-20T22:39:45.6123291Z colorama==0.4.6 2024-08-20T22:39:45.6123585Z colorlog==6.8.2 2024-08-20T22:39:45.6123872Z confection==0.1.5 2024-08-20T22:39:45.6124171Z contourpy==1.2.1 2024-08-20T22:39:45.6124477Z coremltools==5.0b5 2024-08-20T22:39:45.6124778Z cryptography==43.0.0 2024-08-20T22:39:45.6125091Z csvw==3.3.0 2024-08-20T22:39:45.6125366Z cycler==0.12.1 2024-08-20T22:39:45.6125650Z cymem==2.0.8 2024-08-20T22:39:45.6125977Z DALL-E==0.1 2024-08-20T22:39:45.6126253Z datasets==2.21.0 2024-08-20T22:39:45.6126552Z decorator==5.1.1 2024-08-20T22:39:45.6126847Z defusedxml==0.7.1 2024-08-20T22:39:45.6127167Z Deprecated==1.2.14 2024-08-20T22:39:45.6127890Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:39:45.6128663Z diffusers==0.20.2 2024-08-20T22:39:45.6128956Z dill==0.3.8 2024-08-20T22:39:45.6129232Z diskcache==5.6.3 2024-08-20T22:39:45.6129527Z distro==1.9.0 2024-08-20T22:39:45.6129812Z dlinfo==1.2.1 2024-08-20T22:39:45.6130148Z docker-pycreds==0.4.0 2024-08-20T22:39:45.6130471Z dominate==2.9.1 2024-08-20T22:39:45.6131269Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:39:45.6132025Z einops==0.8.0 2024-08-20T22:39:45.6132326Z exceptiongroup==1.2.2 2024-08-20T22:39:45.6132650Z execnet==2.1.1 2024-08-20T22:39:45.6132944Z expecttest==0.1.6 2024-08-20T22:39:45.6133239Z FastNLP==0.6.0 2024-08-20T22:39:45.6133582Z ffmpeg-python==0.2.0 2024-08-20T22:39:45.6133901Z filelock==3.13.1 2024-08-20T22:39:45.6134196Z Flask==3.0.3 2024-08-20T22:39:45.6134488Z flatbuffers==2.0 2024-08-20T22:39:45.6134787Z fonttools==4.53.1 2024-08-20T22:39:45.6135088Z frozenlist==1.4.1 2024-08-20T22:39:45.6135379Z fsspec==2024.6.1 2024-08-20T22:39:45.6135677Z ftfy==6.2.3 2024-08-20T22:39:45.6136281Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:39:45.6136984Z future==1.0.0 2024-08-20T22:39:45.6137276Z fvcore==0.1.5.post20221221 2024-08-20T22:39:45.6137615Z gdown==5.2.0 2024-08-20T22:39:45.6137898Z geojson==2.5.0 2024-08-20T22:39:45.6138179Z ghstack==0.8.0 2024-08-20T22:39:45.6138467Z gitdb==4.0.11 2024-08-20T22:39:45.6138755Z GitPython==3.1.43 2024-08-20T22:39:45.6139113Z google-auth==2.34.0 2024-08-20T22:39:45.6139517Z google-auth-oauthlib==1.0.0 2024-08-20T22:39:45.6139889Z grpcio==1.65.5 2024-08-20T22:39:45.6140178Z gym==0.26.2 2024-08-20T22:39:45.6140484Z gym-notices==0.0.8 2024-08-20T22:39:45.6140789Z h5py==3.11.0 2024-08-20T22:39:45.6141068Z higher==0.2.1 2024-08-20T22:39:45.6141406Z huggingface-hub==0.24.6 2024-08-20T22:39:45.6141778Z hydra-core==1.3.2 2024-08-20T22:39:45.6142083Z hypothesis==5.35.1 2024-08-20T22:39:45.6142384Z idna==3.7 2024-08-20T22:39:45.6142651Z imageio==2.35.1 2024-08-20T22:39:45.6142956Z importlib_metadata==8.4.0 2024-08-20T22:39:45.6143461Z inflect==7.3.1 2024-08-20T22:39:45.6143756Z iniconfig==2.0.0 2024-08-20T22:39:45.6144047Z iopath==0.1.9 2024-08-20T22:39:45.6144337Z isodate==0.6.1 2024-08-20T22:39:45.6144621Z isort==5.13.2 2024-08-20T22:39:45.6144902Z itsdangerous==2.2.0 2024-08-20T22:39:45.6145212Z Jinja2==3.1.4 2024-08-20T22:39:45.6145500Z jmespath==0.10.0 2024-08-20T22:39:45.6145800Z joblib==1.4.2 2024-08-20T22:39:45.6146079Z jsonpatch==1.33 2024-08-20T22:39:45.6146378Z jsonpointer==3.0.0 2024-08-20T22:39:45.6146683Z jsonschema==4.23.0 2024-08-20T22:39:45.6147080Z jsonschema-specifications==2023.12.1 2024-08-20T22:39:45.6147483Z junitparser==2.1.1 2024-08-20T22:39:45.6147825Z kaldi-io==0.9.8 2024-08-20T22:39:45.6148118Z kiwisolver==1.4.5 2024-08-20T22:39:45.6148536Z kornia==0.7.3 2024-08-20T22:39:45.6148820Z kornia_rs==0.1.5 2024-08-20T22:39:45.6149117Z lameenc==1.7.0 2024-08-20T22:39:45.6149411Z langcodes==3.4.0 2024-08-20T22:39:45.6149788Z langdetect==1.0.9 2024-08-20T22:39:45.6150137Z language-tags==1.2.0 2024-08-20T22:39:45.6150468Z language_data==1.2.0 2024-08-20T22:39:45.6150775Z lark==0.12.0 2024-08-20T22:39:45.6151057Z lazy_loader==0.4 2024-08-20T22:39:45.6151351Z libcst==1.4.0 2024-08-20T22:39:45.6151633Z librosa==0.9.2 2024-08-20T22:39:45.6151918Z lintrunner==0.12.5 2024-08-20T22:39:45.6152221Z llvmlite==0.38.1 2024-08-20T22:39:45.6152511Z lxml==4.9.4 2024-08-20T22:39:45.6152823Z marisa-trie==1.2.0 2024-08-20T22:39:45.6153125Z Markdown==3.7 2024-08-20T22:39:45.6153458Z markdown-it-py==3.0.0 2024-08-20T22:39:45.6153789Z MarkupSafe==2.1.5 2024-08-20T22:39:45.6154090Z matplotlib==3.8.4 2024-08-20T22:39:45.6154388Z mccabe==0.7.0 2024-08-20T22:39:45.6154672Z mdurl==0.1.2 2024-08-20T22:39:45.6154985Z ml-dtypes==0.4.0 2024-08-20T22:39:45.6155287Z MonkeyType==23.3.0 2024-08-20T22:39:45.6155642Z more-itertools==10.4.0 2024-08-20T22:39:45.6155967Z mpmath==1.3.0 2024-08-20T22:39:45.6156246Z msgpack==1.0.8 2024-08-20T22:39:45.6156545Z multidict==6.0.5 2024-08-20T22:39:45.6156851Z multiprocess==0.70.16 2024-08-20T22:39:45.6157167Z murmurhash==1.0.10 2024-08-20T22:39:45.6157474Z musdb==0.4.2 2024-08-20T22:39:45.6157759Z museval==0.4.1 2024-08-20T22:39:45.6158048Z mypy==1.10.0 2024-08-20T22:39:45.6158382Z mypy-extensions==1.0.0 2024-08-20T22:39:45.6158710Z networkx==2.8.8 2024-08-20T22:39:45.6158998Z ninja==1.10.2 2024-08-20T22:39:45.6159273Z nose==1.3.7 2024-08-20T22:39:45.6159545Z numba==0.55.2 2024-08-20T22:39:45.6159824Z numpy==1.21.2 2024-08-20T22:39:45.6160168Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:39:45.6160603Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:39:45.6161050Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:39:45.6161503Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:39:45.6161945Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:39:45.6162343Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:39:45.6162763Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:39:45.6163195Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:39:45.6163635Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:39:45.6164063Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:39:45.6164463Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:39:45.6164881Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:39:45.6165216Z oauthlib==3.2.2 2024-08-20T22:39:45.6165512Z omegaconf==2.3.0 2024-08-20T22:39:45.6165810Z onnx==1.16.1 2024-08-20T22:39:45.6166108Z onnxscript==0.1.0.dev20240817 2024-08-20T22:39:45.6166465Z opacus==1.5.2 2024-08-20T22:39:45.6166804Z opencv-python==4.8.0.74 2024-08-20T22:39:45.6167555Z opt-einsum==3.3.0 2024-08-20T22:39:45.6167852Z optree==0.12.1 2024-08-20T22:39:45.6168146Z packaging==24.1 2024-08-20T22:39:45.6168447Z pandas==2.0.3 2024-08-20T22:39:45.6168734Z patch==1.16 2024-08-20T22:39:45.6169008Z pathspec==0.12.1 2024-08-20T22:39:45.6169331Z phonemizer==3.3.0 2024-08-20T22:39:45.6169660Z pillow==10.3.0 2024-08-20T22:39:45.6169958Z platformdirs==4.2.2 2024-08-20T22:39:45.6170270Z pluggy==1.5.0 2024-08-20T22:39:45.6170555Z pooch==1.8.2 2024-08-20T22:39:45.6170845Z portalocker==2.10.1 2024-08-20T22:39:45.6171153Z preshed==3.0.9 2024-08-20T22:39:45.6171446Z prettytable==3.11.0 2024-08-20T22:39:45.6171756Z protobuf==3.20.2 2024-08-20T22:39:45.6172049Z psutil==6.0.0 2024-08-20T22:39:45.6172375Z py-cpuinfo==9.0.0 2024-08-20T22:39:45.6172676Z pyaml==24.7.0 2024-08-20T22:39:45.6172963Z pyarrow==17.0.0 2024-08-20T22:39:45.6173247Z pyasn1==0.6.0 2024-08-20T22:39:45.6173538Z pyasn1_modules==0.4.0 2024-08-20T22:39:45.6173864Z pyclipper==1.3.0.post5 2024-08-20T22:39:45.6174188Z pycocotools==2.0.8 2024-08-20T22:39:45.6174496Z pycparser==2.22 2024-08-20T22:39:45.6174801Z pycryptodomex==3.20.0 2024-08-20T22:39:45.6175124Z pydantic==2.8.2 2024-08-20T22:39:45.6175418Z pydantic_core==2.20.1 2024-08-20T22:39:45.6175737Z pydot==3.0.1 2024-08-20T22:39:45.6176020Z pygame==2.6.0 2024-08-20T22:39:45.6176302Z PyGithub==2.3.0 2024-08-20T22:39:45.6176764Z Pygments==2.15.0 2024-08-20T22:39:45.6177059Z PyJWT==2.9.0 2024-08-20T22:39:45.6177348Z pylatexenc==2.10 2024-08-20T22:39:45.6177641Z pylint==3.2.6 2024-08-20T22:39:45.6178034Z PyNaCl==1.5.0 2024-08-20T22:39:45.6178323Z pynvml==11.5.3 2024-08-20T22:39:45.6178612Z pyparsing==3.1.2 2024-08-20T22:39:45.6178914Z pypdfium2==4.30.0 2024-08-20T22:39:45.6179213Z pysbd==0.3.4 2024-08-20T22:39:45.6179498Z PySocks==1.7.1 2024-08-20T22:39:45.6179780Z pytest==7.3.2 2024-08-20T22:39:45.6180123Z pytest-benchmark==4.0.0 2024-08-20T22:39:45.6180498Z pytest-cpp==2.3.0 2024-08-20T22:39:45.6180853Z pytest-flakefinder==1.1.0 2024-08-20T22:39:45.6181265Z pytest-rerunfailures==14.0 2024-08-20T22:39:45.6181655Z pytest-xdist==3.3.1 2024-08-20T22:39:45.6182025Z python-dateutil==2.9.0.post0 2024-08-20T22:39:45.6182799Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:39:45.6184372Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:39:45.6185344Z pytz==2024.1 2024-08-20T22:39:45.6185638Z PyWavelets==1.4.1 2024-08-20T22:39:45.6186013Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:39:45.6186462Z rapidfuzz==3.9.6 2024-08-20T22:39:45.6186759Z rdflib==7.0.0 2024-08-20T22:39:45.6187042Z redis==5.0.8 2024-08-20T22:39:45.6187323Z referencing==0.35.1 2024-08-20T22:39:45.6187627Z regex==2024.7.24 2024-08-20T22:39:45.6187922Z requests==2.32.3 2024-08-20T22:39:45.6188271Z requests-oauthlib==2.0.0 2024-08-20T22:39:45.6188610Z resampy==0.4.3 2024-08-20T22:39:45.6188895Z rfc3986==1.5.0 2024-08-20T22:39:45.6189180Z rich==13.7.1 2024-08-20T22:39:45.6189455Z rockset==1.0.3 2024-08-20T22:39:45.6189778Z rpds-py==0.20.0 2024-08-20T22:39:45.6190111Z rsa==4.9 2024-08-20T22:39:45.6190386Z s3transfer==0.5.2 2024-08-20T22:39:45.6190684Z safetensors==0.4.4 2024-08-20T22:39:45.6191030Z scikit-image==0.21.0 2024-08-20T22:39:45.6191386Z scikit-learn==1.5.1 2024-08-20T22:39:45.6191701Z scipy==1.10.1 2024-08-20T22:39:45.6191982Z segments==2.2.1 2024-08-20T22:39:45.6192282Z sentencepiece==0.2.0 2024-08-20T22:39:45.6192637Z sentry-sdk==2.13.0 2024-08-20T22:39:45.6192945Z setproctitle==1.3.3 2024-08-20T22:39:45.6193251Z shapely==2.0.6 2024-08-20T22:39:45.6193546Z shellingham==1.5.4 2024-08-20T22:39:45.6193853Z simplejson==3.19.3 2024-08-20T22:39:45.6194241Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:39:45.6194752Z smart-open==7.0.4 2024-08-20T22:39:45.6195058Z smmap==5.0.1 2024-08-20T22:39:45.6195347Z sortedcontainers==2.4.0 2024-08-20T22:39:45.6195679Z soundfile==0.12.1 2024-08-20T22:39:45.6195979Z soupsieve==2.6 2024-08-20T22:39:45.6196268Z soxr==0.4.0 2024-08-20T22:39:45.6196535Z spacy==3.7.6 2024-08-20T22:39:45.6196861Z spacy-legacy==3.0.12 2024-08-20T22:39:45.6197222Z spacy-loggers==1.0.5 2024-08-20T22:39:45.6197541Z srsly==2.4.8 2024-08-20T22:39:45.6197817Z stempeg==0.2.3 2024-08-20T22:39:45.6198109Z submitit==1.5.1 2024-08-20T22:39:45.6198401Z sympy==1.13.1 2024-08-20T22:39:45.6198688Z tabulate==0.9.0 2024-08-20T22:39:45.6199034Z tb-nightly==2.13.0a20230426 2024-08-20T22:39:45.6199386Z tensorboard==2.13.0 2024-08-20T22:39:45.6199769Z tensorboard-data-server==0.7.2 2024-08-20T22:39:45.6200141Z tensorboardX==2.6.2.2 2024-08-20T22:39:45.6200463Z termcolor==2.4.0 2024-08-20T22:39:45.6200766Z thinc==8.2.5 2024-08-20T22:39:45.6201050Z threadpoolctl==3.5.0 2024-08-20T22:39:45.6201369Z tifffile==2024.8.10 2024-08-20T22:39:45.6202168Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:39:45.6202945Z tlparse==0.3.25 2024-08-20T22:39:45.6203238Z tokenizers==0.15.2 2024-08-20T22:39:45.6203542Z tomli==2.0.1 2024-08-20T22:39:45.6203824Z tomlkit==0.13.2 2024-08-20T22:39:45.6205022Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b444e33ace8c516878aca2862d707788a4c01d8c6190fb13e010fb9a131a57c5 2024-08-20T22:39:45.6206633Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:39:45.6208975Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:39:45.6210010Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:39:45.6210997Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:39:45.6212082Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:39:45.6213167Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:39:45.6214150Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:39:45.6214851Z tornado==6.4.1 2024-08-20T22:39:45.6215137Z tqdm==4.66.5 2024-08-20T22:39:45.6215426Z transformers==4.38.1 2024-08-20T22:39:45.6215745Z treetable==0.2.5 2024-08-20T22:39:45.6216561Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:39:45.6217339Z typeguard==4.3.0 2024-08-20T22:39:45.6217642Z typer==0.12.4 2024-08-20T22:39:45.6217945Z typing_extensions==4.12.2 2024-08-20T22:39:45.6218284Z tzdata==2024.1 2024-08-20T22:39:45.6218568Z Unidecode==1.3.8 2024-08-20T22:39:45.6218933Z unittest-xml-reporting==3.2.0 2024-08-20T22:39:45.6219309Z uritemplate==4.1.1 2024-08-20T22:39:45.6219609Z urllib3==1.26.19 2024-08-20T22:39:45.6219903Z visdom==0.2.4 2024-08-20T22:39:45.6220186Z wandb==0.17.7 2024-08-20T22:39:45.6220467Z wasabi==1.1.3 2024-08-20T22:39:45.6220747Z wcwidth==0.2.13 2024-08-20T22:39:45.6221037Z weasel==0.4.1 2024-08-20T22:39:45.6221375Z websocket-client==1.8.0 2024-08-20T22:39:45.6221705Z Werkzeug==3.0.3 2024-08-20T22:39:45.6221998Z wrapt==1.16.0 2024-08-20T22:39:45.6222282Z xdoctest==1.1.0 2024-08-20T22:39:45.6222574Z xxhash==3.5.0 2024-08-20T22:39:45.6222859Z xyzservices==2024.6.0 2024-08-20T22:39:45.6223181Z yacs==0.1.8 2024-08-20T22:39:45.6223574Z yarl==1.9.4 2024-08-20T22:39:45.6223885Z z3-solver==4.12.2.0 2024-08-20T22:39:45.6224192Z zipp==3.20.0 2024-08-20T22:39:45.6419279Z + popd 2024-08-20T22:39:45.6419601Z ~/workspace 2024-08-20T22:39:45.6419989Z + [[ inductor_torchbench != *cpu* ]] 2024-08-20T22:39:45.6420414Z + install_torchrec_and_fbgemm 2024-08-20T22:39:45.6420784Z + local torchrec_commit 2024-08-20T22:39:45.6422603Z ++ get_pinned_commit torchrec 2024-08-20T22:39:45.6423021Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:39:45.6466497Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:39:45.6467215Z + local fbgemm_commit 2024-08-20T22:39:45.6471194Z ++ get_pinned_commit fbgemm 2024-08-20T22:39:45.6471590Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:39:45.6555043Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:45.6555700Z + pip_uninstall torchrec-nightly 2024-08-20T22:39:45.6556182Z + pip uninstall -y torchrec-nightly 2024-08-20T22:39:46.0299012Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:39:46.0647928Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:39:46.0648475Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:39:46.4357464Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:39:46.4716867Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:46.4717848Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:46.9897482Z Collecting setuptools-git-versioning 2024-08-20T22:39:47.0117296Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:39:47.1198671Z Collecting scikit-build 2024-08-20T22:39:47.1240408Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:39:47.2428582Z Collecting pyre-extensions 2024-08-20T22:39:47.2466630Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:39:47.2945647Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:39:47.2948518Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:39:47.3420388Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:39:47.3452567Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:39:47.4015810Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:39:47.4020498Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:39:47.4024263Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:39:47.4586549Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:39:47.4616648Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:39:47.5029074Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:39:47.5107839Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:39:47.5165989Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:39:47.5662632Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:39:47.6156036Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:39:47.6507077Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:39:47.6806496Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:39:48.5636820Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:39:48.9336877Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:39:49.0448389Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:39:49.0450376Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:49.0452234Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:49.4150706Z Collecting fbgemm-gpu 2024-08-20T22:39:49.4153644Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-k_flsy6n/fbgemm-gpu_c0fe920d20db47ebae5644ae55bb5620 2024-08-20T22:39:49.4182374Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-k_flsy6n/fbgemm-gpu_c0fe920d20db47ebae5644ae55bb5620 2024-08-20T22:39:50.5457436Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:39:50.5485954Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:51.0551250Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:51.5707601Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:51.5708849Z Running command git submodule update --init --recursive -q 2024-08-20T22:40:01.2155378Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:40:01.2174912Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:40:01.2183897Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:25:18.6594859Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T23:25:18.8433070Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252619491 sha256=05bb37413b0024d1bca080607f5fc76bef2346c1ccf498d7d4fded3d611e9446 2024-08-20T23:25:18.8436053Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:25:18.8454892Z Successfully built fbgemm-gpu 2024-08-20T23:25:19.6936039Z Installing collected packages: fbgemm-gpu 2024-08-20T23:25:23.1080544Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:25:24.0218951Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.0220942Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.3927110Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:24.3931638Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-rk_7uopv 2024-08-20T23:25:24.3960755Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-rk_7uopv 2024-08-20T23:25:25.1631397Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:25:25.1660516Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.4694058Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:25.8429559Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:25:26.1159641Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:25:26.1194325Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:25:26.1197715Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:25:26.1200674Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:25:26.1731503Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:25:26.1909429Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:25:26.2436284Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:25:26.2438433Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:25:26.2447585Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:25:26.2456734Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:25:26.2720748Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:25:26.2724177Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:26.2727657Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:25:26.2749590Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:25:26.2753107Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:25:26.2908904Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:25:26.2936370Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:25:26.3027304Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:25:26.3030115Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:25:26.3049893Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:25:26.3083286Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:25:26.3086814Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:25:26.3090499Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:25:26.3093054Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:25:26.3098107Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:25:26.3113302Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:25:26.3522426Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:25:26.3690417Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:25:26.3877200Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:25:26.3979570Z Building wheels for collected packages: torchrec 2024-08-20T23:25:26.9174001Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:25:26.9182571Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=54aa4693829b64164ebed65025aa7f5275e373b4f41f0dc94cee8e6306459088 2024-08-20T23:25:26.9184199Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:25:26.9206235Z Successfully built torchrec 2024-08-20T23:25:27.7014108Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:25:28.3150630Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:25:28.4559589Z ++ pwd 2024-08-20T23:25:28.4562706Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:25:28.4563229Z + test_dynamo_benchmark torchbench 0 2024-08-20T23:25:28.4568380Z ++ pwd 2024-08-20T23:25:28.4572382Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4573039Z + local suite=torchbench 2024-08-20T23:25:28.4573384Z + shift 2024-08-20T23:25:28.4573672Z + local shard_id=0 2024-08-20T23:25:28.4576467Z + shift 2024-08-20T23:25:28.4576954Z + [[ inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:28.4577578Z + [[ inductor_torchbench == *perf* ]] 2024-08-20T23:25:28.4578001Z + [[ inductor_torchbench == *cpu* ]] 2024-08-20T23:25:28.4579057Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:28.4579869Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-08-20T23:25:28.4580539Z ++ pwd 2024-08-20T23:25:28.4581019Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4581694Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:25:28.4621470Z + local name=inference 2024-08-20T23:25:28.4621967Z + shift 2024-08-20T23:25:28.4622332Z + local suite=torchbench 2024-08-20T23:25:28.4622684Z + shift 2024-08-20T23:25:28.4622967Z + local shard_id=0 2024-08-20T23:25:28.4623491Z + shift 2024-08-20T23:25:28.4623875Z + partition_flags=() 2024-08-20T23:25:28.4624255Z + local partition_flags 2024-08-20T23:25:28.4624750Z + [[ -n 2 ]] 2024-08-20T23:25:28.4625194Z + [[ -n 0 ]] 2024-08-20T23:25:28.4626006Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:25:28.4626711Z + [[ inductor_torchbench == *perf_compare* ]] 2024-08-20T23:25:28.4627180Z + [[ inductor_torchbench == *perf* ]] 2024-08-20T23:25:28.4627629Z + [[ inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:25:28.4628091Z + [[ inductor_torchbench == *_avx2* ]] 2024-08-20T23:25:28.4628528Z + [[ inductor_torchbench == *_avx512* ]] 2024-08-20T23:25:28.4630113Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:25:34.4248494Z 2024-08-20T23:25:35.2017020Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-08-20T23:25:35.2019021Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-08-20T23:25:35.5965924Z 2024-08-20T23:25:35.5966612Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:25:35.5967735Z cuda eval torchrec_dlrm 2024-08-20T23:25:35.6000647Z Traceback (most recent call last): 2024-08-20T23:25:35.6001559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:25:35.6002567Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:25:35.6003536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:25:35.6004251Z return mod(*inputs) 2024-08-20T23:25:35.6005273Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:35.6006768Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:35.6007968Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:35.6008814Z return forward_call(*args, **kwargs) 2024-08-20T23:25:35.6009693Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-08-20T23:25:35.6010615Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-08-20T23:25:35.6011719Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:35.6012604Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:35.6013532Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:35.6014362Z return forward_call(*args, **kwargs) 2024-08-20T23:25:35.6015243Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-08-20T23:25:35.6016037Z concatenated_dense = self.inter_arch( 2024-08-20T23:25:35.6017029Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:35.6017908Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:35.6018845Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:35.6019672Z return forward_call(*args, **kwargs) 2024-08-20T23:25:35.6020540Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-08-20T23:25:35.6021455Z return self.crossnet(combined_values.reshape([B, -1])) 2024-08-20T23:25:35.6022512Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:25:35.6023520Z return self._call_impl(*args, **kwargs) 2024-08-20T23:25:35.6024458Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:25:35.6025276Z return forward_call(*args, **kwargs) 2024-08-20T23:25:35.6026183Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-08-20T23:25:35.6027106Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-08-20T23:25:35.6027945Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-08-20T23:25:35.6028494Z 2024-08-20T23:25:35.6028788Z The above exception was the direct cause of the following exception: 2024-08-20T23:25:35.6029234Z 2024-08-20T23:25:35.6029379Z Traceback (most recent call last): 2024-08-20T23:25:35.6030029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:25:35.6030680Z ) = runner.load_model( 2024-08-20T23:25:35.6031344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:25:35.6032095Z self.validate_model(model, example_inputs) 2024-08-20T23:25:35.6032852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:25:35.6033609Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:25:35.6034276Z RuntimeError: Eager run failed 2024-08-20T23:25:35.6034532Z 2024-08-20T23:25:35.6034652Z eager_fail_to_run 2024-08-20T23:25:39.3002164Z 2024-08-20T23:25:41.5107009Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:41.5107723Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:25:41.5166705Z cuda eval BERT_pytorch 2024-08-20T23:26:03.1262565Z pass 2024-08-20T23:26:03.1295711Z TIMING: code_gen:5.03062 inductor_compile:9.79884 backend_compile:15.48627 entire_frame_compile:19.62648 total_wall_time:19.62648 2024-08-20T23:26:03.1298591Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:2319 | FakeTensorMode.__torch_dispatch__:28098 | ProxyTorchDispatchMode.__torch_dispatch__:12905 2024-08-20T23:26:03.1300192Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:07.4706438Z 2024-08-20T23:26:10.4368686Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:10.4369751Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:26:10.4414495Z cuda eval Background_Matting 2024-08-20T23:26:10.4417849Z pass_due_to_skip 2024-08-20T23:26:10.4946308Z TIMING: 2024-08-20T23:26:10.4946638Z STATS: call_* op count: 0 2024-08-20T23:26:10.4950334Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:14.1410111Z 2024-08-20T23:26:17.1029683Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:17.1030162Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:26:17.1056862Z cuda eval LearningToPaint 2024-08-20T23:26:26.3028944Z pass 2024-08-20T23:26:26.3039695Z TIMING: code_gen:1.90257 inductor_compile:3.18704 backend_compile:5.76833 entire_frame_compile:7.68051 total_wall_time:7.68051 2024-08-20T23:26:26.3041126Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:8401 | ProxyTorchDispatchMode.__torch_dispatch__:2930 | FakeTensor.__torch_dispatch__:1094 2024-08-20T23:26:26.3042243Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:30.5378267Z 2024-08-20T23:26:31.9756066Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-08-20T23:26:31.9931723Z 2024-08-20T23:26:31.9931915Z 2024-08-20T23:26:32.0935241Z 0% 0.00/528M [00:00 will be ignored 2024-08-20T23:59:55.4967516Z pass 2024-08-20T23:59:55.5227303Z TIMING: entire_frame_compile:55.72439 code_gen:38.14069 inductor_compile:51.48219 backend_compile:46.20758 total_wall_time:73.2976 2024-08-20T23:59:55.5228691Z STATS: call_* op count: 671 | FakeTensorMode.__torch_dispatch__:38758 | ProxyTorchDispatchMode.__torch_dispatch__:17204 | FakeTensor.__torch_dispatch__:10139 2024-08-20T23:59:55.5229804Z Dynamo produced 3 graphs covering 671 ops with 7 graph breaks (5 unique) 2024-08-21T00:00:01.9240289Z 2024-08-21T00:00:03.4655942Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:03.4656580Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:00:03.4657107Z cuda train alexnet 2024-08-21T00:00:22.5374655Z pass 2024-08-21T00:00:22.5566648Z TIMING: entire_frame_compile:5.55857 code_gen:9.06969 inductor_compile:9.85181 backend_compile:4.98328 total_wall_time:13.62153 2024-08-21T00:00:22.5569838Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:2730 | ProxyTorchDispatchMode.__torch_dispatch__:950 | FakeTensor.__torch_dispatch__:340 2024-08-21T00:00:22.5571623Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-08-21T00:00:26.7152232Z 2024-08-21T00:00:28.6781583Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:28.6782146Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:00:28.6784683Z cuda train basic_gnn_edgecnn 2024-08-21T00:00:32.1786003Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:00:34.2484489Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:00:34.2485751Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 190, in propagate 2024-08-21T00:00:34.2486585Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-08-21T00:00:34.2487375Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 128, in _collect 2024-08-21T00:00:34.2488039Z x_i = self._lift(x_i, edge_def, i) 2024-08-21T00:00:34.2488567Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 88, in _lift 2024-08-21T00:00:34.2489125Z return src.index_select(self.node_dim, index) 2024-08-21T00:00:34.2489461Z 2024-08-21T00:00:34.2626799Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:00:34.2627763Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 190, in propagate 2024-08-21T00:00:34.2628429Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-08-21T00:00:34.2629051Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 128, in _collect 2024-08-21T00:00:34.2629592Z x_i = self._lift(x_i, edge_def, i) 2024-08-21T00:00:34.2630125Z File "/tmp/jenkins_pyg/tmp92p5563g.py", line 88, in _lift 2024-08-21T00:00:34.2630685Z return src.index_select(self.node_dim, index) 2024-08-21T00:00:34.2631018Z 2024-08-21T00:00:34.3896174Z W0821 00:00:34.388000 46611 torch/_logging/_internal.py:1058] [24/0] Profiler function will be ignored 2024-08-21T00:00:39.5994487Z pass 2024-08-21T00:00:39.6004948Z TIMING: entire_frame_compile:7.90942 inductor_compile:5.53808 backend_compile:6.03422 code_gen:4.14114 total_wall_time:8.71258 2024-08-21T00:00:39.6006454Z STATS: call_* op count: 102 | FakeTensorMode.__torch_dispatch__:4593 | ProxyTorchDispatchMode.__torch_dispatch__:1787 | FakeTensor.__torch_dispatch__:987 2024-08-21T00:00:39.6007564Z Dynamo produced 8 graphs covering 102 ops with 20 graph breaks (6 unique) 2024-08-21T00:00:43.7548640Z 2024-08-21T00:00:45.7914087Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:45.7914607Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:00:45.7915048Z cuda train basic_gnn_gcn 2024-08-21T00:00:48.5785626Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:00:49.1570850Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:00:49.6426873Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:00:49.8749944Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:00:50.3998414Z W0821 00:00:50.398000 46826 torch/_logging/_internal.py:1058] [18/0] Profiler function will be ignored 2024-08-21T00:00:53.8841066Z pass 2024-08-21T00:00:53.8846270Z TIMING: entire_frame_compile:5.60438 inductor_compile:4.10133 backend_compile:4.61117 code_gen:3.01062 total_wall_time:6.01371 2024-08-21T00:00:53.8847673Z STATS: call_* op count: 98 | FakeTensorMode.__torch_dispatch__:2799 | FakeTensor.__torch_dispatch__:538 | ProxyTorchDispatchMode.__torch_dispatch__:1004 2024-08-21T00:00:53.8848793Z Dynamo produced 10 graphs covering 98 ops with 13 graph breaks (6 unique) 2024-08-21T00:00:57.9147690Z 2024-08-21T00:00:59.8654638Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:00:59.8655172Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:00:59.8655651Z cuda train basic_gnn_gin 2024-08-21T00:01:02.9963071Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:01:03.9316478Z W0821 00:01:03.930000 47048 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:01:08.1400249Z pass 2024-08-21T00:01:08.1428829Z TIMING: entire_frame_compile:6.49735 code_gen:3.19574 inductor_compile:4.79961 backend_compile:5.27475 total_wall_time:7.22649 2024-08-21T00:01:08.1430341Z STATS: call_* op count: 109 | FakeTensorMode.__torch_dispatch__:4058 | ProxyTorchDispatchMode.__torch_dispatch__:1747 | FakeTensor.__torch_dispatch__:954 2024-08-21T00:01:08.1431500Z Dynamo produced 3 graphs covering 109 ops with 7 graph breaks (5 unique) 2024-08-21T00:01:12.2180633Z 2024-08-21T00:01:14.1361136Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:14.1362168Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:01:14.1362605Z cuda train basic_gnn_sage 2024-08-21T00:01:16.9310457Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:01:17.7456901Z W0821 00:01:17.744000 47233 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:01:21.2727818Z pass 2024-08-21T00:01:21.2754192Z TIMING: entire_frame_compile:5.48565 code_gen:2.70529 inductor_compile:4.11282 backend_compile:4.48307 total_wall_time:6.13146 2024-08-21T00:01:21.2755580Z STATS: call_* op count: 106 | FakeTensorMode.__torch_dispatch__:3437 | ProxyTorchDispatchMode.__torch_dispatch__:1396 | FakeTensor.__torch_dispatch__:731 2024-08-21T00:01:21.2756679Z Dynamo produced 3 graphs covering 106 ops with 7 graph breaks (5 unique) 2024-08-21T00:01:25.3218946Z 2024-08-21T00:01:25.9152606Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:25.9153119Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:25.9153580Z cuda train dcgan 2024-08-21T00:01:31.0674227Z pass 2024-08-21T00:01:31.0691709Z TIMING: entire_frame_compile:2.75231 code_gen:1.20054 inductor_compile:2.21563 backend_compile:2.16612 total_wall_time:3.61394 2024-08-21T00:01:31.0693063Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:2164 | FakeTensor.__torch_dispatch__:447 | ProxyTorchDispatchMode.__torch_dispatch__:830 2024-08-21T00:01:31.0694149Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-08-21T00:01:35.0199857Z 2024-08-21T00:01:38.7070421Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:01:38.7071150Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:01:38.7071862Z cuda train demucs 2024-08-21T00:02:48.2865151Z pass 2024-08-21T00:02:48.3116808Z TIMING: entire_frame_compile:7.42311 inductor_compile:9.45388 backend_compile:5.40999 code_gen:6.97677 total_wall_time:12.93924 2024-08-21T00:02:48.3118901Z STATS: call_* op count: 88 | FakeTensorMode.__torch_dispatch__:6572 | ProxyTorchDispatchMode.__torch_dispatch__:2363 | FakeTensor.__torch_dispatch__:1250 2024-08-21T00:02:48.3120587Z Dynamo produced 6 graphs covering 88 ops with 9 graph breaks (6 unique) 2024-08-21T00:02:52.6158537Z 2024-08-21T00:02:54.9045945Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:02:54.9046435Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:02:54.9046883Z cuda train densenet121 2024-08-21T00:04:53.9027948Z pass 2024-08-21T00:04:53.9104217Z TIMING: entire_frame_compile:56.08211 code_gen:55.16435 inductor_compile:72.74769 backend_compile:40.50347 total_wall_time:109.21629 2024-08-21T00:04:53.9107918Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:66535 | ProxyTorchDispatchMode.__torch_dispatch__:30351 | FakeTensor.__torch_dispatch__:14947 2024-08-21T00:04:53.9109064Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-08-21T00:05:01.4768327Z 2024-08-21T00:05:06.6192580Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:05:06.6193076Z loading model: 0it [00:05, ?it/s] 2024-08-21T00:05:06.6193526Z cuda train detectron2_maskrcnn_r_50_c4 2024-08-21T00:05:06.6209052Z Traceback (most recent call last): 2024-08-21T00:05:06.6209885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:05:06.6210659Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:05:06.6213181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-21T00:05:06.6214123Z pred = mod(*cloned_inputs) 2024-08-21T00:05:06.6215347Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:05:06.6216320Z return self._call_impl(*args, **kwargs) 2024-08-21T00:05:06.6217380Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:05:06.6219815Z return forward_call(*args, **kwargs) 2024-08-21T00:05:06.6221177Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-21T00:05:06.6222141Z return self.inference(batched_inputs) 2024-08-21T00:05:06.6223466Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-08-21T00:05:06.6224584Z results, _ = self.roi_heads(images, features, proposals, None) 2024-08-21T00:05:06.6225769Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:05:06.6226730Z return self._call_impl(*args, **kwargs) 2024-08-21T00:05:06.6227748Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:05:06.6228647Z return forward_call(*args, **kwargs) 2024-08-21T00:05:06.6229772Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-08-21T00:05:06.6230759Z box_features = self._shared_roi_transform( 2024-08-21T00:05:06.6231933Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-08-21T00:05:06.6232966Z x = self.pooler(features, boxes) 2024-08-21T00:05:06.6234068Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:05:06.6235027Z return self._call_impl(*args, **kwargs) 2024-08-21T00:05:06.6236042Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:05:06.6236939Z return forward_call(*args, **kwargs) 2024-08-21T00:05:06.6237937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-08-21T00:05:06.6238895Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-08-21T00:05:06.6240092Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:05:06.6241045Z return self._call_impl(*args, **kwargs) 2024-08-21T00:05:06.6242055Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:05:06.6242948Z return forward_call(*args, **kwargs) 2024-08-21T00:05:06.6243937Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-08-21T00:05:06.6244786Z return roi_align( 2024-08-21T00:05:06.6245698Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:05:06.6246926Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:05:06.6248209Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:05:06.6249414Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:05:06.6250756Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:05:06.6251631Z v1 = masked_index(y_low, x_low) 2024-08-21T00:05:06.6252606Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:05:06.6253405Z return input[ 2024-08-21T00:05:06.6256552Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 106241 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-21T00:05:06.6259541Z 2024-08-21T00:05:06.6259849Z The above exception was the direct cause of the following exception: 2024-08-21T00:05:06.6260296Z 2024-08-21T00:05:06.6260450Z Traceback (most recent call last): 2024-08-21T00:05:06.6261087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:05:06.6261747Z ) = runner.load_model( 2024-08-21T00:05:06.6262402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:05:06.6263299Z self.validate_model(model, example_inputs) 2024-08-21T00:05:06.6264046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:05:06.6264807Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:05:06.6265347Z RuntimeError: Eager run failed 2024-08-21T00:05:06.6265609Z 2024-08-21T00:05:06.6265731Z eager_fail_to_run 2024-08-21T00:05:10.4120768Z 2024-08-21T00:05:19.3862208Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:05:19.3862726Z loading model: 0it [00:08, ?it/s] 2024-08-21T00:05:19.3863338Z cuda train dlrm 2024-08-21T00:05:28.0887347Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2024-08-21T00:05:28.0888780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:05:28.0889766Z pred = mod(*cloned_inputs) 2024-08-21T00:05:28.0890613Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 355, in forward 2024-08-21T00:05:28.0891567Z return self.sequential_forward(dense_x, lS_o, lS_i) 2024-08-21T00:05:28.0892583Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 367, in sequential_forward 2024-08-21T00:05:28.0893605Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2024-08-21T00:05:28.0894541Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 308, in apply_emb 2024-08-21T00:05:28.0895507Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2024-08-21T00:05:28.0895899Z 2024-08-21T00:05:28.5973987Z pass 2024-08-21T00:05:28.6373240Z TIMING: entire_frame_compile:6.456 code_gen:1.21226 inductor_compile:2.01063 backend_compile:5.71574 total_wall_time:7.38196 2024-08-21T00:05:28.6374615Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:3620 | ProxyTorchDispatchMode.__torch_dispatch__:1379 | FakeTensor.__torch_dispatch__:387 2024-08-21T00:05:28.6375720Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-08-21T00:05:32.6514646Z 2024-08-21T00:05:34.0752601Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:05:34.0753078Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:05:34.0753564Z cuda train drq 2024-08-21T00:05:41.9051105Z W0821 00:05:41.904000 50382 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:05:49.8745771Z pass 2024-08-21T00:05:49.8774916Z TIMING: entire_frame_compile:13.37455 code_gen:6.30086 inductor_compile:7.80821 backend_compile:11.26015 total_wall_time:14.53902 2024-08-21T00:05:49.8776294Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:6973 | ProxyTorchDispatchMode.__torch_dispatch__:3284 | FakeTensor.__torch_dispatch__:1487 2024-08-21T00:05:49.8777395Z Dynamo produced 5 graphs covering 136 ops with 7 graph breaks (5 unique) 2024-08-21T00:05:54.1737590Z 2024-08-21T00:05:56.0417656Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:05:56.0418243Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:05:56.0421243Z cuda train fastNLP_Bert 2024-08-21T00:06:02.5913340Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-21T00:06:02.5915370Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-21T00:06:02.5916380Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-21T00:06:02.5917348Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-21T00:06:02.5918593Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-21T00:06:02.5919824Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:06:02.5921042Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-21T00:06:02.5923065Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:06:02.5924674Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(*cloned_inputs) 2024-08-21T00:06:02.5926430Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward 2024-08-21T00:06:02.5927857Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] sequence_output = self.bert(words) 2024-08-21T00:06:02.5929518Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward 2024-08-21T00:06:02.5930996Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.model(words) 2024-08-21T00:06:02.5932622Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward 2024-08-21T00:06:02.5934678Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item() # 表示word piece的长度(包括padding) 2024-08-21T00:06:02.5937882Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:06:02.5938607Z W0821 00:06:02.590000 50687 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:06:38.8301945Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:06:38.8304854Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2024-08-21T00:06:38.8305861Z embedding_output = self.embeddings(input_ids, token_type_ids) 2024-08-21T00:06:38.8306934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2024-08-21T00:06:38.8307877Z words_embeddings = self.word_embeddings(input_ids) 2024-08-21T00:06:38.8308241Z 2024-08-21T00:06:41.8306152Z pass 2024-08-21T00:06:41.8954333Z TIMING: entire_frame_compile:25.23275 inductor_compile:21.09052 backend_compile:21.25184 code_gen:13.27494 total_wall_time:36.02305 2024-08-21T00:06:41.8956022Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:38528 | ProxyTorchDispatchMode.__torch_dispatch__:19109 | FakeTensor.__torch_dispatch__:5968 2024-08-21T00:06:41.8957392Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2024-08-21T00:06:47.1681487Z 2024-08-21T00:06:48.0878492Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:06:48.0879011Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:06:48.0879457Z cuda train functorch_dp_cifar10 2024-08-21T00:07:11.0227443Z W0821 00:07:11.021000 51065 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:07:28.9206947Z pass 2024-08-21T00:07:28.9268143Z TIMING: entire_frame_compile:27.02097 code_gen:17.69586 inductor_compile:25.16827 backend_compile:22.37617 total_wall_time:37.75286 2024-08-21T00:07:28.9270588Z STATS: call_* op count: 276 | FakeTensorMode.__torch_dispatch__:18125 | ProxyTorchDispatchMode.__torch_dispatch__:7938 | FakeTensor.__torch_dispatch__:6779 2024-08-21T00:07:28.9271752Z Dynamo produced 3 graphs covering 276 ops with 7 graph breaks (5 unique) 2024-08-21T00:07:34.1413581Z 2024-08-21T00:07:34.8053325Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:07:34.8053822Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:07:34.8054279Z cuda train functorch_maml_omniglot 2024-08-21T00:07:42.8592106Z W0821 00:07:42.858000 51570 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:07:48.5671514Z pass 2024-08-21T00:07:48.5672548Z TIMING: entire_frame_compile:8.84581 code_gen:6.68432 inductor_compile:8.03665 backend_compile:7.53786 total_wall_time:11.91089 2024-08-21T00:07:48.5674075Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:3923 | ProxyTorchDispatchMode.__torch_dispatch__:1433 | FakeTensor.__torch_dispatch__:1195 2024-08-21T00:07:48.5675186Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-21T00:07:52.7387025Z 2024-08-21T00:07:54.1183598Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:07:54.1184216Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:07:54.1184684Z cuda train hf_Albert 2024-08-21T00:08:28.4462812Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:08:28.4465946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:08:28.4467537Z pred = mod(*cloned_inputs) 2024-08-21T00:08:28.4469158Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-08-21T00:08:28.4470296Z outputs = self.albert( 2024-08-21T00:08:28.4471280Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-08-21T00:08:28.4472198Z embedding_output = self.embeddings( 2024-08-21T00:08:28.4473211Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-08-21T00:08:28.4474157Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:08:28.4474500Z 2024-08-21T00:08:28.5683797Z W0821 00:08:28.567000 51857 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:08:37.9679270Z pass 2024-08-21T00:08:37.9837641Z TIMING: entire_frame_compile:24.65752 code_gen:13.69077 inductor_compile:21.93989 backend_compile:20.9827 total_wall_time:35.10447 2024-08-21T00:08:37.9839041Z STATS: call_* op count: 541 | FakeTensorMode.__torch_dispatch__:44382 | ProxyTorchDispatchMode.__torch_dispatch__:21516 | FakeTensor.__torch_dispatch__:7422 2024-08-21T00:08:37.9840205Z Dynamo produced 3 graphs covering 541 ops with 6 graph breaks (5 unique) 2024-08-21T00:08:43.2204860Z 2024-08-21T00:08:47.2506770Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:08:47.2507344Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:08:47.2507858Z cuda train hf_Bart 2024-08-21T00:09:33.8616563Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:09:33.8617763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:09:33.8618728Z pred = mod(*cloned_inputs) 2024-08-21T00:09:33.8619681Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-08-21T00:09:33.8620812Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-08-21T00:09:33.8622505Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2024-08-21T00:09:33.8625855Z outputs = self.model( 2024-08-21T00:09:33.8627057Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2024-08-21T00:09:33.8627970Z encoder_outputs = self.encoder( 2024-08-21T00:09:33.8628969Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2024-08-21T00:09:33.8629978Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-08-21T00:09:33.8630402Z 2024-08-21T00:09:33.9996566Z W0821 00:09:33.998000 52222 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:10:31.5556831Z pass 2024-08-21T00:10:31.6019017Z TIMING: entire_frame_compile:78.98708 code_gen:32.01236 inductor_compile:50.31914 backend_compile:62.58414 total_wall_time:90.30086 2024-08-21T00:10:31.6020460Z STATS: call_* op count: 1249 | FakeTensorMode.__torch_dispatch__:70770 | FakeTensor.__torch_dispatch__:20599 | ProxyTorchDispatchMode.__torch_dispatch__:32344 2024-08-21T00:10:31.6026061Z Dynamo produced 3 graphs covering 1249 ops with 6 graph breaks (5 unique) 2024-08-21T00:10:39.9136048Z 2024-08-21T00:10:42.5687893Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:10:42.5688378Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:10:42.5688810Z cuda train hf_Bert 2024-08-21T00:11:21.1069046Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:11:21.1070223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:11:21.1071195Z pred = mod(*cloned_inputs) 2024-08-21T00:11:21.1076651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-21T00:11:21.1077707Z outputs = self.bert( 2024-08-21T00:11:21.1078658Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-21T00:11:21.1079561Z embedding_output = self.embeddings( 2024-08-21T00:11:21.1080557Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-21T00:11:21.1081484Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:11:21.1081841Z 2024-08-21T00:11:21.2463633Z W0821 00:11:21.245000 52554 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:12:05.7954851Z pass 2024-08-21T00:12:05.8376037Z TIMING: entire_frame_compile:63.18235 code_gen:23.68992 inductor_compile:38.8816 backend_compile:49.98765 total_wall_time:72.14269 2024-08-21T00:12:05.8378830Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:61875 | ProxyTorchDispatchMode.__torch_dispatch__:27774 | FakeTensor.__torch_dispatch__:15919 2024-08-21T00:12:05.8380035Z Dynamo produced 3 graphs covering 994 ops with 6 graph breaks (5 unique) 2024-08-21T00:12:13.0703840Z 2024-08-21T00:12:17.9519920Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:17.9522821Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:12:17.9523547Z cuda train hf_Bert_large 2024-08-21T00:13:35.5172855Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:13:35.5176534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:13:35.5177532Z pred = mod(*cloned_inputs) 2024-08-21T00:13:35.5178729Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-21T00:13:35.5179612Z outputs = self.bert( 2024-08-21T00:13:35.5181203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-21T00:13:35.5182500Z embedding_output = self.embeddings( 2024-08-21T00:13:35.5183758Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-21T00:13:35.5184691Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:13:35.5185041Z 2024-08-21T00:13:35.7197034Z W0821 00:13:35.718000 52778 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:15:06.1957760Z pass 2024-08-21T00:15:06.3099847Z TIMING: entire_frame_compile:125.60555 code_gen:48.43778 inductor_compile:77.41773 backend_compile:98.8844 total_wall_time:142.71296 2024-08-21T00:15:06.3103814Z STATS: call_* op count: 1918 | FakeTensorMode.__torch_dispatch__:120744 | ProxyTorchDispatchMode.__torch_dispatch__:54414 | FakeTensor.__torch_dispatch__:30991 2024-08-21T00:15:06.3104993Z Dynamo produced 3 graphs covering 1918 ops with 6 graph breaks (5 unique) 2024-08-21T00:15:16.8057616Z 2024-08-21T00:15:20.4380993Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:15:20.4381473Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:15:20.4385204Z cuda train hf_BigBird 2024-08-21T00:15:31.7232584Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-08-21T00:16:27.2236453Z W0821 00:16:27.222000 53127 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:16:59.6127893Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_1). Found from : 2024-08-21T00:16:59.6129177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:16:59.6130126Z pred = mod(*cloned_inputs) 2024-08-21T00:16:59.6131378Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2024-08-21T00:16:59.6132313Z outputs = self.bert( 2024-08-21T00:16:59.6133290Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2024-08-21T00:16:59.6134221Z encoder_outputs = self.encoder( 2024-08-21T00:16:59.6135228Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2024-08-21T00:16:59.6136150Z layer_outputs = layer_module( 2024-08-21T00:16:59.6137147Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2024-08-21T00:16:59.6138095Z self_attention_outputs = self.attention( 2024-08-21T00:16:59.6139152Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2024-08-21T00:16:59.6140052Z self_outputs = self.self( 2024-08-21T00:16:59.6141043Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2024-08-21T00:16:59.6142105Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2024-08-21T00:16:59.6143665Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention 2024-08-21T00:16:59.6144708Z rand_attn = np.stack(rand_attn, axis=0) 2024-08-21T00:16:59.6145025Z 2024-08-21T00:17:38.4028610Z W0821 00:17:38.401000 53127 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:18:23.5070387Z pass 2024-08-21T00:18:23.5364072Z TIMING: entire_frame_compile:126.75412 code_gen:59.30563 inductor_compile:97.59101 backend_compile:104.91693 total_wall_time:164.03743 2024-08-21T00:18:23.5365619Z STATS: call_* op count: 3974 | FakeTensorMode.__torch_dispatch__:192095 | ProxyTorchDispatchMode.__torch_dispatch__:84045 | FakeTensor.__torch_dispatch__:25953 2024-08-21T00:18:23.5367712Z Dynamo produced 3 graphs covering 3974 ops with 6 graph breaks (5 unique) 2024-08-21T00:18:34.0578304Z 2024-08-21T00:18:36.1978190Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:36.1978678Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:18:36.1979123Z cuda train hf_DistilBert 2024-08-21T00:18:59.4546533Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:18:59.4549054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:18:59.4550104Z pred = mod(*cloned_inputs) 2024-08-21T00:18:59.4551667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-08-21T00:18:59.4552632Z dlbrt_output = self.distilbert( 2024-08-21T00:18:59.4553743Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-21T00:18:59.4554908Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-21T00:18:59.4556146Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-21T00:18:59.4557259Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-21T00:18:59.4557720Z 2024-08-21T00:18:59.5677625Z W0821 00:18:59.566000 53644 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:19:24.4884595Z pass 2024-08-21T00:19:24.5136399Z TIMING: entire_frame_compile:35.45148 code_gen:15.4769 inductor_compile:24.41258 backend_compile:28.53896 total_wall_time:41.32413 2024-08-21T00:19:24.5139277Z STATS: call_* op count: 539 | FakeTensorMode.__torch_dispatch__:32363 | ProxyTorchDispatchMode.__torch_dispatch__:14156 | FakeTensor.__torch_dispatch__:8621 2024-08-21T00:19:24.5140477Z Dynamo produced 3 graphs covering 539 ops with 6 graph breaks (5 unique) 2024-08-21T00:19:30.1441597Z 2024-08-21T00:19:33.4737460Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:33.4738164Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:19:33.4738839Z cuda train hf_GPT2 2024-08-21T00:20:09.0244373Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:20:09.0245717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:20:09.0246672Z pred = mod(*cloned_inputs) 2024-08-21T00:20:09.0248201Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-08-21T00:20:09.0249140Z transformer_outputs = self.transformer( 2024-08-21T00:20:09.0250212Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-21T00:20:09.0251347Z inputs_embeds = self.wte(input_ids) 2024-08-21T00:20:09.0251684Z 2024-08-21T00:20:09.1379352Z W0821 00:20:09.137000 53891 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:20:43.4196576Z pass 2024-08-21T00:20:43.4972818Z TIMING: entire_frame_compile:52.40168 code_gen:23.01439 inductor_compile:35.83555 backend_compile:42.0381 total_wall_time:62.12309 2024-08-21T00:20:43.4975874Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:49418 | ProxyTorchDispatchMode.__torch_dispatch__:21916 | FakeTensor.__torch_dispatch__:13226 2024-08-21T00:20:43.4978141Z Dynamo produced 3 graphs covering 1087 ops with 6 graph breaks (5 unique) 2024-08-21T00:20:50.1047062Z 2024-08-21T00:21:03.6299276Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:21:03.6302351Z loading model: 0it [00:13, ?it/s] 2024-08-21T00:21:03.6302954Z cuda train hf_GPT2_large 2024-08-21T00:21:03.6303520Z pass_due_to_skip 2024-08-21T00:21:04.1332770Z TIMING: 2024-08-21T00:21:04.1333119Z STATS: call_* op count: 0 2024-08-21T00:21:04.1337574Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:21:07.7715969Z 2024-08-21T00:21:09.4517951Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:21:09.4518424Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:21:09.4518884Z cuda train hf_Reformer 2024-08-21T00:21:28.1254923Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:21:30.9966758Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:21:32.4799246Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:21:35.0171500Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:21:35.0176090Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2024-08-21T00:21:35.0177365Z embedding_output = self.embeddings( 2024-08-21T00:21:35.0178526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2024-08-21T00:21:35.0179507Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:21:35.0179868Z 2024-08-21T00:21:35.0256008Z W0821 00:21:35.024000 54310 torch/_logging/_internal.py:1058] [32/0] Profiler function will be ignored 2024-08-21T00:22:00.2722524Z pass 2024-08-21T00:22:00.2845038Z TIMING: entire_frame_compile:33.7809 inductor_compile:21.43118 backend_compile:26.57572 code_gen:15.71065 total_wall_time:35.14219 2024-08-21T00:22:00.2847849Z STATS: call_* op count: 687 | FakeTensorMode.__torch_dispatch__:22309 | ProxyTorchDispatchMode.__torch_dispatch__:8320 | FakeTensor.__torch_dispatch__:4848 | attempt fast:23 | fast is_contiguous:23 2024-08-21T00:22:00.2849193Z Dynamo produced 17 graphs covering 687 ops with 25 graph breaks (9 unique) 2024-08-21T00:22:05.3942416Z 2024-08-21T00:22:09.7369244Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:09.7370140Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:22:09.7374951Z cuda train hf_T5_base 2024-08-21T00:22:12.8801347Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-08-21T00:22:14.9893104Z ERROR:common: 2024-08-21T00:22:14.9893520Z Traceback (most recent call last): 2024-08-21T00:22:14.9894273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2753, in check_accuracy 2024-08-21T00:22:14.9895048Z correct_rerun_result = self.run_n_iterations( 2024-08-21T00:22:14.9895830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-21T00:22:14.9897165Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-21T00:22:14.9898111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-21T00:22:14.9898964Z self.grad_scaler.scale(loss).backward() 2024-08-21T00:22:14.9900039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-21T00:22:14.9900782Z torch.autograd.backward( 2024-08-21T00:22:14.9901651Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-21T00:22:14.9902435Z _engine_run_backward( 2024-08-21T00:22:14.9903636Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-21T00:22:14.9904793Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-21T00:22:14.9908588Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 111514 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-21T00:22:15.0758501Z eager_2nd_run_OOM 2024-08-21T00:22:15.0793749Z TIMING: 2024-08-21T00:22:15.0794344Z STATS: call_* op count: 0 2024-08-21T00:22:15.0795484Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:22:18.7655118Z 2024-08-21T00:22:29.1429831Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:29.1430386Z loading model: 0it [00:10, ?it/s] 2024-08-21T00:22:29.1430833Z cuda train hf_T5_large 2024-08-21T00:22:29.1433216Z pass_due_to_skip 2024-08-21T00:22:29.6143430Z TIMING: 2024-08-21T00:22:29.6143766Z STATS: call_* op count: 0 2024-08-21T00:22:29.6144347Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:22:33.2916003Z 2024-08-21T00:22:34.4729079Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:34.4729595Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:22:34.4730042Z cuda train hf_Whisper 2024-08-21T00:22:53.7239738Z W0821 00:22:53.723000 54788 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:23:11.9081222Z pass 2024-08-21T00:23:11.9146906Z TIMING: entire_frame_compile:26.73426 code_gen:14.23137 inductor_compile:19.88196 backend_compile:22.08127 total_wall_time:31.64657 2024-08-21T00:23:11.9149381Z STATS: call_* op count: 346 | FakeTensorMode.__torch_dispatch__:19489 | FakeTensor.__torch_dispatch__:5957 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2024-08-21T00:23:11.9150520Z Dynamo produced 3 graphs covering 346 ops with 6 graph breaks (5 unique) 2024-08-21T00:23:17.0298400Z 2024-08-21T00:23:17.3306301Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:17.3308437Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:17.3308937Z cuda train hf_distil_whisper 2024-08-21T00:23:17.3326486Z Traceback (most recent call last): 2024-08-21T00:23:17.3327262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:23:17.3327966Z ) = runner.load_model( 2024-08-21T00:23:17.3328690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-21T00:23:17.3330639Z benchmark = benchmark_cls( 2024-08-21T00:23:17.3331558Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-21T00:23:17.3332374Z obj = type.__call__(cls, *args, **kwargs) 2024-08-21T00:23:17.3333294Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-08-21T00:23:17.3334291Z raise NotImplementedError("Training is not implemented.") 2024-08-21T00:23:17.3334899Z NotImplementedError: Training is not implemented. 2024-08-21T00:23:17.3335244Z 2024-08-21T00:23:17.3335371Z model_fail_to_load 2024-08-21T00:23:20.8106289Z 2024-08-21T00:23:21.2801510Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:21.2802094Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:23:21.2802539Z cuda train lennard_jones 2024-08-21T00:23:26.5386026Z W0821 00:23:26.537000 55216 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:23:30.0680260Z pass 2024-08-21T00:23:30.0681210Z TIMING: entire_frame_compile:6.66947 code_gen:2.26151 inductor_compile:3.13783 backend_compile:5.92947 total_wall_time:7.11984 2024-08-21T00:23:30.0682570Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:2892 | ProxyTorchDispatchMode.__torch_dispatch__:1221 | FakeTensor.__torch_dispatch__:796 2024-08-21T00:23:30.0683669Z Dynamo produced 3 graphs covering 60 ops with 7 graph breaks (5 unique) 2024-08-21T00:23:31.2227050Z accuracy pass_rate=81.82% 2024-08-21T00:23:31.2227656Z calls_captured gmean=0.00x mean=447.394x 2024-08-21T00:23:31.2231344Z unique_graphs gmean=0.00x mean=3.303x 2024-08-21T00:23:31.2234181Z graph_breaks gmean=0.00x mean=6.545x 2024-08-21T00:23:31.2237760Z unique_graph_breaks gmean=0.00x mean=4.333x 2024-08-21T00:23:31.2241575Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:23:31.2245001Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:23:31.2248622Z cudagraph_skips gmean=0.00x mean=0.727x 2024-08-21T00:23:32.5208021Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2024-08-21T00:23:32.8587945Z torchrec_dlrm PASS 2024-08-21T00:23:32.8590288Z BERT_pytorch PASS 2024-08-21T00:23:32.8595085Z Background_Matting XFAIL 2024-08-21T00:23:32.8598951Z LearningToPaint PASS 2024-08-21T00:23:32.8603189Z Super_SloMo PASS 2024-08-21T00:23:32.8607898Z alexnet PASS 2024-08-21T00:23:32.8612177Z basic_gnn_edgecnn PASS 2024-08-21T00:23:32.8616534Z basic_gnn_gcn PASS 2024-08-21T00:23:32.8620792Z basic_gnn_gin PASS 2024-08-21T00:23:32.8625430Z basic_gnn_sage PASS 2024-08-21T00:23:32.8629666Z dcgan PASS 2024-08-21T00:23:32.8634240Z demucs PASS 2024-08-21T00:23:32.8638111Z densenet121 PASS 2024-08-21T00:23:32.8642637Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-08-21T00:23:32.8646637Z dlrm PASS 2024-08-21T00:23:32.8651171Z drq PASS 2024-08-21T00:23:32.8655312Z fastNLP_Bert PASS 2024-08-21T00:23:32.8659800Z functorch_dp_cifar10 PASS 2024-08-21T00:23:32.8664042Z functorch_maml_omniglot PASS 2024-08-21T00:23:32.8668584Z hf_Albert PASS 2024-08-21T00:23:32.8672995Z hf_Bart PASS 2024-08-21T00:23:32.8677141Z hf_Bert PASS 2024-08-21T00:23:32.8681722Z hf_Bert_large PASS 2024-08-21T00:23:32.8685712Z hf_BigBird PASS 2024-08-21T00:23:32.8690191Z hf_DistilBert PASS 2024-08-21T00:23:32.8694268Z hf_GPT2 PASS 2024-08-21T00:23:32.8698750Z hf_GPT2_large XFAIL 2024-08-21T00:23:32.8702989Z hf_Reformer PASS 2024-08-21T00:23:32.8707277Z hf_T5_base XFAIL 2024-08-21T00:23:32.8711622Z hf_T5_large XFAIL 2024-08-21T00:23:32.8715685Z hf_Whisper PASS 2024-08-21T00:23:32.8720199Z hf_distil_whisper XFAIL 2024-08-21T00:23:32.8724128Z lennard_jones PASS 2024-08-21T00:23:32.9248794Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv 2024-08-21T00:23:33.2605543Z torchrec_dlrm PASS 2024-08-21T00:23:33.2609729Z BERT_pytorch PASS 2024-08-21T00:23:33.2614234Z Background_Matting PASS 2024-08-21T00:23:33.2618723Z LearningToPaint PASS 2024-08-21T00:23:33.2623019Z Super_SloMo PASS 2024-08-21T00:23:33.2627830Z alexnet PASS 2024-08-21T00:23:33.2632150Z basic_gnn_edgecnn PASS 2024-08-21T00:23:33.2636664Z basic_gnn_gcn PASS 2024-08-21T00:23:33.2641399Z basic_gnn_gin PASS 2024-08-21T00:23:33.2645489Z basic_gnn_sage PASS 2024-08-21T00:23:33.2649710Z dcgan PASS 2024-08-21T00:23:33.2654025Z demucs PASS 2024-08-21T00:23:33.2658578Z densenet121 PASS 2024-08-21T00:23:33.2662739Z detectron2_maskrcnn_r_50_c4 PASS 2024-08-21T00:23:33.2667242Z dlrm PASS 2024-08-21T00:23:33.2671981Z drq PASS 2024-08-21T00:23:33.2676402Z fastNLP_Bert PASS 2024-08-21T00:23:33.2680723Z functorch_dp_cifar10 PASS 2024-08-21T00:23:33.2684959Z functorch_maml_omniglot PASS 2024-08-21T00:23:33.2689207Z hf_Albert PASS 2024-08-21T00:23:33.2693572Z hf_Bart PASS 2024-08-21T00:23:33.2697872Z hf_Bert PASS 2024-08-21T00:23:33.2702213Z hf_Bert_large PASS 2024-08-21T00:23:33.2706857Z hf_BigBird PASS 2024-08-21T00:23:33.2710993Z hf_DistilBert PASS 2024-08-21T00:23:33.2715277Z hf_GPT2 PASS 2024-08-21T00:23:33.2719582Z hf_GPT2_large PASS 2024-08-21T00:23:33.2724014Z hf_Reformer PASS 2024-08-21T00:23:33.2728240Z hf_T5_base PASS 2024-08-21T00:23:33.2732667Z hf_T5_large PASS 2024-08-21T00:23:33.2736902Z hf_Whisper PASS 2024-08-21T00:23:33.2741235Z hf_distil_whisper PASS 2024-08-21T00:23:33.2745787Z lennard_jones PASS 2024-08-21T00:23:33.3281510Z + cleanup_workspace 2024-08-21T00:23:33.3282712Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:23:33.3283962Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:23:33.3284985Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:23:33.3285718Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:23:33.3286605Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:23:33.3287569Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:23:33.3288349Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:23:34.1072626Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:23:34.1073243Z cat test/**/*_toprint.log || true 2024-08-21T00:23:34.1087148Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.1087654Z env: 2024-08-21T00:23:34.1087939Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.1088386Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.1089113Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.1089752Z ##[endgroup] 2024-08-21T00:23:34.1187990Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:23:34.1219780Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:23:34.1220254Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:23:34.1229086Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.1229584Z env: 2024-08-21T00:23:34.1229867Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.1230324Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.1231057Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.1231724Z MONITOR_SCRIPT_PID: 52478 2024-08-21T00:23:34.1232124Z ##[endgroup] 2024-08-21T00:23:34.1432835Z Prepare all required actions 2024-08-21T00:23:34.1433292Z Getting action download info 2024-08-21T00:23:34.3450924Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:23:34.5291526Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:23:34.5292006Z with: 2024-08-21T00:23:34.5292587Z file-suffix: test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326 2024-08-21T00:23:34.5293825Z s3-bucket: gha-artifacts 2024-08-21T00:23:34.5294165Z env: 2024-08-21T00:23:34.5294451Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.5294905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.5295639Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.5296271Z ##[endgroup] 2024-08-21T00:23:34.5322905Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:23:34.5323558Z # Remove any previous test jsons if they exist 2024-08-21T00:23:34.5324072Z rm -f test-jsons-*.zip 2024-08-21T00:23:34.5324598Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:23:34.5334662Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.5335164Z env: 2024-08-21T00:23:34.5335444Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.5335887Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.5336626Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.5337557Z FILE_SUFFIX: test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326 2024-08-21T00:23:34.5338244Z ##[endgroup] 2024-08-21T00:23:34.5739963Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:23:34.5784708Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:23:34.5790112Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:23:34.5805655Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:23:34.5818327Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:23:34.5850699Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:23:34.5851342Z # Remove any previous test reports if they exist 2024-08-21T00:23:34.5851873Z rm -f test-reports-*.zip 2024-08-21T00:23:34.5852481Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:23:34.5861427Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.5861924Z env: 2024-08-21T00:23:34.5862209Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.5862773Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.5863542Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.5864477Z FILE_SUFFIX: test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326 2024-08-21T00:23:34.5865173Z ##[endgroup] 2024-08-21T00:23:34.6129636Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-08-21T00:23:34.6133449Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-21T00:23:34.6135585Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-08-21T00:23:34.6136475Z adding: test/test-reports/training_torchbench.csv (deflated 63%) 2024-08-21T00:23:34.6143147Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-21T00:23:34.6144424Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-21T00:23:34.6177544Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:23:34.6178173Z # Remove any previous usage logs if they exist 2024-08-21T00:23:34.6178670Z rm -f logs-*.zip 2024-08-21T00:23:34.6179539Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:23:34.6180337Z # so check to see if the file exists first 2024-08-21T00:23:34.6180839Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:23:34.6181368Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:23:34.6181859Z fi 2024-08-21T00:23:34.6182232Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:23:34.6183106Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:23:34.6183603Z fi 2024-08-21T00:23:34.6192249Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.6192749Z env: 2024-08-21T00:23:34.6193026Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.6193489Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.6194228Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.6195174Z FILE_SUFFIX: test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326 2024-08-21T00:23:34.6195850Z ##[endgroup] 2024-08-21T00:23:34.6311650Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:23:34.6399221Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:23:34.6399925Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:23:34.6400466Z rm -f debug-*.zip 2024-08-21T00:23:34.6400850Z if [ -d 'test/debug' ]; then 2024-08-21T00:23:34.6401358Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:23:34.6401834Z fi 2024-08-21T00:23:34.6410177Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:34.6410671Z env: 2024-08-21T00:23:34.6410944Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.6411397Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.6412129Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.6413070Z FILE_SUFFIX: test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326 2024-08-21T00:23:34.6413749Z ##[endgroup] 2024-08-21T00:23:34.6505333Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:23:34.6505762Z with: 2024-08-21T00:23:34.6506047Z s3-bucket: gha-artifacts 2024-08-21T00:23:34.6506483Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:34.6506954Z retention-days: 14 2024-08-21T00:23:34.6507288Z if-no-files-found: warn 2024-08-21T00:23:34.6507648Z path: test-jsons-*.zip 2024-08-21T00:23:34.6507988Z name: artifact 2024-08-21T00:23:34.6508285Z region: us-east-1 2024-08-21T00:23:34.6508583Z env: 2024-08-21T00:23:34.6508856Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:34.6509308Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:34.6510039Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:34.6510674Z ##[endgroup] 2024-08-21T00:23:35.0469075Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:23:35.0470310Z With the provided path, there will be 1 file uploaded 2024-08-21T00:23:35.0471533Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:35.2115070Z Starting upload of test-jsons-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:35.3026547Z Finished upload of test-jsons-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:35.3313637Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:23:35.3314081Z with: 2024-08-21T00:23:35.3314367Z s3-bucket: gha-artifacts 2024-08-21T00:23:35.3314812Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:35.3315289Z retention-days: 14 2024-08-21T00:23:35.3315624Z if-no-files-found: error 2024-08-21T00:23:35.3315993Z path: test-reports-*.zip 2024-08-21T00:23:35.3316347Z name: artifact 2024-08-21T00:23:35.3316650Z region: us-east-1 2024-08-21T00:23:35.3316953Z env: 2024-08-21T00:23:35.3317438Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:35.3317901Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:35.3318637Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:35.3319279Z ##[endgroup] 2024-08-21T00:23:35.6887829Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:23:35.6888446Z With the provided path, there will be 1 file uploaded 2024-08-21T00:23:35.6889330Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:35.6945444Z Starting upload of test-reports-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:35.7917613Z Finished upload of test-reports-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:35.8172303Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:23:35.8172741Z with: 2024-08-21T00:23:35.8173022Z s3-bucket: gha-artifacts 2024-08-21T00:23:35.8173473Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:35.8173944Z retention-days: 14 2024-08-21T00:23:35.8174277Z if-no-files-found: ignore 2024-08-21T00:23:35.8174634Z path: logs-*.zip 2024-08-21T00:23:35.8174943Z name: artifact 2024-08-21T00:23:35.8175241Z region: us-east-1 2024-08-21T00:23:35.8175542Z env: 2024-08-21T00:23:35.8175813Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:35.8176264Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:35.8177005Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:35.8177643Z ##[endgroup] 2024-08-21T00:23:36.1743917Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:23:36.1744533Z With the provided path, there will be 1 file uploaded 2024-08-21T00:23:36.1745165Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:36.1800787Z Starting upload of logs-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:36.2976334Z Finished upload of logs-test-inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026528326.zip 2024-08-21T00:23:36.3268472Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:23:36.3268918Z with: 2024-08-21T00:23:36.3269210Z s3-bucket: gha-artifacts 2024-08-21T00:23:36.3269648Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:23:36.3270122Z retention-days: 14 2024-08-21T00:23:36.3270472Z if-no-files-found: ignore 2024-08-21T00:23:36.3270834Z path: debug-*.zip 2024-08-21T00:23:36.3271149Z name: artifact 2024-08-21T00:23:36.3271449Z region: us-east-1 2024-08-21T00:23:36.3271753Z env: 2024-08-21T00:23:36.3272031Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:36.3272484Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:36.3273258Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:36.3273894Z ##[endgroup] 2024-08-21T00:23:36.6789897Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:23:36.7074492Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:23:36.7074991Z # shellcheck disable=SC2156 2024-08-21T00:23:36.7075788Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:23:36.7084938Z shell: /usr/bin/bash -e {0} 2024-08-21T00:23:36.7085291Z env: 2024-08-21T00:23:36.7085592Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:36.7086044Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:36.7086774Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:36.7087416Z ##[endgroup] 2024-08-21T00:23:37.0434919Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:23:37.0435496Z with: 2024-08-21T00:23:37.0435752Z env: 2024-08-21T00:23:37.0436026Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:37.0436471Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:37.0437197Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:37.0437826Z ##[endgroup] 2024-08-21T00:23:37.0459239Z ##[group]Run set -eou pipefail 2024-08-21T00:23:37.0459668Z set -eou pipefail 2024-08-21T00:23:37.0460036Z  2024-08-21T00:23:37.0460562Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:23:37.0461371Z for _ in $(seq 1440); do 2024-08-21T00:23:37.0461854Z  # Break if no ssh session exists anymore 2024-08-21T00:23:37.0462369Z  if [ "$(who)" = "" ]; then 2024-08-21T00:23:37.0462926Z  break 2024-08-21T00:23:37.0463279Z  fi 2024-08-21T00:23:37.0463585Z  echo "." 2024-08-21T00:23:37.0463925Z  sleep 5 2024-08-21T00:23:37.0464256Z done 2024-08-21T00:23:37.0473568Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:37.0474070Z env: 2024-08-21T00:23:37.0474353Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:37.0474809Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:37.0475542Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:37.0476181Z ##[endgroup] 2024-08-21T00:23:37.0506255Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:23:37.0605012Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:23:37.0605775Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:23:37.0606361Z # shellcheck disable=SC2046 2024-08-21T00:23:37.0606837Z docker stop $(docker ps -q) || true 2024-08-21T00:23:37.0607310Z # Prune all of the docker images 2024-08-21T00:23:37.0607757Z docker system prune -af 2024-08-21T00:23:37.0616582Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:37.0617085Z env: 2024-08-21T00:23:37.0617363Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:23:37.0617818Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:23:37.0618559Z DOCKER_CONTAINER_ID: f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:37.0619204Z ##[endgroup] 2024-08-21T00:23:37.7506309Z f46ac5459924 2024-08-21T00:23:41.7407882Z Deleted Containers: 2024-08-21T00:23:41.7408534Z f46ac54599247738e97fbbfd3cb599fa86148ab86adeb204781bbd39f5e1a138 2024-08-21T00:23:41.7409003Z 2024-08-21T00:23:50.7117074Z Deleted Images: 2024-08-21T00:23:50.7119155Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:23:50.7122078Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:23:50.7124202Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:23:50.7125347Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:23:50.7126266Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:23:50.7127105Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:23:50.7127959Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:23:50.7128781Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:23:50.7129594Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:23:50.7130771Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:23:50.7131606Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:23:50.7132449Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:23:50.7133268Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:23:50.7134082Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:23:50.7134901Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:23:50.7135723Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:23:50.7136785Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:23:50.7137614Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:23:50.7138458Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:23:50.7139289Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:23:50.7140107Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:23:50.7140929Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:23:50.7141761Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:23:50.7142600Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:23:50.7143552Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:23:50.7144369Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:23:50.7145383Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:23:50.7146218Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:23:50.7147049Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:23:50.7147876Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:23:50.7148704Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:23:50.7149525Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:23:50.7150342Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:23:50.7151148Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:23:50.7151966Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:23:50.7152788Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:23:50.7153607Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:23:50.7154416Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:23:50.7155235Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:23:50.7156053Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:23:50.7156874Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:23:50.7157692Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:23:50.7158600Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:23:50.7159547Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:23:50.7160378Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:23:50.7161185Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:23:50.7162147Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:23:50.7162989Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:23:50.7163823Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:23:50.7164628Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:23:50.7165430Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:23:50.7166268Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:23:50.7167373Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:23:50.7168342Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:23:50.7169149Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:23:50.7169977Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:23:50.7170807Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:23:50.7171626Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:23:50.7172432Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:23:50.7173293Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:23:50.7174102Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:23:50.7174913Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:23:50.7175750Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:23:50.7176578Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:23:50.7177399Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:23:50.7178206Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:23:50.7179011Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:23:50.7179822Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:23:50.7180634Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:23:50.7181455Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:23:50.7182281Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:23:50.7183240Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:23:50.7184047Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:23:50.7184851Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:23:50.7185725Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:23:50.7186623Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:23:50.7187691Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:23:50.7188770Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:23:50.7189599Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:23:50.7190420Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:23:50.7191220Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:23:50.7191716Z 2024-08-21T00:23:50.7191867Z Total reclaimed space: 49.69GB 2024-08-21T00:23:50.7259054Z Post job cleanup. 2024-08-21T00:23:50.7310682Z Post job cleanup. 2024-08-21T00:23:50.8254826Z [command]/usr/bin/git version 2024-08-21T00:23:50.8334283Z git version 2.40.1 2024-08-21T00:23:50.8375002Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b20ea70b-4851-47aa-b3dd-aedb6c6902c7' before making global git config changes 2024-08-21T00:23:50.8376344Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:23:50.8380118Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:23:50.8431408Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:23:50.8477396Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:23:50.8887621Z Entering 'android/libs/fbjni' 2024-08-21T00:23:50.8955163Z Entering 'third_party/FP16' 2024-08-21T00:23:50.9021275Z Entering 'third_party/FXdiv' 2024-08-21T00:23:50.9089064Z Entering 'third_party/NNPACK' 2024-08-21T00:23:50.9154050Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:23:50.9220833Z Entering 'third_party/XNNPACK' 2024-08-21T00:23:50.9302548Z Entering 'third_party/benchmark' 2024-08-21T00:23:50.9369372Z Entering 'third_party/cpp-httplib' 2024-08-21T00:23:50.9434981Z Entering 'third_party/cpuinfo' 2024-08-21T00:23:50.9500726Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:23:50.9564893Z Entering 'third_party/cutlass' 2024-08-21T00:23:50.9638104Z Entering 'third_party/eigen' 2024-08-21T00:23:50.9705450Z Entering 'third_party/fbgemm' 2024-08-21T00:23:50.9772876Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:23:50.9840360Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:23:50.9905346Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:23:50.9975824Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:23:51.0040610Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:23:51.0107622Z Entering 'third_party/flatbuffers' 2024-08-21T00:23:51.0176368Z Entering 'third_party/fmt' 2024-08-21T00:23:51.0241132Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:23:51.0312312Z Entering 'third_party/gloo' 2024-08-21T00:23:51.0383010Z Entering 'third_party/googletest' 2024-08-21T00:23:51.0452041Z Entering 'third_party/ideep' 2024-08-21T00:23:51.0519238Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:23:51.0594192Z Entering 'third_party/ittapi' 2024-08-21T00:23:51.0662171Z Entering 'third_party/kineto' 2024-08-21T00:23:51.0730569Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:23:51.0796775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:23:51.0865174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:23:51.0938506Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:23:51.1008719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:23:51.1074890Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:23:51.1145705Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:23:51.1213800Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:23:51.1281788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:23:51.1353098Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:23:51.1423989Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:23:51.1493433Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:23:51.1563622Z Entering 'third_party/mimalloc' 2024-08-21T00:23:51.1632395Z Entering 'third_party/nccl/nccl' 2024-08-21T00:23:51.1703312Z Entering 'third_party/nlohmann' 2024-08-21T00:23:51.1772531Z Entering 'third_party/onnx' 2024-08-21T00:23:51.1854400Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:23:51.1924097Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:23:51.1998343Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:23:51.2066585Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:23:51.2133841Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:23:51.2205142Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:23:51.2277546Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:23:51.2345459Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:23:51.2411123Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:23:51.2477941Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:23:51.2544235Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:23:51.2613528Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:23:51.2683458Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:23:51.2773807Z Entering 'third_party/pocketfft' 2024-08-21T00:23:51.2842705Z Entering 'third_party/protobuf' 2024-08-21T00:23:51.2912825Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:23:51.2980278Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:23:51.3050703Z Entering 'third_party/psimd' 2024-08-21T00:23:51.3119359Z Entering 'third_party/pthreadpool' 2024-08-21T00:23:51.3188657Z Entering 'third_party/pybind11' 2024-08-21T00:23:51.3257488Z Entering 'third_party/python-peachpy' 2024-08-21T00:23:51.3328505Z Entering 'third_party/sleef' 2024-08-21T00:23:51.3397028Z Entering 'third_party/tensorpipe' 2024-08-21T00:23:51.3464423Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:23:51.3530949Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:23:51.3597090Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:23:51.3667783Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:23:51.3733035Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:23:51.3831014Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:23:51.3867390Z http.https://github.com/.extraheader 2024-08-21T00:23:51.3877427Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:23:51.3924126Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:23:51.4294333Z Entering 'android/libs/fbjni' 2024-08-21T00:23:51.4337510Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4383602Z Entering 'third_party/FP16' 2024-08-21T00:23:51.4427738Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4472219Z Entering 'third_party/FXdiv' 2024-08-21T00:23:51.4515498Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4558607Z Entering 'third_party/NNPACK' 2024-08-21T00:23:51.4602325Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4646202Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:23:51.4692497Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4736835Z Entering 'third_party/XNNPACK' 2024-08-21T00:23:51.4782126Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4841808Z Entering 'third_party/benchmark' 2024-08-21T00:23:51.4885833Z http.https://github.com/.extraheader 2024-08-21T00:23:51.4929606Z Entering 'third_party/cpp-httplib' 2024-08-21T00:23:51.4973268Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5017677Z Entering 'third_party/cpuinfo' 2024-08-21T00:23:51.5060694Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5105602Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:23:51.5148325Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5193187Z Entering 'third_party/cutlass' 2024-08-21T00:23:51.5236696Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5288148Z Entering 'third_party/eigen' 2024-08-21T00:23:51.5331059Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5376964Z Entering 'third_party/fbgemm' 2024-08-21T00:23:51.5420497Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5463027Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:23:51.5506645Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5549433Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:23:51.5592314Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5634972Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:23:51.5677094Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5727182Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:23:51.5771379Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5813871Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:23:51.5856316Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5902112Z Entering 'third_party/flatbuffers' 2024-08-21T00:23:51.5946079Z http.https://github.com/.extraheader 2024-08-21T00:23:51.5995815Z Entering 'third_party/fmt' 2024-08-21T00:23:51.6043933Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6089206Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:23:51.6133720Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6178757Z Entering 'third_party/gloo' 2024-08-21T00:23:51.6222849Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6266483Z Entering 'third_party/googletest' 2024-08-21T00:23:51.6313852Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6357366Z Entering 'third_party/ideep' 2024-08-21T00:23:51.6401529Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6443238Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:23:51.6490518Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6540663Z Entering 'third_party/ittapi' 2024-08-21T00:23:51.6585391Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6632485Z Entering 'third_party/kineto' 2024-08-21T00:23:51.6677218Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6720354Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:23:51.6766679Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6811632Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:23:51.6854902Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6901248Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:23:51.6943599Z http.https://github.com/.extraheader 2024-08-21T00:23:51.6989040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:23:51.7031341Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7077303Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:23:51.7120076Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7163656Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:23:51.7207563Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7257037Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:23:51.7300433Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7344480Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:23:51.7388127Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7432771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:23:51.7475940Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7521407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:23:51.7564510Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7611719Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:23:51.7653466Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7697026Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:23:51.7739867Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7788936Z Entering 'third_party/mimalloc' 2024-08-21T00:23:51.7833440Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7877390Z Entering 'third_party/nccl/nccl' 2024-08-21T00:23:51.7921427Z http.https://github.com/.extraheader 2024-08-21T00:23:51.7965362Z Entering 'third_party/nlohmann' 2024-08-21T00:23:51.8009815Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8054653Z Entering 'third_party/onnx' 2024-08-21T00:23:51.8099691Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8158046Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:23:51.8201854Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8246350Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:23:51.8289767Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8337417Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:23:51.8382738Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8429482Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:23:51.8471848Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8514216Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:23:51.8556929Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8600466Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:23:51.8642678Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8685411Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:23:51.8727525Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8773053Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:23:51.8822997Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8866689Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:23:51.8909565Z http.https://github.com/.extraheader 2024-08-21T00:23:51.8952144Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:23:51.8994911Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9035895Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:23:51.9079240Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9124773Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:23:51.9167841Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9216821Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:23:51.9259106Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9323160Z Entering 'third_party/pocketfft' 2024-08-21T00:23:51.9367450Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9414224Z Entering 'third_party/protobuf' 2024-08-21T00:23:51.9457935Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9504043Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:23:51.9549669Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9592585Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:23:51.9635292Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9684889Z Entering 'third_party/psimd' 2024-08-21T00:23:51.9728832Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9773292Z Entering 'third_party/pthreadpool' 2024-08-21T00:23:51.9817551Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9861327Z Entering 'third_party/pybind11' 2024-08-21T00:23:51.9905571Z http.https://github.com/.extraheader 2024-08-21T00:23:51.9953156Z Entering 'third_party/python-peachpy' 2024-08-21T00:23:51.9996699Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0040220Z Entering 'third_party/sleef' 2024-08-21T00:23:52.0083842Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0127657Z Entering 'third_party/tensorpipe' 2024-08-21T00:23:52.0170684Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0213102Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:23:52.0255242Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0299780Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:23:52.0341874Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0384681Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:23:52.0426735Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0471138Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:23:52.0513153Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0556579Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:23:52.0600307Z http.https://github.com/.extraheader 2024-08-21T00:23:52.0769143Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:23:52.0791065Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:23:52.0799171Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:23:52.0799686Z ##[endgroup] 2024-08-21T00:23:58.9974814Z Cleaning up orphan processes