2024-08-20T22:09:47.0751520Z Current runner version: '2.319.1' 2024-08-20T22:09:47.0757786Z Runner name: 'i-06bc7425c52774f50' 2024-08-20T22:09:47.0758595Z Runner group name: 'Default' 2024-08-20T22:09:47.0759437Z Machine name: 'ip-10-0-22-103' 2024-08-20T22:09:47.0776080Z Testing runner upgrade compatibility 2024-08-20T22:09:47.2383906Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:09:47.2386163Z Actions: read 2024-08-20T22:09:47.2386663Z Attestations: read 2024-08-20T22:09:47.2387210Z Checks: read 2024-08-20T22:09:47.2387760Z Contents: read 2024-08-20T22:09:47.2388277Z Deployments: read 2024-08-20T22:09:47.2388790Z Discussions: read 2024-08-20T22:09:47.2389355Z Issues: read 2024-08-20T22:09:47.2389842Z Metadata: read 2024-08-20T22:09:47.2390328Z Packages: read 2024-08-20T22:09:47.2390883Z Pages: read 2024-08-20T22:09:47.2391396Z PullRequests: read 2024-08-20T22:09:47.2391953Z RepositoryProjects: read 2024-08-20T22:09:47.2392568Z SecurityEvents: read 2024-08-20T22:09:47.2393126Z Statuses: read 2024-08-20T22:09:47.2393857Z ##[endgroup] 2024-08-20T22:09:47.2396979Z Secret source: Actions 2024-08-20T22:09:47.2397761Z Prepare workflow directory 2024-08-20T22:09:47.3331574Z Prepare all required actions 2024-08-20T22:09:47.3500032Z Getting action download info 2024-08-20T22:09:47.5130569Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:09:47.9395535Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:09:51.9291730Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:09:52.1146731Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:09:52.5403629Z Getting action download info 2024-08-20T22:09:52.6329311Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:09:52.7963693Z Getting action download info 2024-08-20T22:09:52.8894867Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:09:53.0366943Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:09:53.0369341Z ##[group] Inputs 2024-08-20T22:09:53.0369812Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:09:53.0378949Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:09:53.0388901Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:09:53.0390051Z sync-tag: 2024-08-20T22:09:53.0390872Z timeout-minutes: 240 2024-08-20T22:09:53.0391222Z use-gha: 2024-08-20T22:09:53.0391528Z dashboard-tag: 2024-08-20T22:09:53.0391853Z s3-bucket: gha-artifacts 2024-08-20T22:09:53.0392204Z aws-role-to-assume: 2024-08-20T22:09:53.0392534Z ##[endgroup] 2024-08-20T22:09:53.0393509Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:09:53.1092511Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:09:53.1232405Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:09:53.1244063Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:09:53.1244581Z ##[endgroup] 2024-08-20T22:09:54.3340081Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:09:54.3340638Z Instance Type: g5.4xlarge 2024-08-20T22:09:54.3341323Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:09:54.3341858Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:00.0953132Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:00.0953719Z with: 2024-08-20T22:10:00.0954453Z github-secret: *** 2024-08-20T22:10:00.0955356Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:00.0956331Z activate-with-label: false 2024-08-20T22:10:00.0956688Z label: with-ssh 2024-08-20T22:10:00.0957002Z remove-existing-keys: true 2024-08-20T22:10:00.0957351Z fail-silently: true 2024-08-20T22:10:00.0957660Z env: 2024-08-20T22:10:00.0957928Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.0958268Z ##[endgroup] 2024-08-20T22:10:00.1848993Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:00.1851925Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:00.4262017Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:00.4947899Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:00.4948587Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:00.5532347Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:00.5549462Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:00.5574342Z Login using: ssh ec2-user@ec2-44-195-60-144.compute-1.amazonaws.com 2024-08-20T22:10:00.5575122Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:00.5576074Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:00.5707489Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:00.5708059Z with: 2024-08-20T22:10:00.5708356Z submodules: recursive 2024-08-20T22:10:00.5708701Z fetch-depth: 0 2024-08-20T22:10:00.5709011Z env: 2024-08-20T22:10:00.5709299Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.5709649Z ##[endgroup] 2024-08-20T22:10:00.5900097Z ##[group]Run retry () { 2024-08-20T22:10:00.5900489Z retry () { 2024-08-20T22:10:00.5901209Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:00.5901768Z } 2024-08-20T22:10:00.5902085Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.5902522Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:00.5903008Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.5903478Z else 2024-08-20T22:10:00.5903830Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.5904275Z fi 2024-08-20T22:10:00.5904675Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:00.5916264Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:00.5916771Z env: 2024-08-20T22:10:00.5917057Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.5917402Z NO_SUDO: 2024-08-20T22:10:00.5917683Z ##[endgroup] 2024-08-20T22:10:00.5951820Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.7579768Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:00.7580212Z with: 2024-08-20T22:10:00.7580536Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:00.7580980Z fetch-depth: 0 2024-08-20T22:10:00.7581410Z submodules: recursive 2024-08-20T22:10:00.7581747Z quiet-checkout: true 2024-08-20T22:10:00.7582092Z repository: pytorch/pytorch 2024-08-20T22:10:00.7582571Z token: *** 2024-08-20T22:10:00.7582865Z ssh-strict: true 2024-08-20T22:10:00.7583188Z persist-credentials: true 2024-08-20T22:10:00.7583543Z clean: true 2024-08-20T22:10:00.7583894Z sparse-checkout-cone-mode: true 2024-08-20T22:10:00.7584276Z lfs: false 2024-08-20T22:10:00.7584569Z set-safe-directory: true 2024-08-20T22:10:00.7584907Z env: 2024-08-20T22:10:00.7585179Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:00.7585519Z ##[endgroup] 2024-08-20T22:10:00.8607907Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:00.8610096Z ##[group]Getting Git version info 2024-08-20T22:10:00.8611237Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:00.8612543Z [command]/usr/bin/git version 2024-08-20T22:10:00.8613118Z git version 2.40.1 2024-08-20T22:10:00.8627752Z ##[endgroup] 2024-08-20T22:10:00.8647462Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/ffe4d6c5-3b9d-4a12-b68d-89478c3f85c0' before making global git config changes 2024-08-20T22:10:00.8649262Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:00.8654073Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.8705773Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:00.8710070Z ##[group]Initializing the repository 2024-08-20T22:10:00.8713708Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:00.8757319Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:00.8758260Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:00.8759054Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:00.8759613Z hint: 2024-08-20T22:10:00.8760070Z hint: git config --global init.defaultBranch 2024-08-20T22:10:00.8760540Z hint: 2024-08-20T22:10:00.8761057Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:00.8762163Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:00.8763437Z hint: 2024-08-20T22:10:00.8763883Z hint: git branch -m 2024-08-20T22:10:00.8764938Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:00.8772385Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:00.8814928Z ##[endgroup] 2024-08-20T22:10:00.8815828Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:00.8818680Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:00.8861078Z ##[endgroup] 2024-08-20T22:10:00.8861853Z ##[group]Setting up auth 2024-08-20T22:10:00.8868181Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:00.8912109Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:00.9285743Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:00.9329216Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:00.9682927Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:00.9739718Z ##[endgroup] 2024-08-20T22:10:00.9740551Z ##[group]Fetching the repository 2024-08-20T22:10:00.9748311Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:04.0833898Z remote: Enumerating objects: 1008509 2024-08-20T22:10:04.0834519Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:04.0837637Z remote: Counting objects: 0% (1/1720) 2024-08-20T22:10:04.0838302Z remote: Counting objects: 1% (18/1720) 2024-08-20T22:10:04.0838819Z remote: Counting objects: 2% (35/1720) 2024-08-20T22:10:04.0839316Z remote: Counting objects: 3% (52/1720) 2024-08-20T22:10:04.0839918Z remote: Counting objects: 4% (69/1720) 2024-08-20T22:10:04.0840529Z remote: Counting objects: 5% (86/1720) 2024-08-20T22:10:04.0841205Z remote: Counting objects: 6% (104/1720) 2024-08-20T22:10:04.0841834Z remote: Counting objects: 7% (121/1720) 2024-08-20T22:10:04.0842542Z remote: Counting objects: 8% (138/1720) 2024-08-20T22:10:04.0843207Z remote: Counting objects: 9% (155/1720) 2024-08-20T22:10:04.0843831Z remote: Counting objects: 10% (172/1720) 2024-08-20T22:10:04.0844492Z remote: Counting objects: 11% (190/1720) 2024-08-20T22:10:04.0845167Z remote: Counting objects: 12% (207/1720) 2024-08-20T22:10:04.0846121Z remote: Counting objects: 13% (224/1720) 2024-08-20T22:10:04.0846916Z remote: Counting objects: 14% (241/1720) 2024-08-20T22:10:04.0847592Z remote: Counting objects: 15% (258/1720) 2024-08-20T22:10:04.0848244Z remote: Counting objects: 16% (276/1720) 2024-08-20T22:10:04.0848885Z remote: Counting objects: 17% (293/1720) 2024-08-20T22:10:04.0849378Z remote: Counting objects: 18% (310/1720) 2024-08-20T22:10:04.0849878Z remote: Counting objects: 19% (327/1720) 2024-08-20T22:10:04.0850373Z remote: Counting objects: 20% (344/1720) 2024-08-20T22:10:04.0850872Z remote: Counting objects: 21% (362/1720) 2024-08-20T22:10:04.0851359Z remote: Counting objects: 22% (379/1720) 2024-08-20T22:10:04.0851852Z remote: Counting objects: 23% (396/1720) 2024-08-20T22:10:04.0852344Z remote: Counting objects: 24% (413/1720) 2024-08-20T22:10:04.0852833Z remote: Counting objects: 25% (430/1720) 2024-08-20T22:10:04.0853318Z remote: Counting objects: 26% (448/1720) 2024-08-20T22:10:04.0853804Z remote: Counting objects: 27% (465/1720) 2024-08-20T22:10:04.0854703Z remote: Counting objects: 28% (482/1720) 2024-08-20T22:10:04.0855208Z remote: Counting objects: 29% (499/1720) 2024-08-20T22:10:04.0855700Z remote: Counting objects: 30% (516/1720) 2024-08-20T22:10:04.0856332Z remote: Counting objects: 31% (534/1720) 2024-08-20T22:10:04.0856963Z remote: Counting objects: 32% (551/1720) 2024-08-20T22:10:04.0857546Z remote: Counting objects: 33% (568/1720) 2024-08-20T22:10:04.0858188Z remote: Counting objects: 34% (585/1720) 2024-08-20T22:10:04.0858685Z remote: Counting objects: 35% (602/1720) 2024-08-20T22:10:04.0859177Z remote: Counting objects: 36% (620/1720) 2024-08-20T22:10:04.0859670Z remote: Counting objects: 37% (637/1720) 2024-08-20T22:10:04.0860151Z remote: Counting objects: 38% (654/1720) 2024-08-20T22:10:04.0860645Z remote: Counting objects: 39% (671/1720) 2024-08-20T22:10:04.0861135Z remote: Counting objects: 40% (688/1720) 2024-08-20T22:10:04.0861643Z remote: Counting objects: 41% (706/1720) 2024-08-20T22:10:04.0862138Z remote: Counting objects: 42% (723/1720) 2024-08-20T22:10:04.0862726Z remote: Counting objects: 43% (740/1720) 2024-08-20T22:10:04.0863277Z remote: Counting objects: 44% (757/1720) 2024-08-20T22:10:04.0863801Z remote: Counting objects: 45% (774/1720) 2024-08-20T22:10:04.0864437Z remote: Counting objects: 46% (792/1720) 2024-08-20T22:10:04.0865116Z remote: Counting objects: 47% (809/1720) 2024-08-20T22:10:04.0865793Z remote: Counting objects: 48% (826/1720) 2024-08-20T22:10:04.0866427Z remote: Counting objects: 49% (843/1720) 2024-08-20T22:10:04.0867057Z remote: Counting objects: 50% (860/1720) 2024-08-20T22:10:04.0867694Z remote: Counting objects: 51% (878/1720) 2024-08-20T22:10:04.0868336Z remote: Counting objects: 52% (895/1720) 2024-08-20T22:10:04.0868994Z remote: Counting objects: 53% (912/1720) 2024-08-20T22:10:04.0869570Z remote: Counting objects: 54% (929/1720) 2024-08-20T22:10:04.0870096Z remote: Counting objects: 55% (946/1720) 2024-08-20T22:10:04.0870586Z remote: Counting objects: 56% (964/1720) 2024-08-20T22:10:04.0871099Z remote: Counting objects: 57% (981/1720) 2024-08-20T22:10:04.0871586Z remote: Counting objects: 58% (998/1720) 2024-08-20T22:10:04.0872077Z remote: Counting objects: 59% (1015/1720) 2024-08-20T22:10:04.0872601Z remote: Counting objects: 60% (1032/1720) 2024-08-20T22:10:04.0873117Z remote: Counting objects: 61% (1050/1720) 2024-08-20T22:10:04.0873632Z remote: Counting objects: 62% (1067/1720) 2024-08-20T22:10:04.0874149Z remote: Counting objects: 63% (1084/1720) 2024-08-20T22:10:04.0874653Z remote: Counting objects: 64% (1101/1720) 2024-08-20T22:10:04.0875162Z remote: Counting objects: 65% (1118/1720) 2024-08-20T22:10:04.0875675Z remote: Counting objects: 66% (1136/1720) 2024-08-20T22:10:04.0876186Z remote: Counting objects: 67% (1153/1720) 2024-08-20T22:10:04.0876684Z remote: Counting objects: 68% (1170/1720) 2024-08-20T22:10:04.0877188Z remote: Counting objects: 69% (1187/1720) 2024-08-20T22:10:04.0877695Z remote: Counting objects: 70% (1204/1720) 2024-08-20T22:10:04.0878197Z remote: Counting objects: 71% (1222/1720) 2024-08-20T22:10:04.0878698Z remote: Counting objects: 72% (1239/1720) 2024-08-20T22:10:04.0879205Z remote: Counting objects: 73% (1256/1720) 2024-08-20T22:10:04.0879710Z remote: Counting objects: 74% (1273/1720) 2024-08-20T22:10:04.0880207Z remote: Counting objects: 75% (1290/1720) 2024-08-20T22:10:04.0880711Z remote: Counting objects: 76% (1308/1720) 2024-08-20T22:10:04.0881214Z remote: Counting objects: 77% (1325/1720) 2024-08-20T22:10:04.0881719Z remote: Counting objects: 78% (1342/1720) 2024-08-20T22:10:04.0882374Z remote: Counting objects: 79% (1359/1720) 2024-08-20T22:10:04.0882882Z remote: Counting objects: 80% (1376/1720) 2024-08-20T22:10:04.0883384Z remote: Counting objects: 81% (1394/1720) 2024-08-20T22:10:04.0883886Z remote: Counting objects: 82% (1411/1720) 2024-08-20T22:10:04.0884386Z remote: Counting objects: 83% (1428/1720) 2024-08-20T22:10:04.0884889Z remote: Counting objects: 84% (1445/1720) 2024-08-20T22:10:04.0885392Z remote: Counting objects: 85% (1462/1720) 2024-08-20T22:10:04.0885991Z remote: Counting objects: 86% (1480/1720) 2024-08-20T22:10:04.0886488Z remote: Counting objects: 87% (1497/1720) 2024-08-20T22:10:04.0887102Z remote: Counting objects: 88% (1514/1720) 2024-08-20T22:10:04.0887604Z remote: Counting objects: 89% (1531/1720) 2024-08-20T22:10:04.0888103Z remote: Counting objects: 90% (1548/1720) 2024-08-20T22:10:04.0888601Z remote: Counting objects: 91% (1566/1720) 2024-08-20T22:10:04.0889111Z remote: Counting objects: 92% (1583/1720) 2024-08-20T22:10:04.0889608Z remote: Counting objects: 93% (1600/1720) 2024-08-20T22:10:04.0890109Z remote: Counting objects: 94% (1617/1720) 2024-08-20T22:10:04.0890612Z remote: Counting objects: 95% (1634/1720) 2024-08-20T22:10:04.0891116Z remote: Counting objects: 96% (1652/1720) 2024-08-20T22:10:04.0891616Z remote: Counting objects: 97% (1669/1720) 2024-08-20T22:10:04.0892129Z remote: Counting objects: 98% (1686/1720) 2024-08-20T22:10:04.0892622Z remote: Counting objects: 99% (1703/1720) 2024-08-20T22:10:04.0893123Z remote: Counting objects: 100% (1720/1720) 2024-08-20T22:10:04.0893665Z remote: Counting objects: 100% (1720/1720), done. 2024-08-20T22:10:04.1073999Z remote: Compressing objects: 0% (1/836) 2024-08-20T22:10:04.1422200Z remote: Compressing objects: 1% (9/836) 2024-08-20T22:10:04.1800335Z remote: Compressing objects: 2% (17/836) 2024-08-20T22:10:04.2326525Z remote: Compressing objects: 3% (26/836) 2024-08-20T22:10:04.3122590Z remote: Compressing objects: 4% (34/836) 2024-08-20T22:10:04.3591297Z remote: Compressing objects: 5% (42/836) 2024-08-20T22:10:04.4191851Z remote: Compressing objects: 6% (51/836) 2024-08-20T22:10:04.4517380Z remote: Compressing objects: 7% (59/836) 2024-08-20T22:10:04.4908095Z remote: Compressing objects: 8% (67/836) 2024-08-20T22:10:04.5123180Z remote: Compressing objects: 9% (76/836) 2024-08-20T22:10:04.5366814Z remote: Compressing objects: 10% (84/836) 2024-08-20T22:10:04.5593240Z remote: Compressing objects: 11% (92/836) 2024-08-20T22:10:04.5764515Z remote: Compressing objects: 12% (101/836) 2024-08-20T22:10:04.5927628Z remote: Compressing objects: 13% (109/836) 2024-08-20T22:10:04.6042693Z remote: Compressing objects: 14% (118/836) 2024-08-20T22:10:04.6126809Z remote: Compressing objects: 15% (126/836) 2024-08-20T22:10:04.6213061Z remote: Compressing objects: 16% (134/836) 2024-08-20T22:10:04.6271296Z remote: Compressing objects: 17% (143/836) 2024-08-20T22:10:04.6314904Z remote: Compressing objects: 18% (151/836) 2024-08-20T22:10:04.6362354Z remote: Compressing objects: 19% (159/836) 2024-08-20T22:10:04.6393288Z remote: Compressing objects: 20% (168/836) 2024-08-20T22:10:04.6414155Z remote: Compressing objects: 21% (176/836) 2024-08-20T22:10:04.6417779Z remote: Compressing objects: 22% (184/836) 2024-08-20T22:10:04.6423085Z remote: Compressing objects: 23% (193/836) 2024-08-20T22:10:04.6430754Z remote: Compressing objects: 24% (201/836) 2024-08-20T22:10:04.6435646Z remote: Compressing objects: 25% (209/836) 2024-08-20T22:10:04.6445342Z remote: Compressing objects: 26% (218/836) 2024-08-20T22:10:04.6451039Z remote: Compressing objects: 27% (226/836) 2024-08-20T22:10:04.6469908Z remote: Compressing objects: 28% (235/836) 2024-08-20T22:10:04.6482310Z remote: Compressing objects: 29% (243/836) 2024-08-20T22:10:04.6492288Z remote: Compressing objects: 30% (251/836) 2024-08-20T22:10:04.6502622Z remote: Compressing objects: 31% (260/836) 2024-08-20T22:10:04.6506853Z remote: Compressing objects: 32% (268/836) 2024-08-20T22:10:04.6516402Z remote: Compressing objects: 33% (276/836) 2024-08-20T22:10:04.6523455Z remote: Compressing objects: 34% (285/836) 2024-08-20T22:10:04.6527192Z remote: Compressing objects: 35% (293/836) 2024-08-20T22:10:04.6535395Z remote: Compressing objects: 36% (301/836) 2024-08-20T22:10:04.6541540Z remote: Compressing objects: 37% (310/836) 2024-08-20T22:10:04.6550493Z remote: Compressing objects: 38% (318/836) 2024-08-20T22:10:04.6557390Z remote: Compressing objects: 39% (327/836) 2024-08-20T22:10:04.6562915Z remote: Compressing objects: 40% (335/836) 2024-08-20T22:10:04.6568638Z remote: Compressing objects: 41% (343/836) 2024-08-20T22:10:04.6577526Z remote: Compressing objects: 42% (352/836) 2024-08-20T22:10:04.6581819Z remote: Compressing objects: 43% (360/836) 2024-08-20T22:10:04.6589525Z remote: Compressing objects: 44% (368/836) 2024-08-20T22:10:04.6593936Z remote: Compressing objects: 45% (377/836) 2024-08-20T22:10:04.6600580Z remote: Compressing objects: 46% (385/836) 2024-08-20T22:10:04.6602861Z remote: Compressing objects: 47% (393/836) 2024-08-20T22:10:04.6605148Z remote: Compressing objects: 48% (402/836) 2024-08-20T22:10:04.6610146Z remote: Compressing objects: 49% (410/836) 2024-08-20T22:10:04.6611457Z remote: Compressing objects: 50% (418/836) 2024-08-20T22:10:04.6614884Z remote: Compressing objects: 51% (427/836) 2024-08-20T22:10:04.6617591Z remote: Compressing objects: 52% (435/836) 2024-08-20T22:10:04.6618896Z remote: Compressing objects: 53% (444/836) 2024-08-20T22:10:04.6621928Z remote: Compressing objects: 54% (452/836) 2024-08-20T22:10:04.6623168Z remote: Compressing objects: 55% (460/836) 2024-08-20T22:10:04.6624360Z remote: Compressing objects: 56% (469/836) 2024-08-20T22:10:04.6625040Z remote: Compressing objects: 57% (477/836) 2024-08-20T22:10:04.6628612Z remote: Compressing objects: 58% (485/836) 2024-08-20T22:10:04.6629341Z remote: Compressing objects: 59% (494/836) 2024-08-20T22:10:04.6630034Z remote: Compressing objects: 60% (502/836) 2024-08-20T22:10:04.6630582Z remote: Compressing objects: 61% (510/836) 2024-08-20T22:10:04.6631257Z remote: Compressing objects: 62% (519/836) 2024-08-20T22:10:04.6631902Z remote: Compressing objects: 63% (527/836) 2024-08-20T22:10:04.6632423Z remote: Compressing objects: 64% (536/836) 2024-08-20T22:10:04.6637190Z remote: Compressing objects: 65% (544/836) 2024-08-20T22:10:04.6643089Z remote: Compressing objects: 66% (552/836) 2024-08-20T22:10:04.6648162Z remote: Compressing objects: 67% (561/836) 2024-08-20T22:10:04.6651394Z remote: Compressing objects: 68% (569/836) 2024-08-20T22:10:04.6655892Z remote: Compressing objects: 69% (577/836) 2024-08-20T22:10:04.6660049Z remote: Compressing objects: 70% (586/836) 2024-08-20T22:10:04.6663226Z remote: Compressing objects: 71% (594/836) 2024-08-20T22:10:04.6666791Z remote: Compressing objects: 72% (602/836) 2024-08-20T22:10:04.6670412Z remote: Compressing objects: 73% (611/836) 2024-08-20T22:10:04.6673038Z remote: Compressing objects: 74% (619/836) 2024-08-20T22:10:04.6676613Z remote: Compressing objects: 75% (627/836) 2024-08-20T22:10:04.6678628Z remote: Compressing objects: 76% (636/836) 2024-08-20T22:10:04.6681702Z remote: Compressing objects: 77% (644/836) 2024-08-20T22:10:04.6684741Z remote: Compressing objects: 78% (653/836) 2024-08-20T22:10:04.6685648Z remote: Compressing objects: 79% (661/836) 2024-08-20T22:10:04.6687893Z remote: Compressing objects: 80% (669/836) 2024-08-20T22:10:04.6689791Z remote: Compressing objects: 81% (678/836) 2024-08-20T22:10:04.6691272Z remote: Compressing objects: 82% (686/836) 2024-08-20T22:10:04.6693531Z remote: Compressing objects: 83% (694/836) 2024-08-20T22:10:04.6695637Z remote: Compressing objects: 84% (703/836) 2024-08-20T22:10:04.6696965Z remote: Compressing objects: 85% (711/836) 2024-08-20T22:10:04.6699044Z remote: Compressing objects: 86% (719/836) 2024-08-20T22:10:04.6700421Z remote: Compressing objects: 87% (728/836) 2024-08-20T22:10:04.6703575Z remote: Compressing objects: 88% (736/836) 2024-08-20T22:10:04.6704758Z remote: Compressing objects: 89% (745/836) 2024-08-20T22:10:04.6709275Z remote: Compressing objects: 90% (753/836) 2024-08-20T22:10:04.6709993Z remote: Compressing objects: 91% (761/836) 2024-08-20T22:10:04.6710684Z remote: Compressing objects: 92% (770/836) 2024-08-20T22:10:04.6711388Z remote: Compressing objects: 93% (778/836) 2024-08-20T22:10:04.6712066Z remote: Compressing objects: 94% (786/836) 2024-08-20T22:10:04.6712641Z remote: Compressing objects: 95% (795/836) 2024-08-20T22:10:04.6713486Z remote: Compressing objects: 96% (803/836) 2024-08-20T22:10:04.6714691Z remote: Compressing objects: 97% (811/836) 2024-08-20T22:10:04.6715691Z remote: Compressing objects: 98% (820/836) 2024-08-20T22:10:04.6716390Z remote: Compressing objects: 99% (828/836) 2024-08-20T22:10:04.6717252Z remote: Compressing objects: 100% (836/836) 2024-08-20T22:10:04.6717966Z remote: Compressing objects: 100% (836/836), done. 2024-08-20T22:10:27.9026923Z remote: Total 1010229 (delta 1121), reused 1351 (delta 881), pack-reused 1008509 (from 1) 2024-08-20T22:10:49.9662243Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:10:49.9700610Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:49.9706789Z ##[endgroup] 2024-08-20T22:10:49.9709646Z ##[group]Determining the checkout info 2024-08-20T22:10:49.9710610Z ##[endgroup] 2024-08-20T22:10:49.9711392Z ##[group]Checking out the ref 2024-08-20T22:10:49.9712409Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:51.6447822Z ##[endgroup] 2024-08-20T22:10:51.6448817Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:10:51.6453666Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:51.6518820Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:10:51.6558365Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:10:51.6598151Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:10:51.6635063Z ##[endgroup] 2024-08-20T22:10:51.6635608Z ##[group]Fetching submodules 2024-08-20T22:10:51.6638880Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:10:51.7020034Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:10:51.7383581Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:10:51.7385406Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:10:51.7388617Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:10:51.7391695Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:10:51.7395529Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:10:51.7397968Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:10:51.7401630Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:10:51.7405114Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:10:51.7408823Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:10:51.7412612Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:10:51.7416108Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:10:51.7419846Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:10:51.7423637Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:10:51.7427629Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:10:51.7432065Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:10:51.7437706Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:10:51.7444971Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:10:51.7449857Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:10:51.7453890Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:10:51.7458339Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:10:51.7462674Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:10:51.7467129Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:10:51.7471548Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:10:51.7476344Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:10:51.7480744Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:10:51.7485700Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:10:51.7490381Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:10:51.7495257Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:10:51.7500120Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:10:51.7505155Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:10:51.7510092Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:10:51.7515494Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:10:51.7520414Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:10:51.7528941Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:10:51.7563096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:10:52.0623521Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:10:52.2530128Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:10:52.4302083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:10:52.6995517Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:10:54.6308647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:05.9992885Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:06.4311698Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:06.8374861Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:07.4318078Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:08.6751812Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:10.7752967Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:16.8455535Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:18.2475983Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:20.0397528Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:21.3554090Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:21.8154753Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:22.1457436Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:23.2944342Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:23.6495166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:23.9080784Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:25.5313034Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:26.2269332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:27.0413216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:33.0647647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:35.0319810Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:11:39.2910922Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:11:39.5394626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:11:48.5487673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:11:48.7376712Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:11:48.9292737Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:11:49.8543903Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:11:50.1534356Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:11:50.7926870Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:11:51.2086234Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:11:51.2236696Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:11:51.2355710Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:11:51.2661300Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:11:51.3117154Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:11:52.4879865Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:11:52.5167001Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:11:52.5696387Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:11:52.6810403Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:11:52.7217264Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:11:53.3430478Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:11:53.6292660Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:11:53.7252637Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:11:53.7270801Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:11:53.7274157Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:11:53.7277739Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:11:53.7281233Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:11:53.7284910Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:11:53.7317804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:11:54.7961262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:11:55.4112330Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:11:57.5946853Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:11:58.6819654Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:11:59.0424197Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:11:59.1528578Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:11:59.6624658Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:11:59.7329803Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:11:59.7480218Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:11:59.8984700Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:11:59.9447245Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:11:59.9901453Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:00.0225438Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:00.0756078Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:00.0912591Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:00.0934706Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:00.0965501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:13.1869255Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:13.2086089Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:13.3069608Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:13.3088777Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:13.3092094Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:13.3095484Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:13.3127668Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:13.8650259Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:15.0793601Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:16.2142456Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:16.2163235Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:16.2166726Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:16.2170419Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:16.2174086Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:16.2177875Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:16.2181707Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:16.2185500Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:16.2189325Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:16.2221129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:17.0770927Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:17.4679741Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:18.6986176Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:18.9987301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:19.5357408Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:20.5823786Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:28.0181587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:28.4646920Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:28.4877459Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:28.5302929Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:28.5468441Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:28.5487770Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:28.5519671Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:28.8383448Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:28.8608645Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:28.9085557Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:29.0259716Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:29.0460455Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:29.0903607Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:29.1554086Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:29.2004020Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:29.2303429Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:29.3530643Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:29.8652023Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:29.8687429Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:29.8690538Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:29.8722262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:30.3082110Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:31.2262200Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:31.2667598Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:31.3565924Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:31.3587210Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:31.3590533Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:31.3594471Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:31.3597979Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:31.3601769Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:31.3605341Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:31.3609203Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:31.3612653Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:31.3644269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:31.8058687Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:32.8574017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:33.2260915Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:12:39.3009157Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:12:39.5425914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:12:39.7368843Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:12:40.0381897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:12:46.7570591Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:12:46.8043275Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:12:46.8224546Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:12:46.9452926Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:12:46.9613145Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:12:46.9798209Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:12:46.9998453Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:12:47.0017274Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:47.0021110Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:47.0051829Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:12:48.9331354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:12:50.2695846Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:12:50.3228889Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:50.9653780Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:12:50.9796933Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:12:51.2889172Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:12:51.2912359Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:51.2916282Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:51.2949210Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:12:51.7903487Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:12:52.8269877Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:12:52.9081480Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:12:52.9197670Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:12:52.9351470Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:12:52.9793994Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:12:53.0139257Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:12:53.0624874Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:12:53.0963582Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:12:53.0987599Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:53.0990888Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:53.0994517Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:53.0998392Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:53.1031144Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:12:54.1789978Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:12:54.3985206Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:12:55.6365362Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:12:56.5942463Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:12:56.6135850Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:12:56.6926935Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:12:56.7268968Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:12:56.7286794Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:56.7318269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:12:56.9465424Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:12:56.9513258Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:12:56.9877585Z Entering 'android/libs/fbjni' 2024-08-20T22:12:56.9932031Z Entering 'third_party/FP16' 2024-08-20T22:12:56.9985644Z Entering 'third_party/FXdiv' 2024-08-20T22:12:57.0037350Z Entering 'third_party/NNPACK' 2024-08-20T22:12:57.0089332Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:57.0140551Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:57.0206917Z Entering 'third_party/benchmark' 2024-08-20T22:12:57.0259182Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:57.0310663Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:57.0363351Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:57.0414632Z Entering 'third_party/cutlass' 2024-08-20T22:12:57.0474457Z Entering 'third_party/eigen' 2024-08-20T22:12:57.0526242Z Entering 'third_party/fbgemm' 2024-08-20T22:12:57.0576882Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:57.0627953Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:57.0682564Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:57.0744335Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:57.0795078Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:57.0849424Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:57.0904413Z Entering 'third_party/fmt' 2024-08-20T22:12:57.0954687Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:57.1004933Z Entering 'third_party/gloo' 2024-08-20T22:12:57.1055730Z Entering 'third_party/googletest' 2024-08-20T22:12:57.1105895Z Entering 'third_party/ideep' 2024-08-20T22:12:57.1154706Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:57.1214120Z Entering 'third_party/ittapi' 2024-08-20T22:12:57.1266409Z Entering 'third_party/kineto' 2024-08-20T22:12:57.1315700Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:57.1365462Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:57.1416733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:57.1466504Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:57.1516958Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:57.1564840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:57.1618241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:57.1668659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:57.1719508Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:57.1771842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:57.1825229Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:57.1876316Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:57.1933102Z Entering 'third_party/mimalloc' 2024-08-20T22:12:57.1988151Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:57.2039454Z Entering 'third_party/nlohmann' 2024-08-20T22:12:57.2091238Z Entering 'third_party/onnx' 2024-08-20T22:12:57.2155132Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:57.2204880Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:57.2263510Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:57.2313981Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:57.2362518Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:57.2412439Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:57.2466651Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:57.2516876Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:57.2566262Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:57.2619163Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:57.2668188Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:57.2720411Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:57.2773986Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:57.2847181Z Entering 'third_party/pocketfft' 2024-08-20T22:12:57.2897562Z Entering 'third_party/protobuf' 2024-08-20T22:12:57.2950835Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:57.3000835Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:57.3061064Z Entering 'third_party/psimd' 2024-08-20T22:12:57.3111870Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:57.3162735Z Entering 'third_party/pybind11' 2024-08-20T22:12:57.3213813Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:57.3268194Z Entering 'third_party/sleef' 2024-08-20T22:12:57.3319705Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:57.3371003Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:57.3421337Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:57.3471901Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:57.3521521Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:57.3569636Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:57.3643978Z ##[endgroup] 2024-08-20T22:12:57.3646261Z ##[group]Persisting credentials for submodules 2024-08-20T22:12:57.3648115Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:12:57.4011567Z Entering 'android/libs/fbjni' 2024-08-20T22:12:57.4078571Z Entering 'third_party/FP16' 2024-08-20T22:12:57.4146431Z Entering 'third_party/FXdiv' 2024-08-20T22:12:57.4213065Z Entering 'third_party/NNPACK' 2024-08-20T22:12:57.4280938Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:57.4347418Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:57.4430752Z Entering 'third_party/benchmark' 2024-08-20T22:12:57.4498479Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:57.4565562Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:57.4638206Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:57.4706959Z Entering 'third_party/cutlass' 2024-08-20T22:12:57.4781683Z Entering 'third_party/eigen' 2024-08-20T22:12:57.4851959Z Entering 'third_party/fbgemm' 2024-08-20T22:12:57.4921059Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:57.4987844Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:57.5055066Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:57.5127377Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:57.5192548Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:57.5260790Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:57.5329477Z Entering 'third_party/fmt' 2024-08-20T22:12:57.5398140Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:57.5466909Z Entering 'third_party/gloo' 2024-08-20T22:12:57.5534751Z Entering 'third_party/googletest' 2024-08-20T22:12:57.5603138Z Entering 'third_party/ideep' 2024-08-20T22:12:57.5674827Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:57.5748284Z Entering 'third_party/ittapi' 2024-08-20T22:12:57.5814686Z Entering 'third_party/kineto' 2024-08-20T22:12:57.5879947Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:57.5944611Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:57.6016796Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:57.6088815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:57.6156685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:57.6222086Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:57.6293941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:57.6366367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:57.6437667Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:57.6508973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:57.6578438Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:57.6645350Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:57.6714166Z Entering 'third_party/mimalloc' 2024-08-20T22:12:57.6785276Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:57.6852922Z Entering 'third_party/nlohmann' 2024-08-20T22:12:57.6924581Z Entering 'third_party/onnx' 2024-08-20T22:12:57.7004075Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:57.7071850Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:57.7143961Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:57.7215259Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:57.7282078Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:57.7348228Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:57.7413986Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:57.7481575Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:57.7547070Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:57.7614677Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:57.7678416Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:57.7747229Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:57.7817714Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:57.7905904Z Entering 'third_party/pocketfft' 2024-08-20T22:12:57.7972496Z Entering 'third_party/protobuf' 2024-08-20T22:12:57.8040566Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:57.8105970Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:57.8174701Z Entering 'third_party/psimd' 2024-08-20T22:12:57.8246326Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:57.8317950Z Entering 'third_party/pybind11' 2024-08-20T22:12:57.8384882Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:57.8451554Z Entering 'third_party/sleef' 2024-08-20T22:12:57.8517860Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:57.8584490Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:57.8651171Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:57.8718859Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:57.8784851Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:57.8848415Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:57.8944290Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:12:57.9312642Z Entering 'android/libs/fbjni' 2024-08-20T22:12:57.9378339Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:12:57.9399293Z Entering 'third_party/FP16' 2024-08-20T22:12:57.9460738Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:12:57.9481713Z Entering 'third_party/FXdiv' 2024-08-20T22:12:57.9542681Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:12:57.9572369Z Entering 'third_party/NNPACK' 2024-08-20T22:12:57.9626091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:12:57.9647127Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:57.9711656Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:12:57.9731862Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:57.9796715Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:12:57.9834159Z Entering 'third_party/benchmark' 2024-08-20T22:12:57.9898202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:57.9919283Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:57.9980698Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:12:58.0001870Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:58.0069740Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:12:58.0091350Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:58.0157869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:12:58.0179815Z Entering 'third_party/cutlass' 2024-08-20T22:12:58.0244599Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:12:58.0275467Z Entering 'third_party/eigen' 2024-08-20T22:12:58.0337802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:12:58.0362571Z Entering 'third_party/fbgemm' 2024-08-20T22:12:58.0425867Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:12:58.0446308Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:58.0507858Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:12:58.0528110Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:58.0589909Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:12:58.0611306Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:58.0672721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:12:58.0700431Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:58.0761256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.0781934Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:58.0842027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:12:58.0867472Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:58.0928564Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:12:58.0952989Z Entering 'third_party/fmt' 2024-08-20T22:12:58.1014121Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.1035702Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:58.1097933Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:12:58.1119497Z Entering 'third_party/gloo' 2024-08-20T22:12:58.1183568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:12:58.1205992Z Entering 'third_party/googletest' 2024-08-20T22:12:58.1269627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.1292096Z Entering 'third_party/ideep' 2024-08-20T22:12:58.1354705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:12:58.1374289Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:58.1434321Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:12:58.1464934Z Entering 'third_party/ittapi' 2024-08-20T22:12:58.1525691Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:12:58.1547484Z Entering 'third_party/kineto' 2024-08-20T22:12:58.1609910Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:12:58.1630090Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:58.1698079Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:12:58.1716774Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:58.1779734Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:12:58.1801706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:58.1869401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:12:58.1890150Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:58.1952102Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.1972399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:58.2039288Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:12:58.2058314Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:58.2121050Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:12:58.2143868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:58.2208304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:12:58.2227055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:58.2289438Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.2309760Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:58.2372287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:12:58.2393513Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:58.2456171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:12:58.2479372Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:58.2541299Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:12:58.2562869Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:58.2623666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.2647906Z Entering 'third_party/mimalloc' 2024-08-20T22:12:58.2709644Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:12:58.2731245Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:58.2795301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:12:58.2815037Z Entering 'third_party/nlohmann' 2024-08-20T22:12:58.2877436Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:12:58.2900016Z Entering 'third_party/onnx' 2024-08-20T22:12:58.2962752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:12:58.2996398Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:58.3057856Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.3078273Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:58.3139976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:58.3166367Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:58.3229331Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:12:58.3250681Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:58.3310850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.3330527Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:58.3392122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.3416792Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:58.3477395Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:12:58.3497122Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:58.3558393Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:12:58.3579766Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:58.3640447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:12:58.3660027Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:58.3721115Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:12:58.3740576Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:58.3804538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:12:58.3825479Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:58.3887179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:12:58.3909479Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:58.3972135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:12:58.3994818Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:58.4061094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:12:58.4101950Z Entering 'third_party/pocketfft' 2024-08-20T22:12:58.4164480Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:12:58.4185606Z Entering 'third_party/protobuf' 2024-08-20T22:12:58.4251227Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:12:58.4273197Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:58.4334134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:12:58.4354424Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:58.4414821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.4438026Z Entering 'third_party/psimd' 2024-08-20T22:12:58.4503127Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:12:58.4523997Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:58.4586863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:12:58.4608121Z Entering 'third_party/pybind11' 2024-08-20T22:12:58.4669246Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:58.4690932Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:58.4752603Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:12:58.4773851Z Entering 'third_party/sleef' 2024-08-20T22:12:58.4834649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:12:58.4856361Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:58.4917829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:12:58.4937475Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:58.4998862Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:12:58.5018683Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:58.5079731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:12:58.5099454Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:58.5160522Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:12:58.5180935Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:58.5242032Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:12:58.5260255Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:58.5321274Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:12:58.6221781Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:12:58.6586915Z Entering 'android/libs/fbjni' 2024-08-20T22:12:58.6638506Z Entering 'third_party/FP16' 2024-08-20T22:12:58.6691163Z Entering 'third_party/FXdiv' 2024-08-20T22:12:58.6743096Z Entering 'third_party/NNPACK' 2024-08-20T22:12:58.6796833Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:58.6850245Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:58.6915191Z Entering 'third_party/benchmark' 2024-08-20T22:12:58.6966611Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:58.7017710Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:58.7070602Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:58.7122721Z Entering 'third_party/cutlass' 2024-08-20T22:12:58.7182754Z Entering 'third_party/eigen' 2024-08-20T22:12:58.7236514Z Entering 'third_party/fbgemm' 2024-08-20T22:12:58.7290114Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:58.7345195Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:58.7395635Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:58.7452828Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:58.7506421Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:58.7558797Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:58.7612484Z Entering 'third_party/fmt' 2024-08-20T22:12:58.7665282Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:58.7716329Z Entering 'third_party/gloo' 2024-08-20T22:12:58.7769128Z Entering 'third_party/googletest' 2024-08-20T22:12:58.7819555Z Entering 'third_party/ideep' 2024-08-20T22:12:58.7869399Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:58.7927616Z Entering 'third_party/ittapi' 2024-08-20T22:12:58.7981567Z Entering 'third_party/kineto' 2024-08-20T22:12:58.8032387Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:58.8082842Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:58.8134754Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:58.8184860Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:58.8235893Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:58.8282836Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:58.8342051Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:58.8392016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:58.8441509Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:58.8494180Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:58.8546363Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:58.8597260Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:58.8651265Z Entering 'third_party/mimalloc' 2024-08-20T22:12:58.8703222Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:58.8755888Z Entering 'third_party/nlohmann' 2024-08-20T22:12:58.8810460Z Entering 'third_party/onnx' 2024-08-20T22:12:58.8874719Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:58.8925503Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:58.8981655Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:58.9034478Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:58.9085274Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:58.9138701Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:58.9189096Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:58.9240212Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:58.9290683Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:58.9340115Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:58.9389874Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:58.9441170Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:58.9495539Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:58.9570508Z Entering 'third_party/pocketfft' 2024-08-20T22:12:58.9622461Z Entering 'third_party/protobuf' 2024-08-20T22:12:58.9676941Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:58.9730012Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:58.9784744Z Entering 'third_party/psimd' 2024-08-20T22:12:58.9837160Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:58.9889979Z Entering 'third_party/pybind11' 2024-08-20T22:12:58.9941918Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:58.9995091Z Entering 'third_party/sleef' 2024-08-20T22:12:59.0047982Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:59.0099108Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:59.0149375Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:59.0199165Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:59.0249599Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:59.0298297Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:59.0375259Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:12:59.0735870Z Entering 'android/libs/fbjni' 2024-08-20T22:12:59.0787796Z Entering 'third_party/FP16' 2024-08-20T22:12:59.0839535Z Entering 'third_party/FXdiv' 2024-08-20T22:12:59.0891909Z Entering 'third_party/NNPACK' 2024-08-20T22:12:59.0943814Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:12:59.0997392Z Entering 'third_party/XNNPACK' 2024-08-20T22:12:59.1064949Z Entering 'third_party/benchmark' 2024-08-20T22:12:59.1116243Z Entering 'third_party/cpp-httplib' 2024-08-20T22:12:59.1169997Z Entering 'third_party/cpuinfo' 2024-08-20T22:12:59.1222336Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:12:59.1274942Z Entering 'third_party/cutlass' 2024-08-20T22:12:59.1335838Z Entering 'third_party/eigen' 2024-08-20T22:12:59.1390694Z Entering 'third_party/fbgemm' 2024-08-20T22:12:59.1442111Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:59.1493808Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:59.1548596Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:59.1609581Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:59.1665663Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:59.1718585Z Entering 'third_party/flatbuffers' 2024-08-20T22:12:59.1772754Z Entering 'third_party/fmt' 2024-08-20T22:12:59.1823615Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:12:59.1875537Z Entering 'third_party/gloo' 2024-08-20T22:12:59.1926479Z Entering 'third_party/googletest' 2024-08-20T22:12:59.1978256Z Entering 'third_party/ideep' 2024-08-20T22:12:59.2026904Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:59.2085521Z Entering 'third_party/ittapi' 2024-08-20T22:12:59.2137589Z Entering 'third_party/kineto' 2024-08-20T22:12:59.2189107Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:59.2238427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:59.2291167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:59.2341637Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:59.2393692Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:59.2443027Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:59.2497450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:59.2548296Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:59.2599267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:59.2651409Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:59.2704597Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:59.2755251Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:59.2808023Z Entering 'third_party/mimalloc' 2024-08-20T22:12:59.2863926Z Entering 'third_party/nccl/nccl' 2024-08-20T22:12:59.2915813Z Entering 'third_party/nlohmann' 2024-08-20T22:12:59.2968772Z Entering 'third_party/onnx' 2024-08-20T22:12:59.3034077Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:59.3095927Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:59.3154363Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:12:59.3206948Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:59.3260254Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:59.3310307Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:59.3360727Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:59.3412084Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:59.3462106Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:59.3511218Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:59.3560164Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:12:59.3612926Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:12:59.3666438Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:59.3737525Z Entering 'third_party/pocketfft' 2024-08-20T22:12:59.3789890Z Entering 'third_party/protobuf' 2024-08-20T22:12:59.3842480Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:12:59.3896190Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:12:59.3953640Z Entering 'third_party/psimd' 2024-08-20T22:12:59.4005181Z Entering 'third_party/pthreadpool' 2024-08-20T22:12:59.4056636Z Entering 'third_party/pybind11' 2024-08-20T22:12:59.4107455Z Entering 'third_party/python-peachpy' 2024-08-20T22:12:59.4158523Z Entering 'third_party/sleef' 2024-08-20T22:12:59.4209067Z Entering 'third_party/tensorpipe' 2024-08-20T22:12:59.4260551Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:12:59.4310265Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:12:59.4360064Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:12:59.4409669Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:12:59.4458732Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:12:59.4531080Z ##[endgroup] 2024-08-20T22:12:59.4581291Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:12:59.4616978Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:12:59.4837760Z Prepare all required actions 2024-08-20T22:12:59.4838212Z Getting action download info 2024-08-20T22:12:59.6188201Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:12:59.6188634Z env: 2024-08-20T22:12:59.6188921Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:12:59.6189266Z ##[endgroup] 2024-08-20T22:12:59.6248439Z ##[group]Run set -euo pipefail 2024-08-20T22:12:59.6248860Z set -euo pipefail 2024-08-20T22:12:59.6249249Z function get_ec2_metadata() { 2024-08-20T22:12:59.6249781Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:12:59.6250660Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:12:59.6251409Z  category=$1 2024-08-20T22:12:59.6251955Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:12:59.6252584Z  runner_name_str=i-06bc7425c52774f50 2024-08-20T22:12:59.6253073Z  if [[ -f /.inarc ]]; then 2024-08-20T22:12:59.6253572Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:12:59.6254135Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:12:59.6254816Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:12:59.6255409Z  else 2024-08-20T22:12:59.6255902Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:12:59.6256469Z  fi 2024-08-20T22:12:59.6256764Z } 2024-08-20T22:12:59.6257122Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:12:59.6257705Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:12:59.6258350Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:12:59.6258922Z echo "system info $(uname -a)" 2024-08-20T22:12:59.6271394Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:12:59.6271900Z env: 2024-08-20T22:12:59.6272192Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:12:59.6272552Z ##[endgroup] 2024-08-20T22:12:59.6371365Z ami-id: ami-06c68f701d8090592 2024-08-20T22:12:59.6434172Z instance-id: i-06bc7425c52774f50 2024-08-20T22:12:59.6490530Z instance-type: g5.4xlarge 2024-08-20T22:12:59.6503982Z system info Linux ip-10-0-22-103.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:12:59.6528000Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:12:59.6528917Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:12:59.6538075Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:12:59.6538750Z env: 2024-08-20T22:12:59.6539045Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:12:59.6539404Z ##[endgroup] 2024-08-20T22:12:59.6643140Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:12:59.6643795Z if systemctl is-active --quiet docker; then 2024-08-20T22:12:59.6644325Z  echo "Docker daemon is running..."; 2024-08-20T22:12:59.6644771Z else 2024-08-20T22:12:59.6645254Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:12:59.6646004Z fi 2024-08-20T22:12:59.6654777Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:12:59.6655274Z env: 2024-08-20T22:12:59.6655567Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:12:59.6655925Z ##[endgroup] 2024-08-20T22:12:59.6747243Z Docker daemon is running... 2024-08-20T22:12:59.6795067Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:12:59.6795631Z with: 2024-08-20T22:12:59.6795920Z shell: bash 2024-08-20T22:12:59.6796218Z timeout_minutes: 5 2024-08-20T22:12:59.6796548Z max_attempts: 3 2024-08-20T22:12:59.6796870Z retry_wait_seconds: 30 2024-08-20T22:12:59.6800078Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:12:59.6803012Z polling_interval_seconds: 1 2024-08-20T22:12:59.6803473Z warning_on_retry: true 2024-08-20T22:12:59.6803835Z continue_on_error: false 2024-08-20T22:12:59.6804195Z env: 2024-08-20T22:12:59.6804483Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:12:59.6804851Z AWS_RETRY_MODE: standard 2024-08-20T22:12:59.6805205Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:12:59.6805558Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:12:59.6805946Z ##[endgroup] 2024-08-20T22:13:00.8314349Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:00.8315173Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:00.8315993Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:00.8316495Z 2024-08-20T22:13:00.8316614Z Login Succeeded 2024-08-20T22:13:01.7358588Z Command completed after 1 attempt(s). 2024-08-20T22:13:01.7420969Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:01.7421699Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:01.7422358Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:01.7432381Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:01.7432874Z env: 2024-08-20T22:13:01.7433157Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:01.7433510Z ##[endgroup] 2024-08-20T22:13:01.7540834Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:01.7541977Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:01.7542832Z # shellcheck disable=SC2046 2024-08-20T22:13:01.7543536Z docker stop $(docker ps -q) || true 2024-08-20T22:13:01.7544254Z # Prune all of the docker images 2024-08-20T22:13:01.7544909Z docker system prune -af 2024-08-20T22:13:01.7556889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:01.7557599Z env: 2024-08-20T22:13:01.7558003Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:01.7558502Z ##[endgroup] 2024-08-20T22:13:01.7867257Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:01.7868554Z See 'docker stop --help'. 2024-08-20T22:13:01.7868864Z 2024-08-20T22:13:01.7869157Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:01.7869643Z 2024-08-20T22:13:01.7869838Z Stop one or more running containers 2024-08-20T22:13:01.8288719Z Total reclaimed space: 0B 2024-08-20T22:13:01.8335738Z ##[group]Run set +e 2024-08-20T22:13:01.8336105Z set +e 2024-08-20T22:13:01.8336423Z set -x 2024-08-20T22:13:01.8336720Z  2024-08-20T22:13:01.8337039Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:01.8337815Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:01.8338876Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:01.8339618Z # one is returned at random 2024-08-20T22:13:01.8340165Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:01.8340682Z  2024-08-20T22:13:01.8341037Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:01.8341658Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:01.8342614Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:01.8343176Z  2024-08-20T22:13:01.8343518Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:01.8344078Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:01.8344594Z  exit 1 2024-08-20T22:13:01.8344914Z  fi 2024-08-20T22:13:01.8345217Z fi 2024-08-20T22:13:01.8345512Z  2024-08-20T22:13:01.8346256Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:01.8346771Z  # Clean up any old records first 2024-08-20T22:13:01.8347279Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:01.8347729Z fi 2024-08-20T22:13:01.8348008Z  2024-08-20T22:13:01.8348447Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:01.8348990Z cat /etc/hosts 2024-08-20T22:13:01.8357982Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:01.8358484Z env: 2024-08-20T22:13:01.8358786Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:01.8359142Z ##[endgroup] 2024-08-20T22:13:01.8388266Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:01.8394633Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:01.8395099Z ++ tail -n1 2024-08-20T22:13:01.8883033Z + RESOLVED_IP=18.160.10.28 2024-08-20T22:13:01.8883593Z + '[' -z 18.160.10.28 ']' 2024-08-20T22:13:01.8884037Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:01.8902382Z + echo '18.160.10.28 download.pytorch.org' 2024-08-20T22:13:01.8903236Z + sudo tee -a /etc/hosts 2024-08-20T22:13:02.1323378Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:02.1346838Z + cat /etc/hosts 2024-08-20T22:13:02.1357717Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:02.1370613Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:02.1371116Z 18.160.10.28 download.pytorch.org 2024-08-20T22:13:02.1522514Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:02.1523116Z with: 2024-08-20T22:13:02.1524222Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1525450Z docker-build-dir: .ci/docker 2024-08-20T22:13:02.1525839Z working-directory: . 2024-08-20T22:13:02.1526310Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.1526832Z force-push: false 2024-08-20T22:13:02.1527211Z env: 2024-08-20T22:13:02.1527497Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.1527840Z ##[endgroup] 2024-08-20T22:13:02.1547213Z ##[group]Run set -ex 2024-08-20T22:13:02.1547597Z set -ex 2024-08-20T22:13:02.1547901Z  2024-08-20T22:13:02.1548637Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:02.1549610Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:02.1550409Z # job could then download the pre-built image as usual 2024-08-20T22:13:02.1551152Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:02.1551828Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1552469Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1553030Z  2024-08-20T22:13:02.1553542Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:02.1554173Z  exit 0 2024-08-20T22:13:02.1554492Z else 2024-08-20T22:13:02.1554865Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1555330Z fi 2024-08-20T22:13:02.1555623Z  2024-08-20T22:13:02.1556099Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:02.1556943Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:02.1557719Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:02.1558426Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:02.1559148Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1559843Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1560406Z else 2024-08-20T22:13:02.1560854Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:02.1561518Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1562410Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1563157Z fi 2024-08-20T22:13:02.1574379Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.1574897Z env: 2024-08-20T22:13:02.1575193Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.1575561Z REPO_NAME: pytorch 2024-08-20T22:13:02.1576713Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1577913Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:02.1578419Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.1578934Z ##[endgroup] 2024-08-20T22:13:02.1609907Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:02.1610386Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:02.1610759Z + echo skip=false 2024-08-20T22:13:02.1612756Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:02.1617525Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1618805Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:02.1643789Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1644514Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1646467Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1678868Z ##[group]Run set +e 2024-08-20T22:13:02.1679372Z set +e 2024-08-20T22:13:02.1679863Z set -x 2024-08-20T22:13:02.1680268Z  2024-08-20T22:13:02.1680804Z login() { 2024-08-20T22:13:02.1681653Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:02.1682475Z } 2024-08-20T22:13:02.1682890Z  2024-08-20T22:13:02.1683318Z retry () { 2024-08-20T22:13:02.1683864Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:02.1684451Z } 2024-08-20T22:13:02.1684854Z  2024-08-20T22:13:02.1685304Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:02.1685827Z  2024-08-20T22:13:02.1686455Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:02.1687333Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:02.1687942Z  exit 0 2024-08-20T22:13:02.1688413Z fi 2024-08-20T22:13:02.1688821Z  2024-08-20T22:13:02.1689394Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:02.1690385Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:02.1691258Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:02.1692106Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:02.1692788Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:02.1693327Z else 2024-08-20T22:13:02.1693960Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:02.1694777Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:02.1695347Z fi 2024-08-20T22:13:02.1695782Z  2024-08-20T22:13:02.1696231Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:02.1696838Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1697416Z  2024-08-20T22:13:02.1698189Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:02.1699051Z  exit 0 2024-08-20T22:13:02.1699528Z fi 2024-08-20T22:13:02.1699886Z  2024-08-20T22:13:02.1700439Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:02.1701578Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:02.1702479Z  exit 1 2024-08-20T22:13:02.1702855Z fi 2024-08-20T22:13:02.1703380Z  2024-08-20T22:13:02.1703970Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:02.1705045Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:02.1706030Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:02.1707095Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:02.1708325Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:02.1709297Z fi 2024-08-20T22:13:02.1709725Z  2024-08-20T22:13:02.1710157Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:02.1718677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:02.1719446Z env: 2024-08-20T22:13:02.1719841Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:02.1720310Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:02.1720953Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:02.1722371Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1723740Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.1724485Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.1725247Z ##[endgroup] 2024-08-20T22:13:02.1755574Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.1756363Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.1758770Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:02.1760233Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:02.7054837Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:02.7056004Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:02.7057063Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:02.7057712Z 2024-08-20T22:13:02.7057901Z Login Succeeded 2024-08-20T22:13:02.7079869Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:02.9781219Z { 2024-08-20T22:13:02.9781687Z "schemaVersion": 2, 2024-08-20T22:13:02.9782558Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:02.9783904Z "config": { 2024-08-20T22:13:02.9784452Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:02.9785071Z "size": 48711, 2024-08-20T22:13:02.9786786Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:02.9787741Z }, 2024-08-20T22:13:02.9788381Z "layers": [ 2024-08-20T22:13:02.9788782Z { 2024-08-20T22:13:02.9789874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9790527Z "size": 28580681, 2024-08-20T22:13:02.9791686Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:02.9792685Z }, 2024-08-20T22:13:02.9793475Z { 2024-08-20T22:13:02.9794428Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9795062Z "size": 7943451, 2024-08-20T22:13:02.9795876Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:02.9797525Z }, 2024-08-20T22:13:02.9797988Z { 2024-08-20T22:13:02.9798975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9800044Z "size": 55728572, 2024-08-20T22:13:02.9801014Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:02.9801843Z }, 2024-08-20T22:13:02.9802862Z { 2024-08-20T22:13:02.9803460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9804309Z "size": 186, 2024-08-20T22:13:02.9805104Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:02.9805787Z }, 2024-08-20T22:13:02.9806313Z { 2024-08-20T22:13:02.9806907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9807857Z "size": 6886, 2024-08-20T22:13:02.9808652Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:02.9809525Z }, 2024-08-20T22:13:02.9810265Z { 2024-08-20T22:13:02.9810931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9811879Z "size": 1329001756, 2024-08-20T22:13:02.9812818Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:02.9813759Z }, 2024-08-20T22:13:02.9814168Z { 2024-08-20T22:13:02.9815174Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9816161Z "size": 62501, 2024-08-20T22:13:02.9817457Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:02.9818442Z }, 2024-08-20T22:13:02.9819087Z { 2024-08-20T22:13:02.9819895Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9821199Z "size": 1684, 2024-08-20T22:13:02.9822212Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:02.9823472Z }, 2024-08-20T22:13:02.9823964Z { 2024-08-20T22:13:02.9825003Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9826184Z + exit 0 2024-08-20T22:13:02.9826832Z "size": 1523, 2024-08-20T22:13:02.9827795Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:02.9828671Z }, 2024-08-20T22:13:02.9829132Z { 2024-08-20T22:13:02.9829662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9830308Z "size": 2528295403, 2024-08-20T22:13:02.9831000Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:02.9831707Z }, 2024-08-20T22:13:02.9832052Z { 2024-08-20T22:13:02.9832658Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9833346Z "size": 86016, 2024-08-20T22:13:02.9834006Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:02.9834807Z }, 2024-08-20T22:13:02.9835152Z { 2024-08-20T22:13:02.9835665Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9836361Z "size": 1822, 2024-08-20T22:13:02.9836992Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:02.9837677Z }, 2024-08-20T22:13:02.9838074Z { 2024-08-20T22:13:02.9838597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9839268Z "size": 246761563, 2024-08-20T22:13:02.9840013Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:02.9840735Z }, 2024-08-20T22:13:02.9841113Z { 2024-08-20T22:13:02.9841688Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9842317Z "size": 544, 2024-08-20T22:13:02.9842930Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:02.9843691Z }, 2024-08-20T22:13:02.9844033Z { 2024-08-20T22:13:02.9844580Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9845286Z "size": 1258, 2024-08-20T22:13:02.9846340Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:02.9847080Z }, 2024-08-20T22:13:02.9847577Z { 2024-08-20T22:13:02.9848127Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9848828Z "size": 484, 2024-08-20T22:13:02.9849435Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:02.9850134Z }, 2024-08-20T22:13:02.9850546Z { 2024-08-20T22:13:02.9851101Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9851722Z "size": 91711766, 2024-08-20T22:13:02.9852432Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:02.9853168Z }, 2024-08-20T22:13:02.9853484Z { 2024-08-20T22:13:02.9854148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9854974Z "size": 3390, 2024-08-20T22:13:02.9855632Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:02.9856380Z }, 2024-08-20T22:13:02.9856738Z { 2024-08-20T22:13:02.9857280Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9857995Z "size": 1909, 2024-08-20T22:13:02.9858577Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:02.9859286Z }, 2024-08-20T22:13:02.9859683Z { 2024-08-20T22:13:02.9860172Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9860835Z "size": 701, 2024-08-20T22:13:02.9861585Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:02.9862298Z }, 2024-08-20T22:13:02.9862766Z { 2024-08-20T22:13:02.9863350Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9864009Z "size": 2864254879, 2024-08-20T22:13:02.9864724Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:02.9865447Z }, 2024-08-20T22:13:02.9865808Z { 2024-08-20T22:13:02.9866337Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9867021Z "size": 380, 2024-08-20T22:13:02.9867635Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:02.9868398Z }, 2024-08-20T22:13:02.9868806Z { 2024-08-20T22:13:02.9869295Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9869943Z "size": 12900, 2024-08-20T22:13:02.9870675Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:02.9871412Z }, 2024-08-20T22:13:02.9871816Z { 2024-08-20T22:13:02.9872371Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9872973Z "size": 803, 2024-08-20T22:13:02.9873660Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:02.9874381Z }, 2024-08-20T22:13:02.9874825Z { 2024-08-20T22:13:02.9875410Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9876053Z "size": 106, 2024-08-20T22:13:02.9876676Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:02.9877453Z }, 2024-08-20T22:13:02.9877769Z { 2024-08-20T22:13:02.9878297Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9879031Z "size": 503, 2024-08-20T22:13:02.9879611Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:02.9880312Z }, 2024-08-20T22:13:02.9880731Z { 2024-08-20T22:13:02.9881335Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9881977Z "size": 121477267, 2024-08-20T22:13:02.9882671Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:02.9883364Z }, 2024-08-20T22:13:02.9883725Z { 2024-08-20T22:13:02.9884312Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9884948Z "size": 109, 2024-08-20T22:13:02.9885597Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:02.9886311Z }, 2024-08-20T22:13:02.9886647Z { 2024-08-20T22:13:02.9887290Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9887997Z "size": 490, 2024-08-20T22:13:02.9888678Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:02.9889376Z }, 2024-08-20T22:13:02.9889768Z { 2024-08-20T22:13:02.9890317Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9890929Z "size": 297, 2024-08-20T22:13:02.9891608Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:02.9892443Z }, 2024-08-20T22:13:02.9892754Z { 2024-08-20T22:13:02.9893356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9893996Z "size": 103, 2024-08-20T22:13:02.9894664Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:02.9895422Z }, 2024-08-20T22:13:02.9895758Z { 2024-08-20T22:13:02.9896327Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9897005Z "size": 1473, 2024-08-20T22:13:02.9897611Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:02.9898397Z }, 2024-08-20T22:13:02.9898735Z { 2024-08-20T22:13:02.9899260Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9899945Z "size": 594729516, 2024-08-20T22:13:02.9900690Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:02.9901416Z }, 2024-08-20T22:13:02.9901794Z { 2024-08-20T22:13:02.9902403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9903074Z "size": 164, 2024-08-20T22:13:02.9903779Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:02.9904502Z }, 2024-08-20T22:13:02.9904876Z { 2024-08-20T22:13:02.9905473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9906125Z "size": 564, 2024-08-20T22:13:02.9906747Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:02.9907511Z }, 2024-08-20T22:13:02.9907893Z { 2024-08-20T22:13:02.9908387Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9909165Z "size": 35874203, 2024-08-20T22:13:02.9909849Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:02.9910579Z }, 2024-08-20T22:13:02.9910943Z { 2024-08-20T22:13:02.9911491Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9912197Z "size": 104, 2024-08-20T22:13:02.9912886Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:02.9913542Z }, 2024-08-20T22:13:02.9913880Z { 2024-08-20T22:13:02.9914473Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9915167Z "size": 425, 2024-08-20T22:13:02.9915857Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:02.9916659Z }, 2024-08-20T22:13:02.9916999Z { 2024-08-20T22:13:02.9917487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9918198Z "size": 20262183, 2024-08-20T22:13:02.9918857Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:02.9919635Z }, 2024-08-20T22:13:02.9919950Z { 2024-08-20T22:13:02.9920486Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9921193Z "size": 438, 2024-08-20T22:13:02.9921828Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:02.9922493Z }, 2024-08-20T22:13:02.9923001Z { 2024-08-20T22:13:02.9923533Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9924137Z "size": 701, 2024-08-20T22:13:02.9924884Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:02.9925612Z }, 2024-08-20T22:13:02.9925983Z { 2024-08-20T22:13:02.9926526Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9927273Z "size": 141, 2024-08-20T22:13:02.9927950Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:02.9928723Z }, 2024-08-20T22:13:02.9929032Z { 2024-08-20T22:13:02.9929651Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9930451Z "size": 135, 2024-08-20T22:13:02.9931106Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:02.9931772Z }, 2024-08-20T22:13:02.9932176Z { 2024-08-20T22:13:02.9932740Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9933354Z "size": 4838068375, 2024-08-20T22:13:02.9934061Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:02.9934787Z }, 2024-08-20T22:13:02.9935132Z { 2024-08-20T22:13:02.9935672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9936413Z "size": 194, 2024-08-20T22:13:02.9937046Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:02.9937826Z }, 2024-08-20T22:13:02.9938230Z { 2024-08-20T22:13:02.9938767Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9939502Z "size": 564, 2024-08-20T22:13:02.9940156Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:02.9940852Z }, 2024-08-20T22:13:02.9941245Z { 2024-08-20T22:13:02.9941773Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9942403Z "size": 43162921, 2024-08-20T22:13:02.9943116Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:02.9943885Z }, 2024-08-20T22:13:02.9944307Z { 2024-08-20T22:13:02.9944805Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9945459Z "size": 106, 2024-08-20T22:13:02.9946424Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:02.9947112Z }, 2024-08-20T22:13:02.9947435Z { 2024-08-20T22:13:02.9948007Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9948650Z "size": 1380, 2024-08-20T22:13:02.9949261Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:02.9950026Z }, 2024-08-20T22:13:02.9950436Z { 2024-08-20T22:13:02.9950972Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9951620Z "size": 701, 2024-08-20T22:13:02.9952243Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:02.9952975Z }, 2024-08-20T22:13:02.9953371Z { 2024-08-20T22:13:02.9953884Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9954519Z "size": 139, 2024-08-20T22:13:02.9955184Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:02.9955904Z }, 2024-08-20T22:13:02.9956203Z { 2024-08-20T22:13:02.9956764Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9957489Z "size": 118, 2024-08-20T22:13:02.9958078Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:02.9958822Z }, 2024-08-20T22:13:02.9959178Z { 2024-08-20T22:13:02.9959690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9960359Z "size": 4324672728, 2024-08-20T22:13:02.9961017Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:02.9961725Z }, 2024-08-20T22:13:02.9962128Z { 2024-08-20T22:13:02.9962614Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9963253Z "size": 174, 2024-08-20T22:13:02.9964007Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:02.9964660Z }, 2024-08-20T22:13:02.9964993Z { 2024-08-20T22:13:02.9965589Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9966237Z "size": 907, 2024-08-20T22:13:02.9966865Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:02.9967891Z }, 2024-08-20T22:13:02.9968242Z { 2024-08-20T22:13:02.9968848Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9969458Z "size": 701, 2024-08-20T22:13:02.9970110Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:02.9970921Z }, 2024-08-20T22:13:02.9971223Z { 2024-08-20T22:13:02.9971759Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9972455Z "size": 134, 2024-08-20T22:13:02.9973093Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:02.9973783Z }, 2024-08-20T22:13:02.9974170Z { 2024-08-20T22:13:02.9974685Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9975346Z "size": 32, 2024-08-20T22:13:02.9976127Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:02.9976860Z }, 2024-08-20T22:13:02.9977277Z { 2024-08-20T22:13:02.9977842Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9978462Z "size": 158, 2024-08-20T22:13:02.9979090Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:02.9979858Z }, 2024-08-20T22:13:02.9980158Z { 2024-08-20T22:13:02.9980671Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9981378Z "size": 1841, 2024-08-20T22:13:02.9982005Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:02.9982657Z }, 2024-08-20T22:13:02.9983060Z { 2024-08-20T22:13:02.9983573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9984311Z "size": 7529776, 2024-08-20T22:13:02.9984990Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:02.9985704Z }, 2024-08-20T22:13:02.9986069Z { 2024-08-20T22:13:02.9986610Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9987257Z "size": 163, 2024-08-20T22:13:02.9987903Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:02.9988673Z }, 2024-08-20T22:13:02.9988981Z { 2024-08-20T22:13:02.9989525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9990201Z "size": 7943, 2024-08-20T22:13:02.9990895Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:02.9991575Z }, 2024-08-20T22:13:02.9991960Z { 2024-08-20T22:13:02.9992496Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9993175Z "size": 8068, 2024-08-20T22:13:02.9993771Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:02.9994500Z }, 2024-08-20T22:13:02.9994905Z { 2024-08-20T22:13:02.9995417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9996070Z "size": 302, 2024-08-20T22:13:02.9996735Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:02.9997422Z }, 2024-08-20T22:13:02.9997802Z { 2024-08-20T22:13:02.9998370Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:02.9999023Z "size": 7629638, 2024-08-20T22:13:02.9999647Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:03.0000341Z }, 2024-08-20T22:13:03.0000697Z { 2024-08-20T22:13:03.0001211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0001894Z "size": 108, 2024-08-20T22:13:03.0002516Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:03.0003220Z }, 2024-08-20T22:13:03.0003617Z { 2024-08-20T22:13:03.0004124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0004927Z "size": 54145661, 2024-08-20T22:13:03.0005660Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:03.0006363Z }, 2024-08-20T22:13:03.0006662Z { 2024-08-20T22:13:03.0007351Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0007996Z "size": 473, 2024-08-20T22:13:03.0008657Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:03.0009392Z }, 2024-08-20T22:13:03.0009737Z { 2024-08-20T22:13:03.0010282Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0010965Z "size": 1374858766, 2024-08-20T22:13:03.0011689Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:03.0012411Z }, 2024-08-20T22:13:03.0012808Z { 2024-08-20T22:13:03.0013389Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0014102Z "size": 106, 2024-08-20T22:13:03.0014808Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:03.0015536Z }, 2024-08-20T22:13:03.0015834Z { 2024-08-20T22:13:03.0016397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0017053Z "size": 559, 2024-08-20T22:13:03.0017718Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:03.0018434Z }, 2024-08-20T22:13:03.0018789Z { 2024-08-20T22:13:03.0019356Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0019952Z "size": 46248488, 2024-08-20T22:13:03.0020618Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:03.0021368Z }, 2024-08-20T22:13:03.0021732Z { 2024-08-20T22:13:03.0022211Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0022898Z "size": 111, 2024-08-20T22:13:03.0023549Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:03.0024257Z }, 2024-08-20T22:13:03.0024935Z { 2024-08-20T22:13:03.0025634Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0026398Z "size": 32, 2024-08-20T22:13:03.0027126Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.0027784Z }, 2024-08-20T22:13:03.0028114Z { 2024-08-20T22:13:03.0028726Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0029333Z "size": 32, 2024-08-20T22:13:03.0029960Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.0030743Z }, 2024-08-20T22:13:03.0031085Z { 2024-08-20T22:13:03.0031598Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0032384Z "size": 32, 2024-08-20T22:13:03.0033008Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.0033791Z }, 2024-08-20T22:13:03.0034098Z { 2024-08-20T22:13:03.0034615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:03.0035358Z "size": 32, 2024-08-20T22:13:03.0035939Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:03.0036634Z } 2024-08-20T22:13:03.0037041Z ] 2024-08-20T22:13:03.0037372Z } 2024-08-20T22:13:03.0331810Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:03.0332344Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:03.0332893Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:03.0342164Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:03.0342662Z env: 2024-08-20T22:13:03.0342946Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.0344150Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.0345298Z ##[endgroup] 2024-08-20T22:13:03.0377327Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.0429789Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:03.0430374Z with: 2024-08-20T22:13:03.0431436Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.0432726Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.0433250Z env: 2024-08-20T22:13:03.0433542Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.0433894Z ##[endgroup] 2024-08-20T22:13:03.0451507Z ##[group]Run set -x 2024-08-20T22:13:03.0452011Z set -x 2024-08-20T22:13:03.0452306Z set +e 2024-08-20T22:13:03.0452598Z  2024-08-20T22:13:03.0452873Z login() { 2024-08-20T22:13:03.0453518Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:03.0454220Z } 2024-08-20T22:13:03.0454493Z  2024-08-20T22:13:03.0454798Z retry () { 2024-08-20T22:13:03.0455191Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:03.0455632Z } 2024-08-20T22:13:03.0455906Z  2024-08-20T22:13:03.0456223Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:03.0456629Z  2024-08-20T22:13:03.0456903Z set -e 2024-08-20T22:13:03.0457375Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:03.0458062Z # only pull docker image if it's not available locally 2024-08-20T22:13:03.0458806Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:03.0459494Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:03.0459931Z fi 2024-08-20T22:13:03.0468827Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:03.0469330Z env: 2024-08-20T22:13:03.0469618Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:03.0470741Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.0472019Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.0472526Z ##[endgroup] 2024-08-20T22:13:03.0504390Z + set +e 2024-08-20T22:13:03.0504931Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.0505580Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.0508262Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:03.0509603Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:03.5825909Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:03.5826738Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:03.5827567Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:03.5828100Z 2024-08-20T22:13:03.5828229Z Login Succeeded 2024-08-20T22:13:03.5849275Z + set -e 2024-08-20T22:13:03.5852725Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.6011909Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.6014021Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:03.8317887Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:03.8319173Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:03.8319674Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:03.8320646Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:03.8321173Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:03.8321650Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:03.8322025Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:03.8322471Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:03.8322986Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:03.8323491Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:03.8323961Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:03.8324486Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:03.8325008Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:03.8325540Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:03.8326265Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:03.8326763Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:03.8327417Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:03.8327890Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:03.8328303Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:03.8328776Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:03.8329426Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:03.8329948Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:03.8330463Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:03.8331015Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:03.8331520Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:03.8331953Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:03.8332421Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:03.8332787Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:03.8333154Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:03.8333521Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:03.8333881Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:03.8334250Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:03.8334611Z b57676e46aee: Waiting 2024-08-20T22:13:03.8334922Z a41a8d1c11c8: Waiting 2024-08-20T22:13:03.8335238Z a8c1e85b5e14: Waiting 2024-08-20T22:13:03.8335545Z 0c1227890755: Waiting 2024-08-20T22:13:03.8335868Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:03.8336314Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:03.8336686Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:03.8337100Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:03.8337465Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:03.8337860Z 019620471c25: Pulling fs layer 2024-08-20T22:13:03.8338225Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:03.8338580Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:03.8338904Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:03.8339256Z d527cbbb87e3: Waiting 2024-08-20T22:13:03.8339571Z ca426ffc9d2b: Waiting 2024-08-20T22:13:03.8339896Z ec1e7978c1fe: Waiting 2024-08-20T22:13:03.8340228Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:03.8340602Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:03.8340980Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:03.8341335Z 34c47d63c5de: Waiting 2024-08-20T22:13:03.8341651Z 0bf61114d37b: Waiting 2024-08-20T22:13:03.8341959Z 5645520256fa: Waiting 2024-08-20T22:13:03.8342285Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:03.8342635Z 7c47122f4e09: Waiting 2024-08-20T22:13:03.8342970Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:03.8343331Z 6efb96fc87ea: Waiting 2024-08-20T22:13:03.8343647Z 6658e558ebe4: Waiting 2024-08-20T22:13:03.8343968Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:03.8344335Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:03.8344705Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:03.8345068Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:03.8345419Z 1811ff4d3556: Waiting 2024-08-20T22:13:03.8346058Z b119c4fc6c58: Waiting 2024-08-20T22:13:03.8346392Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:03.8346739Z 385d0421d67c: Waiting 2024-08-20T22:13:03.8347133Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:03.8347509Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:03.8347862Z 1fef4bd19792: Waiting 2024-08-20T22:13:03.8348169Z 019620471c25: Waiting 2024-08-20T22:13:03.8348484Z 829ed7d91e38: Waiting 2024-08-20T22:13:03.8348811Z f957c1624481: Pulling fs layer 2024-08-20T22:13:03.8349169Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:03.8349734Z 5a4eb19d7219: Waiting 2024-08-20T22:13:03.8350054Z 7ed32bc8e469: Waiting 2024-08-20T22:13:03.8350374Z 33444a5a7e01: Waiting 2024-08-20T22:13:03.8350696Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:03.8351049Z 4a0412373305: Waiting 2024-08-20T22:13:03.8351370Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:03.8351721Z 302c8c177944: Waiting 2024-08-20T22:13:03.8352022Z 7754ee068b33: Waiting 2024-08-20T22:13:03.8352333Z 02970347dff6: Waiting 2024-08-20T22:13:03.8352654Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:03.8353002Z 0479fbac3361: Waiting 2024-08-20T22:13:03.8353314Z afc429e87b59: Waiting 2024-08-20T22:13:03.8353767Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:03.8354161Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:03.8354553Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:03.8354914Z 560cd6122a38: Waiting 2024-08-20T22:13:03.8355230Z 33b639226eee: Waiting 2024-08-20T22:13:03.8355552Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:03.8355907Z 829c0e19f4b7: Waiting 2024-08-20T22:13:03.8356225Z cbb79a705353: Waiting 2024-08-20T22:13:03.8356545Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:03.8356868Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:03.8357240Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:03.8357598Z f1d6874dee9c: Waiting 2024-08-20T22:13:03.8357908Z 916c5e858016: Waiting 2024-08-20T22:13:03.8358213Z c42b3602a1fb: Waiting 2024-08-20T22:13:03.8358536Z ee1273153675: Pulling fs layer 2024-08-20T22:13:03.8358909Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:03.8359275Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:03.8359651Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:03.8360017Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:03.8360343Z 0f085d4fa279: Waiting 2024-08-20T22:13:03.8360665Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:03.8361022Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:03.8361337Z 7cbfce999d76: Waiting 2024-08-20T22:13:03.8361670Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:03.8362016Z ee1273153675: Waiting 2024-08-20T22:13:03.8362329Z e22979f9e0ee: Waiting 2024-08-20T22:13:03.8362662Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:03.8363033Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:03.8363405Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:03.8363762Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:03.8364138Z 768c31dca016: Waiting 2024-08-20T22:13:03.8364452Z a7a5392e2bf6: Waiting 2024-08-20T22:13:03.8364768Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:03.8365083Z 5d115b29b5d5: Waiting 2024-08-20T22:13:03.8365389Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:03.8365702Z 37513e9b43d7: Waiting 2024-08-20T22:13:03.8366014Z 11fad972ef6c: Waiting 2024-08-20T22:13:03.8366326Z 75b2abb16c68: Waiting 2024-08-20T22:13:03.8366636Z f957c1624481: Waiting 2024-08-20T22:13:03.8367167Z 9752d6fbbcab: Waiting 2024-08-20T22:13:03.8367537Z 83808b2eff4d: Waiting 2024-08-20T22:13:03.8367850Z e56eeff2db93: Waiting 2024-08-20T22:13:03.8368282Z 78ee77eddef2: Waiting 2024-08-20T22:13:03.8368597Z eb09cbf3e416: Waiting 2024-08-20T22:13:03.8368955Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:03.8369265Z b10de02255c0: Waiting 2024-08-20T22:13:03.8369573Z d79fd7d52698: Waiting 2024-08-20T22:13:03.8369889Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:03.9571264Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:03.9571702Z 224fe954d725: Download complete 2024-08-20T22:13:04.0302612Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:04.0303039Z d527cbbb87e3: Download complete 2024-08-20T22:13:04.1023717Z b57676e46aee: Download complete 2024-08-20T22:13:04.1697006Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:04.1697430Z 7a2c55901189: Download complete 2024-08-20T22:13:04.2306433Z a41a8d1c11c8: Download complete 2024-08-20T22:13:04.2929319Z 0c1227890755: Verifying Checksum 2024-08-20T22:13:04.2929809Z 0c1227890755: Download complete 2024-08-20T22:13:04.3711309Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:04.3711838Z d8d1234baab3: Download complete 2024-08-20T22:13:04.4633803Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:04.4634398Z 75722010b82e: Download complete 2024-08-20T22:13:04.5510357Z ec1e7978c1fe: Download complete 2024-08-20T22:13:04.6169882Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:04.6170349Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:05.3429466Z 7a2c55901189: Pull complete 2024-08-20T22:13:05.6546717Z 224fe954d725: Pull complete 2024-08-20T22:13:06.5799963Z 75722010b82e: Pull complete 2024-08-20T22:13:06.5993290Z d527cbbb87e3: Pull complete 2024-08-20T22:13:06.6174954Z b57676e46aee: Pull complete 2024-08-20T22:13:07.1329798Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:07.1330277Z ca426ffc9d2b: Download complete 2024-08-20T22:13:07.2011787Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:07.2012487Z 4373aa743caf: Download complete 2024-08-20T22:13:07.2720726Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:07.2721135Z 75b2abb16c68: Download complete 2024-08-20T22:13:07.4541647Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:08.4244018Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:08.4244618Z 0bf61114d37b: Download complete 2024-08-20T22:13:08.4837457Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:08.4837900Z 2df849f72ba4: Download complete 2024-08-20T22:13:08.5653369Z 34c47d63c5de: Download complete 2024-08-20T22:13:08.6347941Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:13:08.6348520Z 1fef4bd19792: Download complete 2024-08-20T22:13:17.4393161Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:17.4393640Z a8c1e85b5e14: Download complete 2024-08-20T22:13:17.5086468Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:17.5086903Z b119c4fc6c58: Download complete 2024-08-20T22:13:17.5920623Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:17.5921070Z 33b639226eee: Download complete 2024-08-20T22:13:17.6570340Z d3547bd43146: Verifying Checksum 2024-08-20T22:13:17.6570754Z d3547bd43146: Download complete 2024-08-20T22:13:17.7425625Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:17.7426165Z 5645520256fa: Download complete 2024-08-20T22:13:17.8171204Z 829ed7d91e38: Download complete 2024-08-20T22:13:19.0824211Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:19.0824859Z 385d0421d67c: Download complete 2024-08-20T22:13:19.1630526Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:19.1631135Z 7cbfce999d76: Download complete 2024-08-20T22:13:19.2239315Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:19.2239919Z 6658e558ebe4: Download complete 2024-08-20T22:13:19.2932403Z 33444a5a7e01: Download complete 2024-08-20T22:13:19.3690944Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:19.3691384Z 7c47122f4e09: Download complete 2024-08-20T22:13:19.4431099Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:19.4431523Z 1811ff4d3556: Download complete 2024-08-20T22:13:25.4380415Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:25.5102304Z 0479fbac3361: Download complete 2024-08-20T22:13:25.5976048Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:25.5976494Z f1d6874dee9c: Download complete 2024-08-20T22:13:26.0264870Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:26.0265461Z afc429e87b59: Download complete 2024-08-20T22:13:26.0972057Z 019620471c25: Download complete 2024-08-20T22:13:26.1748903Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:26.4288497Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:26.4289158Z 829c0e19f4b7: Download complete 2024-08-20T22:13:26.4978054Z 560cd6122a38: Verifying Checksum 2024-08-20T22:13:26.4978594Z 560cd6122a38: Download complete 2024-08-20T22:13:26.5716267Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:26.5716834Z 5a4eb19d7219: Download complete 2024-08-20T22:13:26.6446925Z da6a838a66c7: Download complete 2024-08-20T22:13:29.7121146Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:29.7121619Z 7ed32bc8e469: Download complete 2024-08-20T22:13:29.7841426Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:29.8579902Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:13:29.8580386Z 0f085d4fa279: Download complete 2024-08-20T22:13:30.3363753Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:30.3364315Z 4a0412373305: Download complete 2024-08-20T22:13:30.4097624Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:30.4098376Z 302c8c177944: Download complete 2024-08-20T22:13:30.4784147Z 02970347dff6: Verifying Checksum 2024-08-20T22:13:30.4784939Z 02970347dff6: Download complete 2024-08-20T22:13:30.5486389Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:30.5486946Z 916c5e858016: Download complete 2024-08-20T22:13:30.6314506Z c42b3602a1fb: Download complete 2024-08-20T22:13:30.6424231Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:30.8223086Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:30.9861155Z 0c1227890755: Pull complete 2024-08-20T22:13:31.1330211Z d8d1234baab3: Pull complete 2024-08-20T22:13:37.3280076Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:13:37.3281012Z 6efb96fc87ea: Download complete 2024-08-20T22:13:37.4603931Z f957c1624481: Download complete 2024-08-20T22:13:37.5351373Z 768c31dca016: Verifying Checksum 2024-08-20T22:13:37.5351792Z 768c31dca016: Download complete 2024-08-20T22:13:37.5949955Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:13:37.5950522Z 8c32d0ecebbe: Download complete 2024-08-20T22:13:37.6035425Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:13:37.6035891Z 4f4fb700ef54: Download complete 2024-08-20T22:13:37.6745469Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:13:37.7520652Z 37513e9b43d7: Verifying Checksum 2024-08-20T22:13:37.7521119Z 37513e9b43d7: Download complete 2024-08-20T22:13:37.8841921Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:13:37.8842393Z a7a5392e2bf6: Download complete 2024-08-20T22:13:37.9428220Z b10de02255c0: Verifying Checksum 2024-08-20T22:13:37.9428716Z b10de02255c0: Download complete 2024-08-20T22:13:38.0150937Z e56eeff2db93: Download complete 2024-08-20T22:13:38.1027703Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:13:38.1028319Z 70ac3e38d0fd: Download complete 2024-08-20T22:13:38.1752196Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:13:38.1752817Z eb09cbf3e416: Download complete 2024-08-20T22:13:38.3042451Z ee1273153675: Verifying Checksum 2024-08-20T22:13:38.3042982Z ee1273153675: Download complete 2024-08-20T22:13:38.3683156Z e22979f9e0ee: Download complete 2024-08-20T22:13:38.9590050Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:13:38.9590537Z 6c1f0aabce7c: Download complete 2024-08-20T22:13:39.0341905Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:04.6562891Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:04.6563425Z 5d115b29b5d5: Download complete 2024-08-20T22:14:04.7149599Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:14:04.7150057Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:04.7951282Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:04.7951734Z d79fd7d52698: Download complete 2024-08-20T22:14:05.8557863Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:05.8558467Z 9752d6fbbcab: Download complete 2024-08-20T22:14:05.9217947Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:05.9218506Z 83808b2eff4d: Download complete 2024-08-20T22:14:45.3988170Z 11fad972ef6c: Download complete 2024-08-20T22:14:45.8779347Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:00.3813340Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:00.5915245Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:00.8018329Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:09.4355975Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:09.6529425Z 4373aa743caf: Pull complete 2024-08-20T22:15:09.8754371Z 75b2abb16c68: Pull complete 2024-08-20T22:15:10.0906773Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:12.6761916Z 0bf61114d37b: Pull complete 2024-08-20T22:15:12.8971616Z 2df849f72ba4: Pull complete 2024-08-20T22:15:13.0087008Z 34c47d63c5de: Pull complete 2024-08-20T22:15:13.0383965Z 1fef4bd19792: Pull complete 2024-08-20T22:16:11.1025928Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:11.3123092Z b119c4fc6c58: Pull complete 2024-08-20T22:16:11.5419811Z 33b639226eee: Pull complete 2024-08-20T22:16:11.6574557Z d3547bd43146: Pull complete 2024-08-20T22:16:11.8375711Z 5645520256fa: Pull complete 2024-08-20T22:16:11.9783336Z 829ed7d91e38: Pull complete 2024-08-20T22:16:15.4835806Z 385d0421d67c: Pull complete 2024-08-20T22:16:15.6073019Z 7cbfce999d76: Pull complete 2024-08-20T22:16:15.8067000Z 6658e558ebe4: Pull complete 2024-08-20T22:16:15.9685808Z 33444a5a7e01: Pull complete 2024-08-20T22:16:16.0921116Z 7c47122f4e09: Pull complete 2024-08-20T22:16:16.2785742Z 1811ff4d3556: Pull complete 2024-08-20T22:16:26.6109097Z 7754ee068b33: Pull complete 2024-08-20T22:16:26.6920246Z 0479fbac3361: Pull complete 2024-08-20T22:16:26.8931764Z f1d6874dee9c: Pull complete 2024-08-20T22:16:27.8561976Z afc429e87b59: Pull complete 2024-08-20T22:16:28.0668501Z 019620471c25: Pull complete 2024-08-20T22:16:28.2666019Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:28.7615687Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:28.9532740Z 560cd6122a38: Pull complete 2024-08-20T22:16:29.3252451Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:29.5115818Z da6a838a66c7: Pull complete 2024-08-20T22:17:08.3225905Z cbb79a705353: Pull complete 2024-08-20T22:17:08.5407023Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:08.7594067Z 0f085d4fa279: Pull complete 2024-08-20T22:17:11.1745234Z 4a0412373305: Pull complete 2024-08-20T22:17:11.1977640Z 302c8c177944: Pull complete 2024-08-20T22:17:11.2174676Z 02970347dff6: Pull complete 2024-08-20T22:17:11.2657589Z 916c5e858016: Pull complete 2024-08-20T22:17:11.2881638Z c42b3602a1fb: Pull complete 2024-08-20T22:18:33.7143923Z 11fad972ef6c: Pull complete 2024-08-20T22:18:33.9241518Z f957c1624481: Pull complete 2024-08-20T22:18:34.1372079Z 768c31dca016: Pull complete 2024-08-20T22:18:34.5509746Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:34.7573499Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:34.9277293Z 78ee77eddef2: Pull complete 2024-08-20T22:18:35.1465352Z 37513e9b43d7: Pull complete 2024-08-20T22:18:35.5068972Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:35.7316385Z b10de02255c0: Pull complete 2024-08-20T22:18:35.9350276Z e56eeff2db93: Pull complete 2024-08-20T22:18:36.1163777Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:36.3401891Z eb09cbf3e416: Pull complete 2024-08-20T22:18:37.7686422Z ee1273153675: Pull complete 2024-08-20T22:18:37.9814122Z e22979f9e0ee: Pull complete 2024-08-20T22:18:40.1915260Z 6c1f0aabce7c: Pull complete 2024-08-20T22:18:40.3849100Z 7ebe49c3a23e: Pull complete 2024-08-20T22:18:53.9381756Z 5d115b29b5d5: Pull complete 2024-08-20T22:18:54.1338698Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:18:54.3382151Z d79fd7d52698: Pull complete 2024-08-20T22:18:55.2105192Z 9752d6fbbcab: Pull complete 2024-08-20T22:18:55.4354616Z 83808b2eff4d: Pull complete 2024-08-20T22:18:56.3607066Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:18:56.4069693Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:56.4305983Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:18:56.4372255Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:18:56.4373170Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:18:56.4384892Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:18:56.4385385Z env: 2024-08-20T22:18:56.4385670Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:56.4386033Z ##[endgroup] 2024-08-20T22:18:56.4574994Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:18:56.4575541Z with: 2024-08-20T22:18:56.4575837Z driver-version: 550.54.15 2024-08-20T22:18:56.4576186Z env: 2024-08-20T22:18:56.4576476Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:56.4576830Z ##[endgroup] 2024-08-20T22:18:56.4705676Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:18:56.4706240Z with: 2024-08-20T22:18:56.4706528Z timeout_minutes: 10 2024-08-20T22:18:56.4706857Z max_attempts: 3 2024-08-20T22:18:56.4738605Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:18:56.4771182Z retry_wait_seconds: 10 2024-08-20T22:18:56.4771550Z polling_interval_seconds: 1 2024-08-20T22:18:56.4771924Z warning_on_retry: true 2024-08-20T22:18:56.4772290Z continue_on_error: false 2024-08-20T22:18:56.4772638Z env: 2024-08-20T22:18:56.4772927Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:18:56.4773287Z DRIVER_VERSION: 550.54.15 2024-08-20T22:18:56.4773651Z ##[endgroup] 2024-08-20T22:18:56.5578356Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:18:56.5579469Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:18:56.5580163Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:18:57.0126873Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:18:57.0127503Z No packages marked for removal. 2024-08-20T22:18:57.0188740Z Dependencies resolved. 2024-08-20T22:18:57.0198687Z Nothing to do. 2024-08-20T22:18:57.0199102Z Complete! 2024-08-20T22:18:57.1038115Z + install_nvidia_driver_common 2024-08-20T22:18:57.1040858Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:18:57.1041405Z + lspci 2024-08-20T22:18:57.1041749Z Before installing NVIDIA driver 2024-08-20T22:18:57.1993858Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:18:57.1995421Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:18:57.1997564Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:18:57.1999622Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:18:57.2000756Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:18:57.2001547Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:18:57.2002289Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:18:57.2003094Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:18:57.2003687Z + lsmod 2024-08-20T22:18:57.2037231Z Module Size Used by 2024-08-20T22:18:57.2037791Z xt_conntrack 16384 1 2024-08-20T22:18:57.2038302Z nft_chain_nat 16384 3 2024-08-20T22:18:57.2038694Z xt_MASQUERADE 20480 1 2024-08-20T22:18:57.2039146Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:18:57.2039643Z nf_conntrack_netlink 57344 0 2024-08-20T22:18:57.2040233Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:18:57.2040922Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:18:57.2041386Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:18:57.2041864Z xfrm_user 57344 1 2024-08-20T22:18:57.2042364Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:18:57.2042951Z xt_addrtype 16384 2 2024-08-20T22:18:57.2043363Z nft_compat 20480 4 2024-08-20T22:18:57.2043973Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:18:57.2044782Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:18:57.2045342Z br_netfilter 36864 0 2024-08-20T22:18:57.2047041Z bridge 307200 1 br_netfilter 2024-08-20T22:18:57.2047656Z stp 16384 1 bridge 2024-08-20T22:18:57.2048277Z llc 16384 2 bridge,stp 2024-08-20T22:18:57.2049163Z overlay 167936 0 2024-08-20T22:18:57.2049667Z tls 114688 0 2024-08-20T22:18:57.2050235Z nls_ascii 16384 1 2024-08-20T22:18:57.2050718Z nls_cp437 20480 1 2024-08-20T22:18:57.2051213Z vfat 24576 1 2024-08-20T22:18:57.2051777Z fat 86016 1 vfat 2024-08-20T22:18:57.2052368Z sunrpc 692224 1 2024-08-20T22:18:57.2052873Z ena 167936 0 2024-08-20T22:18:57.2053376Z ghash_clmulni_intel 16384 0 2024-08-20T22:18:57.2053917Z ptp 36864 1 ena 2024-08-20T22:18:57.2054497Z pps_core 24576 1 ptp 2024-08-20T22:18:57.2055043Z aesni_intel 393216 0 2024-08-20T22:18:57.2055530Z i8042 45056 0 2024-08-20T22:18:57.2056021Z serio 28672 3 i8042 2024-08-20T22:18:57.2056598Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:18:57.2057267Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:18:57.2057914Z button 24576 0 2024-08-20T22:18:57.2058399Z sch_fq_codel 20480 17 2024-08-20T22:18:57.2058831Z dm_mod 188416 0 2024-08-20T22:18:57.2059240Z fuse 163840 1 2024-08-20T22:18:57.2059610Z configfs 57344 1 2024-08-20T22:18:57.2060010Z dax 45056 1 dm_mod 2024-08-20T22:18:57.2060439Z loop 36864 0 2024-08-20T22:18:57.2060832Z dmi_sysfs 20480 0 2024-08-20T22:18:57.2061334Z crc32_pclmul 16384 0 2024-08-20T22:18:57.2061806Z crc32c_intel 24576 0 2024-08-20T22:18:57.2062276Z efivarfs 24576 1 2024-08-20T22:18:57.2062746Z + modinfo nvidia 2024-08-20T22:18:57.2063498Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:18:57.2064201Z alias: char-major-195-* 2024-08-20T22:18:57.2064585Z version: 550.54.15 2024-08-20T22:18:57.2064937Z supported: external 2024-08-20T22:18:57.2065287Z license: NVIDIA 2024-08-20T22:18:57.2065662Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:18:57.2066148Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:18:57.2066614Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:18:57.2067078Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:18:57.2067580Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:18:57.2068075Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:18:57.2068572Z depends: i2c-core,drm 2024-08-20T22:18:57.2068926Z retpoline: Y 2024-08-20T22:18:57.2069238Z name: nvidia 2024-08-20T22:18:57.2070052Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:18:57.2070763Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:18:57.2071402Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:18:57.2072001Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:18:57.2072453Z parm: NVreg_RmLogonRC:int 2024-08-20T22:18:57.2072891Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:18:57.2073347Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:18:57.2073786Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:18:57.2074226Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:18:57.2074751Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:18:57.2075314Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:18:57.2075796Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:18:57.2076254Z parm: NVreg_EnableMSI:int 2024-08-20T22:18:57.2076678Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:18:57.2077133Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:18:57.2077663Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:18:57.2078246Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:18:57.2078809Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:18:57.2080866Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:57.2081463Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:18:57.2082072Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:57.2082665Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:18:57.2083150Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:18:57.2083685Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:18:57.2084229Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:18:57.2084726Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:18:57.2085196Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:18:57.2085679Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:18:57.2086151Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:18:57.2086685Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:18:57.2087193Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:18:57.2087726Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:18:57.2088205Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:18:57.2088685Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:18:57.2089187Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:18:57.2089673Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:18:57.2090216Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:18:57.2090698Z parm: NVreg_RmMsg:charp 2024-08-20T22:18:57.2091110Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:18:57.2091578Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:18:57.2092056Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:18:57.2092507Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:18:57.2092987Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:18:57.2093465Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:18:57.2093917Z parm: rm_firmware_active:charp 2024-08-20T22:18:57.2094334Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:18:57.2094728Z ++ command -v nvidia-smi 2024-08-20T22:18:57.2095135Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:18:57.2095498Z + set +e 2024-08-20T22:18:57.2096009Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:18:59.4347363Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:18:59.4347993Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:18:59.4348662Z + '[' 0 -ne 0 ']' 2024-08-20T22:18:59.4349096Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:18:59.4349497Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:18:59.4350613Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:18:59.4351343Z + set -e 2024-08-20T22:18:59.4351658Z + '[' 1 -eq 0 ']' 2024-08-20T22:18:59.4352237Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:18:59.4352961Z + post_install_nvidia_driver_common 2024-08-20T22:18:59.4353512Z + sudo modprobe nvidia 2024-08-20T22:18:59.5546011Z + echo 'After installing NVIDIA driver' 2024-08-20T22:18:59.5546580Z + lspci 2024-08-20T22:18:59.5546951Z After installing NVIDIA driver 2024-08-20T22:18:59.5652881Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:18:59.5653875Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:18:59.5655033Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:18:59.5655869Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:18:59.5656708Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:18:59.5657500Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:18:59.5658239Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:18:59.5659049Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:18:59.5659890Z + lsmod 2024-08-20T22:18:59.5686773Z Module Size Used by 2024-08-20T22:18:59.5687283Z nvidia_uvm 4706304 0 2024-08-20T22:18:59.5687845Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:18:59.5688386Z drm 602112 1 nvidia 2024-08-20T22:18:59.5688968Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:18:59.5689561Z backlight 24576 1 drm 2024-08-20T22:18:59.5690102Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:18:59.5690654Z xt_conntrack 16384 1 2024-08-20T22:18:59.5691082Z nft_chain_nat 16384 3 2024-08-20T22:18:59.5691465Z xt_MASQUERADE 20480 1 2024-08-20T22:18:59.5691923Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:18:59.5692405Z nf_conntrack_netlink 57344 0 2024-08-20T22:18:59.5692984Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:18:59.5693622Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:18:59.5694094Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:18:59.5694530Z xfrm_user 57344 1 2024-08-20T22:18:59.5694932Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:18:59.5695360Z xt_addrtype 16384 2 2024-08-20T22:18:59.5695739Z nft_compat 20480 4 2024-08-20T22:18:59.5696199Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:18:59.5696814Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:18:59.5697362Z br_netfilter 36864 0 2024-08-20T22:18:59.5697773Z bridge 307200 1 br_netfilter 2024-08-20T22:18:59.5698222Z stp 16384 1 bridge 2024-08-20T22:18:59.5698642Z llc 16384 2 bridge,stp 2024-08-20T22:18:59.5699062Z overlay 167936 0 2024-08-20T22:18:59.5699425Z tls 114688 0 2024-08-20T22:18:59.5699790Z nls_ascii 16384 1 2024-08-20T22:18:59.5700158Z nls_cp437 20480 1 2024-08-20T22:18:59.5700538Z vfat 24576 1 2024-08-20T22:18:59.5700907Z fat 86016 1 vfat 2024-08-20T22:18:59.5701302Z sunrpc 692224 1 2024-08-20T22:18:59.5701665Z ena 167936 0 2024-08-20T22:18:59.5702035Z ghash_clmulni_intel 16384 0 2024-08-20T22:18:59.5702425Z ptp 36864 1 ena 2024-08-20T22:18:59.5702823Z pps_core 24576 1 ptp 2024-08-20T22:18:59.5703215Z aesni_intel 393216 0 2024-08-20T22:18:59.5703575Z i8042 45056 0 2024-08-20T22:18:59.5703946Z serio 28672 3 i8042 2024-08-20T22:18:59.5704564Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:18:59.5705090Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:18:59.5705582Z button 24576 0 2024-08-20T22:18:59.5705957Z sch_fq_codel 20480 17 2024-08-20T22:18:59.5706337Z dm_mod 188416 0 2024-08-20T22:18:59.5706702Z fuse 163840 1 2024-08-20T22:18:59.5707069Z configfs 57344 1 2024-08-20T22:18:59.5707445Z dax 45056 1 dm_mod 2024-08-20T22:18:59.5707916Z loop 36864 0 2024-08-20T22:18:59.5708398Z dmi_sysfs 20480 0 2024-08-20T22:18:59.5708877Z crc32_pclmul 16384 0 2024-08-20T22:18:59.5709357Z crc32c_intel 24576 0 2024-08-20T22:18:59.5709833Z efivarfs 24576 1 2024-08-20T22:18:59.5710291Z + modinfo nvidia 2024-08-20T22:18:59.5711145Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:18:59.5711864Z alias: char-major-195-* 2024-08-20T22:18:59.5712253Z version: 550.54.15 2024-08-20T22:18:59.5712601Z supported: external 2024-08-20T22:18:59.5712954Z license: NVIDIA 2024-08-20T22:18:59.5713339Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:18:59.5713827Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:18:59.5714417Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:18:59.5714888Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:18:59.5715392Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:18:59.5715890Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:18:59.5716383Z depends: i2c-core,drm 2024-08-20T22:18:59.5716742Z retpoline: Y 2024-08-20T22:18:59.5717050Z name: nvidia 2024-08-20T22:18:59.5717645Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:18:59.5718338Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:18:59.5718979Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:18:59.5719565Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:18:59.5720000Z parm: NVreg_RmLogonRC:int 2024-08-20T22:18:59.5720430Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:18:59.5720894Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:18:59.5721338Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:18:59.5721765Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:18:59.5722292Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:18:59.5722857Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:18:59.5723340Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:18:59.5723772Z parm: NVreg_EnableMSI:int 2024-08-20T22:18:59.5724193Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:18:59.5724654Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:18:59.5725185Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:18:59.5725767Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:18:59.5726333Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:18:59.5727054Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:59.5727655Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:18:59.5728270Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:18:59.5728867Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:18:59.5729363Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:18:59.5729900Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:18:59.5730443Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:18:59.5730939Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:18:59.5731396Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:18:59.5731869Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:18:59.5732342Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:18:59.5732888Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:18:59.5733394Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:18:59.5733912Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:18:59.5734390Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:18:59.5734879Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:18:59.5735380Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:18:59.5735857Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:18:59.5736355Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:18:59.5736829Z parm: NVreg_RmMsg:charp 2024-08-20T22:18:59.5737238Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:18:59.5737695Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:18:59.5738161Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:18:59.5738614Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:18:59.5739093Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:18:59.5739556Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:18:59.5740014Z parm: rm_firmware_active:charp 2024-08-20T22:18:59.5740419Z + set +e 2024-08-20T22:18:59.5740726Z + nvidia-smi 2024-08-20T22:19:01.1342310Z Tue Aug 20 22:19:01 2024 2024-08-20T22:19:01.1343292Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:01.1344535Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:01.1345335Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:01.1346327Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:01.1347223Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:01.1347916Z | | | MIG M. | 2024-08-20T22:19:01.1348475Z |=========================================+========================+======================| 2024-08-20T22:19:01.1440538Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:01.1441533Z | 0% 31C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:01.1442912Z | | | N/A | 2024-08-20T22:19:01.1444694Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:01.1446301Z 2024-08-20T22:19:01.1448260Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:01.1449683Z | Processes: | 2024-08-20T22:19:01.1451089Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:01.1451869Z | ID ID Usage | 2024-08-20T22:19:01.1452435Z |=========================================================================================| 2024-08-20T22:19:01.1453080Z | No running processes found | 2024-08-20T22:19:01.1453853Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:01.7559447Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:03.3196864Z NVIDIA A10G 2024-08-20T22:19:03.7691010Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:03.7692814Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:03.7693243Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:03.7693682Z + set -e 2024-08-20T22:19:03.7693992Z INFO: Ignoring allowed status 0 2024-08-20T22:19:03.7699562Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:03.7703612Z + sudo yum install -y yum-utils 2024-08-20T22:19:04.1750355Z Last metadata expiration check: 0:09:36 ago on Tue Aug 20 22:09:28 2024. 2024-08-20T22:19:04.2056293Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:04.2360447Z Dependencies resolved. 2024-08-20T22:19:04.2484536Z Nothing to do. 2024-08-20T22:19:04.2484892Z Complete! 2024-08-20T22:19:04.3446070Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:04.3447417Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:04.3448718Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:04.5927566Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:04.6631736Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:05.2158070Z nvidia-container-toolkit 11 kB/s | 833 B 00:00 2024-08-20T22:19:05.2496296Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:05.2790825Z Dependencies resolved. 2024-08-20T22:19:05.2913944Z Nothing to do. 2024-08-20T22:19:05.2914641Z Complete! 2024-08-20T22:19:05.3828556Z + sudo systemctl restart docker 2024-08-20T22:19:59.5305070Z Tue Aug 20 22:19:59 2024 2024-08-20T22:19:59.5307695Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:59.5308860Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:59.5309954Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:59.5311040Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:59.5312230Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:59.5313169Z | | | MIG M. | 2024-08-20T22:19:59.5313902Z |=========================================+========================+======================| 2024-08-20T22:19:59.5432149Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:59.5433140Z | 0% 32C P0 59W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:19:59.5433971Z | | | N/A | 2024-08-20T22:19:59.5434899Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:59.5435707Z 2024-08-20T22:19:59.5436638Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:59.5437503Z | Processes: | 2024-08-20T22:19:59.5438413Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:59.5439296Z | ID ID Usage | 2024-08-20T22:19:59.5440037Z |=========================================================================================| 2024-08-20T22:19:59.5440887Z | No running processes found | 2024-08-20T22:19:59.5441903Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:00.5954182Z Command completed after 1 attempt(s). 2024-08-20T22:20:00.6032513Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:00.6033273Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:00.6033962Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:00.6034608Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:00.6049038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:00.6049547Z env: 2024-08-20T22:20:00.6049842Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:00.6050305Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:00.6050801Z ##[endgroup] 2024-08-20T22:20:00.8883283Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:01.3011228Z Collecting psutil==5.9.1 2024-08-20T22:20:01.3389803Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:20:01.3940618Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:20:01.3978528Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:20:01.4802889Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:20:01.6360541Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:20:01.8356688Z Prepare all required actions 2024-08-20T22:20:01.8357234Z Getting action download info 2024-08-20T22:20:02.2973318Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:02.4805239Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:02.7016582Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:02.7017076Z with: 2024-08-20T22:20:02.7017422Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:02.7017916Z s3-bucket: gha-artifacts 2024-08-20T22:20:02.7018297Z env: 2024-08-20T22:20:02.7018569Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:02.7019024Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:02.7019502Z ##[endgroup] 2024-08-20T22:20:02.7053095Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:02.7053523Z with: 2024-08-20T22:20:02.7053888Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:02.7054343Z s3-bucket: gha-artifacts 2024-08-20T22:20:02.7054697Z region: us-east-1 2024-08-20T22:20:02.7054997Z env: 2024-08-20T22:20:02.7055273Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:02.7055716Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:02.7056196Z ##[endgroup] 2024-08-20T22:20:03.1812806Z (node:52329) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:03.1813494Z 2024-08-20T22:20:03.1813746Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:03.1814450Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:03.1815315Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:03.2513105Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:03.2514360Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:24.8795887Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:24.8802344Z Artifact download has finished successfully 2024-08-20T22:20:24.9155326Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:24.9155765Z unzip -o artifacts.zip 2024-08-20T22:20:24.9165728Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:24.9166239Z env: 2024-08-20T22:20:24.9166522Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:24.9167106Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:24.9167603Z ##[endgroup] 2024-08-20T22:20:24.9247768Z Archive: artifacts.zip 2024-08-20T22:20:24.9248999Z creating: dist/ 2024-08-20T22:20:27.0573168Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:27.0573870Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:27.0574470Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:27.0575211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:27.0576043Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.0582522Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.0583473Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.0584421Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.0585424Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.0586394Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.0587704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.0589422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.0590433Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.0591415Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.0592988Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.0594784Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.0596610Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.0597747Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.0599393Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.0600947Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.0601981Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.0602976Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.0642931Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.0682831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.0684273Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.0729767Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.0731187Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.0732616Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.0734087Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.0735503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.0736881Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.0738267Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.0739648Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.0740997Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.0742272Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.0743502Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.0744732Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.0746220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.0747410Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.0748621Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.0805713Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.0865398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.0867733Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.0870011Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.0871727Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.0873532Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.0875167Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:27.0876213Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.0877359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.0878459Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:27.0879489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:27.0880567Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.0881660Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:27.0882742Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.0883806Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:27.0885039Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:27.0897612Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:27.1040599Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:27.1043219Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:27.1044345Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:27.1045787Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:27.1047270Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:27.1048750Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:27.1050136Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:27.1051308Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:27.1052454Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:27.1053589Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:27.1054708Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:27.1070951Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:27.1154406Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:27.1155979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.1157363Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.1158776Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.1160008Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.1160949Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.1161845Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:27.1163106Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:27.1164121Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:27.1165212Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:27.1285096Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:27.1349430Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:27.1351262Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:27.1352694Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:27.1354322Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.1357299Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.1358522Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.1359663Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.1360645Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.1361844Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.1363058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.1364494Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.1365508Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.1366581Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.1368016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.1369477Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.1371224Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.1372530Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.1373999Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.1375520Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.1376637Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.1377649Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.1416694Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.1456735Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.1458288Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.1503616Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.1505572Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.1508016Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.1509962Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.1511633Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.1513619Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.1515372Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.1517233Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.1518860Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.1520206Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.1521434Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.1522652Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.1523889Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.1525070Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.1526293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.1580013Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.1641641Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.1644678Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.1646581Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.1647787Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.1648963Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.1650200Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:27.1651573Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:27.1652799Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:27.1653938Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:27.1655042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:27.1656141Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:27.1657246Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:27.1658344Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:27.1659428Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:27.1660524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:27.1672886Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:27.1737283Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:27.1738820Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.1740133Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.1741438Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.1742587Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.1743556Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.1744451Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:27.1745890Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:27.1747062Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:27.1748134Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:27.1798212Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:27.1799205Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:27.1800034Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:27.1800928Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:27.1807290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:27.1808397Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:27.1809442Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:27.1810495Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:27.1811527Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:27.1812926Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:27.1814465Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:27.1815596Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:27.1816712Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:27.1818172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:27.1819691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:27.1821214Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:27.1822548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:27.1824066Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:27.1825684Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:27.1826846Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:27.1827929Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:27.1867077Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:27.1906681Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:27.1908468Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:27.1953949Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:27.1955888Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:27.1958040Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:27.1960072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:27.1961938Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:27.1963850Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:27.1965755Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:27.1967589Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:27.1969220Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:27.1970547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:27.1971831Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:27.1973097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:27.1974375Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:27.1975670Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:27.1976929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:27.2029794Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:27.2089307Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:27.2090934Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:27.2092316Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:27.2093512Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:27.2094799Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:27.2096128Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:27.2097581Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.2099192Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.2100702Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:27.2101870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:27.2103054Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.2104679Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:27.2105973Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.2107146Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:27.2108447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:27.2109691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:27.2231912Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:27.2234973Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:27.2236629Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:27.2238348Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:27.2239612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:27.2240815Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:27.2242072Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:27.2243301Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:27.2244550Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:27.2246102Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:27.2247423Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:27.2260238Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:27.2316106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:27.2317746Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:27.2319210Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:27.2320580Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:27.2321562Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:27.2322515Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:27.2323767Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:27.2324992Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:27.2325975Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:27.2326840Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:27.2429055Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:27.2472182Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:27.2472984Z creating: build/lib/ 2024-08-20T22:20:27.2561288Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:27.3017467Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:27.3027211Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:27.3035455Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:27.3043669Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:27.3044320Z inflating: build/lib/libclog.a 2024-08-20T22:20:27.3063541Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:27.3066312Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:27.3131047Z inflating: build/lib/libgtest.a 2024-08-20T22:20:27.3206076Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:27.3270750Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:27.3277660Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:27.3305412Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:27.3433160Z inflating: build/lib/libgloo.a 2024-08-20T22:20:27.3453596Z inflating: build/lib/libfmt.a 2024-08-20T22:20:27.3549605Z inflating: build/lib/libc10.so 2024-08-20T22:20:27.3551162Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:27.3553369Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:27.3572998Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:27.4076175Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:27.4093986Z inflating: build/lib/libgmock.a 2024-08-20T22:20:27.4094780Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:27.4095628Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:27.4484866Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:28.4660000Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:28.5234760Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:28.5291836Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:28.5292841Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:28.6550987Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:28.6808137Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:28.7314510Z inflating: build/lib/libkineto.a 2024-08-20T22:20:28.7498133Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:28.7540507Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:28.8233027Z inflating: build/lib/libonnx.a 2024-08-20T22:20:31.2720604Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:31.2725350Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:31.2729751Z inflating: build/lib/libshm.so 2024-08-20T22:20:33.3403502Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:33.3404797Z inflating: build/lib/libtorch.so 2024-08-20T22:20:33.3407301Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:34.1604825Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:34.3601540Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:34.3674344Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:34.3694791Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:34.3721022Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:34.3746708Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:34.3781338Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:34.3781957Z creating: build/bin/ 2024-08-20T22:20:34.3833734Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:34.3884643Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:34.3936041Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:34.3995489Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:34.4049487Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:34.4098345Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:34.4150797Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:34.4204838Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:34.4261095Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:34.4317690Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:34.4388993Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:34.4442300Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:34.4491647Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:34.4541991Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:34.4593702Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:34.4650555Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:34.4704945Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:34.4756118Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:34.4811335Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:34.4863382Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:34.4915459Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:34.4964335Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:34.5016377Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:34.5072236Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:34.5124037Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:34.5180497Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:34.5236274Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:34.5290105Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:34.5340645Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:34.5391092Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:34.5556332Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:34.5607567Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:34.5661322Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:34.5718758Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:34.5781727Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:34.5857427Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:34.6009061Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:34.6063622Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:34.6115346Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:34.6167826Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:34.6218487Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:34.6277953Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:34.6326587Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:34.6383204Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:34.6831303Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:34.7278712Z inflating: build/bin/protoc 2024-08-20T22:20:34.7331895Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:34.7385143Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:34.7438251Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:34.7490877Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:34.7544666Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:34.7598998Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:34.7647680Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:34.7700819Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:34.8032074Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:34.8378046Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:34.8738615Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:34.8791634Z inflating: build/bin/BackoffTest 2024-08-20T22:20:34.8844969Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:34.8901248Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:34.8956614Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:34.8970019Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:34.9023966Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:34.9028805Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:34.9031482Z inflating: build/bin/example_allreduce 2024-08-20T22:20:34.9087200Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:34.9156891Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:34.9158467Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:34.9226261Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:34.9236215Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:34.9582576Z inflating: build/bin/test_lazy 2024-08-20T22:20:35.0766617Z inflating: build/bin/test_api 2024-08-20T22:20:35.0841333Z inflating: build/bin/Dict_test 2024-08-20T22:20:35.0893622Z inflating: build/bin/Dimname_test 2024-08-20T22:20:35.0958646Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:35.1016310Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:35.1076538Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:35.1134997Z inflating: build/bin/atest 2024-08-20T22:20:35.1198503Z inflating: build/bin/basic 2024-08-20T22:20:35.1254013Z inflating: build/bin/broadcast_test 2024-08-20T22:20:35.1305014Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:35.1363939Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:35.1417577Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:35.1510902Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:35.1561533Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:35.1612114Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:35.1671412Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:35.1726070Z inflating: build/bin/half_test 2024-08-20T22:20:35.1822938Z inflating: build/bin/ivalue_test 2024-08-20T22:20:35.1873907Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:35.1927893Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:35.1982426Z inflating: build/bin/memory_format_test 2024-08-20T22:20:35.2036446Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:35.2089193Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:35.2145426Z inflating: build/bin/native_test 2024-08-20T22:20:35.2196591Z inflating: build/bin/operator_name_test 2024-08-20T22:20:35.2248746Z inflating: build/bin/operators_test 2024-08-20T22:20:35.2300540Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:35.2368197Z inflating: build/bin/pow_test 2024-08-20T22:20:35.2425549Z inflating: build/bin/quantized_test 2024-08-20T22:20:35.2477003Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:35.2528076Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:35.2584800Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:35.2643650Z inflating: build/bin/scalar_test 2024-08-20T22:20:35.2696019Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:35.2749704Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:35.2829016Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:35.2883009Z inflating: build/bin/test_parallel 2024-08-20T22:20:35.2886098Z inflating: build/bin/thread_init_test 2024-08-20T22:20:35.2942748Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:35.3004021Z inflating: build/bin/type_test 2024-08-20T22:20:35.3056410Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:35.3057773Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:35.3128690Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:35.3180795Z inflating: build/bin/weakref_test 2024-08-20T22:20:35.3232408Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:35.3285139Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:35.3344922Z inflating: build/bin/IListRef_test 2024-08-20T22:20:35.3452882Z inflating: build/bin/List_test 2024-08-20T22:20:35.3519237Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:35.3642008Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:35.3739278Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:35.3869842Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:35.3973073Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:35.4035644Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:35.4132181Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:35.4183761Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:35.4240387Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:35.4289301Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:35.4593715Z inflating: build/bin/op_registration_test 2024-08-20T22:20:35.4657241Z inflating: build/bin/inline_container_test 2024-08-20T22:20:35.4711236Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:35.4762824Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:35.4822345Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:35.4877929Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:35.4948911Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:35.5008384Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:35.5057925Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:35.5116094Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:35.5167197Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:35.5232371Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:35.5289363Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:35.5339343Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:35.5391335Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:35.5440881Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:35.5493082Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:35.5546193Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:35.5596360Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:35.5657824Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:35.5707726Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:35.5761042Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:35.5775690Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:35.5842383Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:35.5900772Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:35.5964927Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:35.6027524Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:35.6859083Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:35.7442082Z inflating: build/bin/test_jit 2024-08-20T22:20:35.7442655Z creating: .additional_ci_files/ 2024-08-20T22:20:35.7503734Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:35.7744629Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:35.7780312Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:35.7780718Z rm artifacts.zip 2024-08-20T22:20:35.7789847Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.7790344Z env: 2024-08-20T22:20:35.7790620Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.7791074Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.7791569Z ##[endgroup] 2024-08-20T22:20:35.9042513Z ##[group]Run df -H 2024-08-20T22:20:35.9042840Z df -H 2024-08-20T22:20:35.9052082Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:35.9052591Z env: 2024-08-20T22:20:35.9052883Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:35.9053341Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:35.9053822Z ##[endgroup] 2024-08-20T22:20:35.9103410Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:35.9104074Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:35.9104682Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:35.9105267Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:35.9105817Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:35.9108830Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:20:35.9109373Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:35.9109881Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:35.9149594Z Prepare all required actions 2024-08-20T22:20:35.9150039Z Getting action download info 2024-08-20T22:20:36.0394740Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:36.0395205Z with: 2024-08-20T22:20:36.0395469Z env: 2024-08-20T22:20:36.0395757Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.0396202Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.0396693Z ##[endgroup] 2024-08-20T22:20:36.0429434Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:36.0429873Z with: 2024-08-20T22:20:36.0430139Z name: td_results 2024-08-20T22:20:36.0430465Z s3-bucket: gha-artifacts 2024-08-20T22:20:36.0430813Z region: us-east-1 2024-08-20T22:20:36.0431115Z env: 2024-08-20T22:20:36.0431385Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.0431836Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.0432317Z ##[endgroup] 2024-08-20T22:20:36.5300094Z (node:52357) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:36.5300861Z 2024-08-20T22:20:36.5301123Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:36.5301837Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:36.5302676Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:36.6041218Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:36.6049078Z Artifact download has finished successfully 2024-08-20T22:20:36.6372500Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:36.6372992Z mkdir -p .additional_ci_files 2024-08-20T22:20:36.6373552Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:36.6383411Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.6383909Z env: 2024-08-20T22:20:36.6384189Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.6384656Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.6385132Z ##[endgroup] 2024-08-20T22:20:36.6440712Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:36.6455676Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:36.6517579Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:36.6518052Z .github/scripts/parse_ref.py 2024-08-20T22:20:36.6526261Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:36.6526608Z env: 2024-08-20T22:20:36.6526893Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.6527437Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.6527911Z ##[endgroup] 2024-08-20T22:20:36.6813895Z Prepare all required actions 2024-08-20T22:20:36.6852531Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:36.6852992Z with: 2024-08-20T22:20:36.6853545Z github-token: *** 2024-08-20T22:20:36.6853845Z env: 2024-08-20T22:20:36.6854110Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.6854555Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.6855031Z ##[endgroup] 2024-08-20T22:20:36.6871820Z ##[group]Run set -eux 2024-08-20T22:20:36.6872180Z set -eux 2024-08-20T22:20:36.6872760Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:36.6881651Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:36.6882140Z env: 2024-08-20T22:20:36.6882428Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:36.6882879Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:36.6883521Z GITHUB_TOKEN: *** 2024-08-20T22:20:36.6883833Z ##[endgroup] 2024-08-20T22:20:36.6915585Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-06bc7425c52774f50 2024-08-20T22:20:38.4191246Z setting job-id=29026529448 2024-08-20T22:20:38.4192179Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.4410348Z Prepare all required actions 2024-08-20T22:20:38.4410794Z Getting action download info 2024-08-20T22:20:38.5428584Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:38.5429252Z with: 2024-08-20T22:20:38.5429720Z github-token: *** 2024-08-20T22:20:38.5438543Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:38.5448280Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:38.5449056Z env: 2024-08-20T22:20:38.5449327Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.5449782Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.5450260Z ##[endgroup] 2024-08-20T22:20:38.5493196Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:38.5493753Z with: 2024-08-20T22:20:38.5494034Z shell: bash 2024-08-20T22:20:38.5494596Z timeout_minutes: 10 2024-08-20T22:20:38.5494926Z max_attempts: 5 2024-08-20T22:20:38.5495256Z retry_wait_seconds: 30 2024-08-20T22:20:38.5496370Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.5497544Z polling_interval_seconds: 1 2024-08-20T22:20:38.5497932Z warning_on_retry: true 2024-08-20T22:20:38.5498274Z continue_on_error: false 2024-08-20T22:20:38.5498620Z env: 2024-08-20T22:20:38.5498908Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:38.5499355Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:38.5499998Z GITHUB_TOKEN: *** 2024-08-20T22:20:38.5500318Z ##[endgroup] 2024-08-20T22:20:38.6258023Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:38.8627547Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:39.0114858Z Collecting requests==2.27.1 2024-08-20T22:20:39.0480210Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:20:39.2439198Z Collecting pyyaml==6.0.1 2024-08-20T22:20:39.2482655Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:20:39.6001413Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:20:39.6046493Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:20:39.6086440Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:39.6096211Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:39.6550505Z Collecting certifi>=2017.4.17 2024-08-20T22:20:39.6592900Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:20:39.7409567Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:20:39.8612580Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:20:40.6040239Z Command completed after 1 attempt(s). 2024-08-20T22:20:40.6102487Z ##[group]Run set -x 2024-08-20T22:20:40.6102840Z set -x 2024-08-20T22:20:40.6103139Z  2024-08-20T22:20:40.6103675Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.6104330Z # in runner workspace 2024-08-20T22:20:40.6104861Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:40.6115793Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.6116291Z env: 2024-08-20T22:20:40.6116569Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.6117025Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.6117506Z ##[endgroup] 2024-08-20T22:20:40.6148277Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:40.6389092Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.6389644Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:40.6390118Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:40.6390524Z  2024-08-20T22:20:40.6391070Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:40.6391750Z # in runner workspace 2024-08-20T22:20:40.6392320Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:40.6392957Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:40.6393412Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:40.6402481Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:40.6411910Z  --selected-test-configs "" \ 2024-08-20T22:20:40.6412387Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:40.6412815Z  --tag "${TAG}" \ 2024-08-20T22:20:40.6413209Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:40.6413646Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:40.6414089Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:40.6423132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:40.6423633Z env: 2024-08-20T22:20:40.6423917Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:40.6424381Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:40.6425060Z GITHUB_TOKEN: *** 2024-08-20T22:20:40.6425773Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.6426552Z PR_NUMBER: 2024-08-20T22:20:40.6426861Z TAG: ciflow/inductor/133712 2024-08-20T22:20:40.6427241Z EVENT_NAME: push 2024-08-20T22:20:40.6427552Z SCHEDULE: 2024-08-20T22:20:40.6427836Z HEAD_BRANCH: 2024-08-20T22:20:40.6428155Z ##[endgroup] 2024-08-20T22:20:40.6457596Z Workflow: inductor 2024-08-20T22:20:40.6458605Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:40.8770226Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:41.0229318Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:41.0229779Z echo "Filtered matrix:" 2024-08-20T22:20:41.0238714Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:41.0248266Z  2024-08-20T22:20:41.0248594Z echo 2024-08-20T22:20:41.0248972Z echo "Is the current job unstable? False" 2024-08-20T22:20:41.0249433Z  2024-08-20T22:20:41.0249697Z echo 2024-08-20T22:20:41.0250058Z echo "Is keep-going label set? False" 2024-08-20T22:20:41.0250503Z  2024-08-20T22:20:41.0250776Z echo 2024-08-20T22:20:41.0251098Z echo "Renabled issues? " 2024-08-20T22:20:41.0259967Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.0260468Z env: 2024-08-20T22:20:41.0260739Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.0261185Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.0261665Z ##[endgroup] 2024-08-20T22:20:41.0290685Z Filtered matrix: 2024-08-20T22:20:41.0301113Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:41.0309804Z 2024-08-20T22:20:41.0309955Z Is the current job unstable? False 2024-08-20T22:20:41.0310237Z 2024-08-20T22:20:41.0310503Z Is keep-going label set? False 2024-08-20T22:20:41.0310764Z 2024-08-20T22:20:41.0310889Z Renabled issues? 2024-08-20T22:20:41.0354184Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.0354894Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:41.0363736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:41.0364231Z env: 2024-08-20T22:20:41.0364521Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.0364985Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.0365477Z JOB_TIMEOUT: 240 2024-08-20T22:20:41.0365786Z ##[endgroup] 2024-08-20T22:20:41.0448648Z ##[group]Run set -x 2024-08-20T22:20:41.0449214Z set -x 2024-08-20T22:20:41.0449506Z  2024-08-20T22:20:41.0449851Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:41.0450377Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:41.0450915Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:41.0451407Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:41.0451805Z else 2024-08-20T22:20:41.0452139Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.0452542Z fi 2024-08-20T22:20:41.0452813Z  2024-08-20T22:20:41.0453265Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:41.0454003Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:41.0454651Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:41.0455221Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:41.0455680Z container_name=$(docker run \ 2024-08-20T22:20:41.0456085Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:41.0456446Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:41.0456834Z  -e PR_NUMBER \ 2024-08-20T22:20:41.0457183Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:41.0457569Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:41.0457973Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:41.0458356Z  -e GITHUB_JOB \ 2024-08-20T22:20:41.0458702Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:41.0459079Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:41.0459480Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:41.0459875Z  -e JOB_ID \ 2024-08-20T22:20:41.0460202Z  -e JOB_NAME \ 2024-08-20T22:20:41.0460547Z  -e BASE_SHA \ 2024-08-20T22:20:41.0460887Z  -e BRANCH \ 2024-08-20T22:20:41.0461216Z  -e SHA1 \ 2024-08-20T22:20:41.0461553Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:41.0461954Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:41.0462324Z  -e SHARD_NUMBER \ 2024-08-20T22:20:41.0462692Z  -e TEST_CONFIG \ 2024-08-20T22:20:41.0463054Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:41.0463443Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:41.0463852Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:41.0464262Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:41.0464663Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:41.0465051Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:41.0465419Z  -e NO_TD \ 2024-08-20T22:20:41.0465755Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:41.0466132Z  -e PR_LABELS \ 2024-08-20T22:20:41.0466528Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:41.0466980Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:41.0467364Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:41.0467765Z  -e XLA_CUDA \ 2024-08-20T22:20:41.0468170Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:41.0468676Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:41.0469181Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:41.0469733Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:41.0470199Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:41.0470653Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:41.0471076Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:41.0471546Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:41.0472092Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:41.0472548Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:41.0472932Z  --ipc=host \ 2024-08-20T22:20:41.0473287Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:41.0473678Z  --tty \ 2024-08-20T22:20:41.0473989Z  --detach \ 2024-08-20T22:20:41.0474456Z  --name="${container_name}" \ 2024-08-20T22:20:41.0474944Z  --user jenkins \ 2024-08-20T22:20:41.0475425Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:41.0475972Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:41.0476394Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:41.0476736Z ) 2024-08-20T22:20:41.0477138Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:41.0478033Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:41.0478965Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:41.0479904Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:41.0488319Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:41.0488665Z env: 2024-08-20T22:20:41.0488928Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:41.0489384Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:41.0489978Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:41.0490474Z PR_NUMBER: 2024-08-20T22:20:41.0490786Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:41.0491194Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:41.0491539Z GITHUB_JOB: test 2024-08-20T22:20:41.0491852Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:41.0492200Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:41.0492546Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:41.0492871Z JOB_ID: 29026529448 2024-08-20T22:20:41.0493568Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:41.0494332Z BRANCH: 2024-08-20T22:20:41.0494678Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.0495196Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:41.0495683Z TEST_CONFIG: dynamic_inductor_torchbench 2024-08-20T22:20:41.0496109Z SHARD_NUMBER: 1 2024-08-20T22:20:41.0496425Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:41.0496752Z REENABLED_ISSUES: 2024-08-20T22:20:41.0497081Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:41.0497468Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:41.0497825Z TEST_SHOWLOCALS: False 2024-08-20T22:20:41.0498169Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:41.0498491Z NO_TD: False 2024-08-20T22:20:41.0498795Z TD_DISTRIBUTED: False 2024-08-20T22:20:41.0499206Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:41.0499690Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:41.0500063Z SHM_SIZE: 2g 2024-08-20T22:20:41.0501127Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:41.0502263Z XLA_CUDA: 2024-08-20T22:20:41.0502741Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:41.0503338Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:41.0503773Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:41.0504182Z DASHBOARD_TAG: 2024-08-20T22:20:41.0504684Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:41.0505058Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:41.0505429Z ##[endgroup] 2024-08-20T22:20:41.0532698Z + [[ dynamic_inductor_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:41.0533530Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:41.0534027Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:41.0542460Z +++ nproc --ignore=2 2024-08-20T22:20:41.0571902Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:52.4759539Z + container_name=586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-20T22:20:52.4764775Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:52.4767813Z + docker exec -i 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:52.6409205Z + echo DOCKER_CONTAINER_ID=586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-20T22:20:52.6414015Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:52.6417103Z + docker exec -t 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:53.0380008Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:53.3623579Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:53.3626140Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:53.3630467Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:53.3633428Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:53.3636986Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:53.3642306Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:53.3659681Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:53.3673211Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:53.3691199Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:53.4085773Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:20:54.0045326Z Installing collected packages: torch 2024-08-20T22:21:03.7631228Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:03.7635127Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:03.7636140Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:03.8443515Z + export TERM=vt100 2024-08-20T22:21:03.8443876Z + TERM=vt100 2024-08-20T22:21:03.8447485Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:03.8458682Z + source .ci/pytorch/common.sh 2024-08-20T22:21:03.8461755Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:03.8470877Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:03.8472886Z +++ declare -f -t trap_add 2024-08-20T22:21:03.8478035Z ++ set -ex 2024-08-20T22:21:03.8478921Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:03.8479430Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:03.8479939Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:03.8482546Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:03.8502111Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:03.8502539Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:03.8505222Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:03.8505739Z + shift 2024-08-20T22:21:03.8506045Z + for trap_add_name in "$@" 2024-08-20T22:21:03.8509897Z +++ trap -p EXIT 2024-08-20T22:21:03.8513102Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:03.8513477Z +++ extract_trap_cmd 2024-08-20T22:21:03.8513843Z +++ printf '%s\n' '' 2024-08-20T22:21:03.8514312Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:03.8516216Z + trap -- ' 2024-08-20T22:21:03.8516675Z cleanup_workspace' EXIT 2024-08-20T22:21:03.8517185Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:04.5635803Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:04.5657630Z + echo 'Environment variables:' 2024-08-20T22:21:04.5658135Z Environment variables: 2024-08-20T22:21:04.5658466Z + env 2024-08-20T22:21:04.5668056Z INSTALLED_DB=yes 2024-08-20T22:21:04.5668673Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5669204Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.5669759Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5671377Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.5672150Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.5672651Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5673853Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.5674672Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.5675367Z HOSTNAME=586d7babfd6e 2024-08-20T22:21:04.5676211Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5677070Z GITHUB_ACTION=__self 2024-08-20T22:21:04.5677506Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.5682232Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.5686097Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.5686607Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.5687075Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.5687439Z TEST_CONFIG=dynamic_inductor_torchbench 2024-08-20T22:21:04.5687876Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.5688357Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.5688858Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5689300Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5689939Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.5690487Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5690933Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5691413Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.5691872Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.5692195Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.5692568Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.5692955Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5693396Z XLA_CUDA= 2024-08-20T22:21:04.5693899Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.5696118Z *** 2024-08-20T22:21:04.5696428Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.5696817Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.5697179Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.5697513Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.5698023Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.5698560Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.5699030Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5699538Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.5700003Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5700781Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5701486Z UCC_HOME=/usr 2024-08-20T22:21:04.5701830Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5702220Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.5702558Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.5703018Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.5703500Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5703956Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.5704324Z SHARD_NUMBER=1 2024-08-20T22:21:04.5704644Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.5704982Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.5705339Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.5705771Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.5706245Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.5706717Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.5707081Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.5707505Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.5707937Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.5708231Z UCX_HOME=/usr 2024-08-20T22:21:04.5708927Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.5709996Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5711284Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.5712538Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5713634Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.5714280Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.5714639Z DASHBOARD_TAG= 2024-08-20T22:21:04.5714960Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.5715419Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.5715952Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.5716932Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5717819Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5718220Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5718592Z PR_NUMBER= 2024-08-20T22:21:04.5718868Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.5719205Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.5719656Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.5720233Z TERM=vt100 2024-08-20T22:21:04.5720574Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5720990Z INSTALLED_VISION=yes 2024-08-20T22:21:04.5721306Z BRANCH= 2024-08-20T22:21:04.5721719Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.5722194Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.5722611Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.5723368Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.5724097Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.5724582Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.5725041Z REENABLED_ISSUES= 2024-08-20T22:21:04.5725337Z SHLVL=1 2024-08-20T22:21:04.5725592Z MAX_JOBS=14 2024-08-20T22:21:04.5725908Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.5726262Z NVARCH=x86_64 2024-08-20T22:21:04.5726559Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.5727097Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5727625Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.5728115Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.5728776Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.5729345Z GITHUB_JOB=test 2024-08-20T22:21:04.5729746Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5730290Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.5730791Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.5731119Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.5731501Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.5731940Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.5732352Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.5732716Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.5733061Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.5733408Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.5734423Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.5735524Z GITHUB_BASE_REF= 2024-08-20T22:21:04.5735848Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.5736214Z CI=true 2024-08-20T22:21:04.5736533Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.5736948Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.5737326Z JOB_ID=29026529448 2024-08-20T22:21:04.5737634Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.5737958Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.5738256Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.5738689Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.5739139Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.5739475Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.5739831Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.5740718Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5741518Z NO_TD=False 2024-08-20T22:21:04.5741825Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.5742185Z _=/usr/bin/env 2024-08-20T22:21:04.5742654Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:04.5908123Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:04.5909291Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:04.5910447Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:04.5911578Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:04.5912178Z + BUILD_DIR=build 2024-08-20T22:21:04.5912518Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:04.5912905Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:04.5913240Z + SHARD_NUMBER=1 2024-08-20T22:21:04.5913535Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:04.5913853Z + export VALGRIND=ON 2024-08-20T22:21:04.5914169Z + VALGRIND=ON 2024-08-20T22:21:04.5914632Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:04.5915121Z + [[ 0 == \1 ]] 2024-08-20T22:21:04.5915413Z + [[ False == \1 ]] 2024-08-20T22:21:04.5915885Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:04.5916666Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:04.5929541Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:04.5930449Z + [[ -n '' ]] 2024-08-20T22:21:04.5930828Z + echo 'Environment variables' 2024-08-20T22:21:04.5931200Z Environment variables 2024-08-20T22:21:04.5931514Z + env 2024-08-20T22:21:04.5940085Z INSTALLED_DB=yes 2024-08-20T22:21:04.5940555Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5941073Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:04.5941646Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5942421Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:04.5943149Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:04.5943792Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5944455Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.5945199Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:04.5946118Z HOSTNAME=586d7babfd6e 2024-08-20T22:21:04.5946984Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5947850Z GITHUB_ACTION=__self 2024-08-20T22:21:04.5948204Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:04.5951892Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:04.5956072Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:04.5956588Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:04.5956943Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:04.5957316Z TEST_CONFIG=dynamic_inductor_torchbench 2024-08-20T22:21:04.5957750Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:04.5958224Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:04.5958679Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:04.5959120Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5959527Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:04.5959927Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5960366Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5960856Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:04.5961320Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:04.5961650Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:04.5962023Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:04.5962424Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5962872Z XLA_CUDA= 2024-08-20T22:21:04.5963329Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:04.5963713Z *** 2024-08-20T22:21:04.5963993Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:04.5964387Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:04.5964747Z GITHUB_ACTIONS=true 2024-08-20T22:21:04.5965067Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:04.5965559Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:04.5966100Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:04.5966558Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5967155Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:04.5967635Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5968413Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5969115Z UCC_HOME=/usr 2024-08-20T22:21:04.5969454Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:04.5970039Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:04.5970501Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:04.5970960Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:04.5971459Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:04.5971927Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:04.5972307Z SHARD_NUMBER=1 2024-08-20T22:21:04.5972609Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:04.5972962Z HOME=/var/lib/jenkins 2024-08-20T22:21:04.5973323Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:04.5973752Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:04.5974224Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:04.5974709Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:04.5975076Z CUDA_VERSION=12.1.1 2024-08-20T22:21:04.5975493Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:04.5975926Z NUM_TEST_SHARDS=2 2024-08-20T22:21:04.5976231Z UCX_HOME=/usr 2024-08-20T22:21:04.5976753Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:04.5977819Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5979116Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_torchbench, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:04.5980383Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5981488Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:04.5982135Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:04.5982461Z DASHBOARD_TAG= 2024-08-20T22:21:04.5982764Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:04.5983225Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:04.5983805Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:04.5984798Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.5985702Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:04.5986110Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:04.5986485Z PR_NUMBER= 2024-08-20T22:21:04.5986762Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:04.5987079Z VALGRIND=ON 2024-08-20T22:21:04.5987379Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:04.5987814Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:04.5988270Z TERM=vt100 2024-08-20T22:21:04.5988617Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:04.5989028Z INSTALLED_VISION=yes 2024-08-20T22:21:04.5989334Z BRANCH= 2024-08-20T22:21:04.5989617Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:04.5990018Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:04.5990437Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:04.5991189Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:04.5991931Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:04.5992419Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:04.5992888Z REENABLED_ISSUES= 2024-08-20T22:21:04.5993182Z SHLVL=1 2024-08-20T22:21:04.5993444Z MAX_JOBS=14 2024-08-20T22:21:04.5993766Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:04.5994108Z NVARCH=x86_64 2024-08-20T22:21:04.5994405Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:04.5994858Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:04.5995385Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:04.5995865Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:04.5996524Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:04.5997092Z GITHUB_JOB=test 2024-08-20T22:21:04.5997498Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:04.5998042Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:04.5998551Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:04.5998880Z TD_DISTRIBUTED=False 2024-08-20T22:21:04.5999253Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:04.5999792Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:04.6000292Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:04.6000659Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:04.6001000Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:04.6001353Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:04.6002383Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:04.6003447Z GITHUB_BASE_REF= 2024-08-20T22:21:04.6003769Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:04.6004143Z CI=true 2024-08-20T22:21:04.6004459Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:04.6004873Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:04.6005237Z JOB_ID=29026529448 2024-08-20T22:21:04.6005557Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:04.6005891Z GITHUB_HEAD_REF= 2024-08-20T22:21:04.6006187Z GITHUB_ACTION_REF= 2024-08-20T22:21:04.6006623Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:04.6007223Z TEST_SHOWLOCALS=False 2024-08-20T22:21:04.6007566Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:04.6007918Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:04.6008815Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_21c79969-1c64-48e8-bbd9-6f7a153b6c86 2024-08-20T22:21:04.6009623Z NO_TD=False 2024-08-20T22:21:04.6009924Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:04.6010282Z _=/usr/bin/env 2024-08-20T22:21:04.6010634Z + echo 'Testing pytorch' 2024-08-20T22:21:04.6010976Z Testing pytorch 2024-08-20T22:21:04.6011297Z + export LANG=C.UTF-8 2024-08-20T22:21:04.6011638Z + LANG=C.UTF-8 2024-08-20T22:21:04.6011933Z + PR_NUMBER= 2024-08-20T22:21:04.6012307Z + [[ dynamic_inductor_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:04.6012874Z + [[ dynamic_inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:04.6013419Z + [[ dynamic_inductor_torchbench == \s\l\o\w ]] 2024-08-20T22:21:04.6014085Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:04.6014834Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:04.6015373Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.6015848Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:04.6016318Z + [[ dynamic_inductor_torchbench == *crossref* ]] 2024-08-20T22:21:04.6016926Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:04.6017550Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:04.6018200Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:04.6018766Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:04.6019296Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:05.6985248Z Collecting ninja==1.10.2 2024-08-20T22:21:05.7148313Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:05.7611203Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:06.3650892Z Installing collected packages: ninja 2024-08-20T22:21:06.3729291Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:06.3730553Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:06.4203724Z Successfully installed ninja-1.10.2 2024-08-20T22:21:06.4971500Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.4973741Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:06.4975485Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:06.4975991Z + install_tlparse 2024-08-20T22:21:06.4976799Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:06.4977523Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:06.9203161Z Collecting tlparse==0.3.25 2024-08-20T22:21:06.9395721Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:06.9898019Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:07.6359400Z Installing collected packages: tlparse 2024-08-20T22:21:07.7110360Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:07.7885861Z ++ python -m site --user-base 2024-08-20T22:21:07.8129176Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:07.8131187Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:07.8131882Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:07.8132551Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:07.8133486Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:07.8134604Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:07.8135291Z + cd test 2024-08-20T22:21:07.8135824Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:09.4297455Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:09.4298469Z + [[ dynamic_inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:09.4301759Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:09.4302449Z + [[ dynamic_inductor_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:21:09.4303302Z + [[ dynamic_inductor_torchbench == *dynamo_eager* ]] 2024-08-20T22:21:09.4304130Z + [[ dynamic_inductor_torchbench == *aot_eager* ]] 2024-08-20T22:21:09.4304942Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T22:21:09.4305713Z + [[ dynamic_inductor_torchbench == *inductor* ]] 2024-08-20T22:21:09.4306449Z + [[ dynamic_inductor_torchbench != *perf* ]] 2024-08-20T22:21:09.4307391Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:09.4308047Z + [[ dynamic_inductor_torchbench == *dynamic* ]] 2024-08-20T22:21:09.4309017Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-08-20T22:21:09.4309877Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:09.4310627Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:09.4339473Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:09.4340176Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:09.4342156Z + cd test 2024-08-20T22:21:09.4342634Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:10.8922173Z PyTorch built with: 2024-08-20T22:21:10.8922653Z - GCC 9.4 2024-08-20T22:21:10.8923020Z - C++ Version: 201703 2024-08-20T22:21:10.8923878Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:10.8924949Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:10.8925610Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:10.8926128Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:10.8926663Z - NNPACK is enabled 2024-08-20T22:21:10.8927132Z - CPU capability usage: AVX2 2024-08-20T22:21:10.8927552Z - CUDA Runtime 12.1 2024-08-20T22:21:10.8928077Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:10.8928690Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:10.8929127Z - Magma 2.6.1 2024-08-20T22:21:10.8936112Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:10.8942503Z 2024-08-20T22:21:11.2110924Z + cd test 2024-08-20T22:21:11.2111696Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:12.5374716Z ATen/Parallel: 2024-08-20T22:21:12.5375466Z at::get_num_threads() : 8 2024-08-20T22:21:12.5376229Z at::get_num_interop_threads() : 16 2024-08-20T22:21:12.5376843Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:12.5377272Z omp_get_max_threads() : 8 2024-08-20T22:21:12.5378332Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:12.5379129Z mkl_get_max_threads() : 8 2024-08-20T22:21:12.5379759Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:12.5380387Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:12.5380816Z Environment variables: 2024-08-20T22:21:12.5381172Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:12.5381558Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:12.5381954Z ATen parallel backend: OpenMP 2024-08-20T22:21:12.5382217Z 2024-08-20T22:21:12.8093389Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:12.8094127Z + [[ dynamic_inductor_torchbench == *backward* ]] 2024-08-20T22:21:12.8094637Z + [[ dynamic_inductor_torchbench == *xla* ]] 2024-08-20T22:21:12.8095210Z + [[ dynamic_inductor_torchbench == *executorch* ]] 2024-08-20T22:21:12.8095809Z + [[ dynamic_inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:12.8096529Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:12.8097149Z + [[ dynamic_inductor_torchbench == distributed ]] 2024-08-20T22:21:12.8097795Z + [[ dynamic_inductor_torchbench == *inductor_distributed* ]] 2024-08-20T22:21:12.8098511Z + [[ dynamic_inductor_torchbench == *inductor-halide* ]] 2024-08-20T22:21:12.8099233Z + [[ dynamic_inductor_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:21:12.8099870Z + [[ dynamic_inductor_torchbench == *huggingface* ]] 2024-08-20T22:21:12.8100454Z + [[ dynamic_inductor_torchbench == *timm* ]] 2024-08-20T22:21:12.8100999Z + [[ dynamic_inductor_torchbench == *torchbench* ]] 2024-08-20T22:21:12.8101518Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T22:21:12.8102023Z + install_torchaudio cuda 2024-08-20T22:21:12.8102378Z + local commit 2024-08-20T22:21:12.8102740Z ++ get_pinned_commit audio 2024-08-20T22:21:12.8103122Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:21:12.8117662Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.8118147Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:21:12.8118637Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:21:12.8119621Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:12.8121193Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.1460903Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:13.1464829Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-dzlvoy17 2024-08-20T22:21:13.1498074Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-dzlvoy17 2024-08-20T22:21:13.9257173Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:21:13.9283622Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.3489658Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:21:14.3490897Z Running command git submodule update --init --recursive -q 2024-08-20T22:21:16.7584537Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:21:16.7613631Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:16.7642042Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:21:16.7646566Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:21:16.7649714Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:21:16.7651612Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:21:16.7655108Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:21:16.7660221Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:21:16.7675005Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:21:16.8156247Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:21:16.8218001Z Building wheels for collected packages: torchaudio 2024-08-20T22:23:50.9688414Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:23:50.9724022Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310395 sha256=f5c5cddc641b85fc5e16eb825be0e52f2a94bf67ae2e2e33fe76e36fc3c06019 2024-08-20T22:23:50.9727525Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:23:50.9753930Z Successfully built torchaudio 2024-08-20T22:23:51.4637933Z Installing collected packages: torchaudio 2024-08-20T22:23:51.6755125Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:23:51.8521096Z + install_torchtext 2024-08-20T22:23:51.8521499Z + local data_commit 2024-08-20T22:23:51.8521816Z + local text_commit 2024-08-20T22:23:51.8525406Z ++ get_pinned_commit data 2024-08-20T22:23:51.8525881Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:23:51.8550825Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.8553253Z ++ get_pinned_commit text 2024-08-20T22:23:51.8553701Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:23:51.8569360Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:23:51.8570605Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:51.8572028Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.1961742Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.1966931Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-ffxb8fxf 2024-08-20T22:23:52.2011803Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-ffxb8fxf 2024-08-20T22:23:52.6446857Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:23:52.6474053Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:52.9242079Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:53.0765425Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:23:53.0766538Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:12.8792987Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:12.8830596Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:25:12.8833542Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:25:12.8839466Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:12.8870296Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:25:12.8875019Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:25:12.8879558Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:25:12.8883735Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:25:12.8887387Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:25:12.8894396Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:25:12.8909871Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:25:12.8972304Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:25:12.8975082Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:25:12.8981282Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:25:12.9479894Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:25:12.9543708Z Building wheels for collected packages: torchdata 2024-08-20T22:25:14.8973092Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:25:14.8980988Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=455d4ede48ea5d7d41a84ea827c85061e04feb9eaefaf2f486ce622326958da3 2024-08-20T22:25:14.8982807Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:25:14.9014169Z Successfully built torchdata 2024-08-20T22:25:15.3808750Z Installing collected packages: torchdata 2024-08-20T22:25:15.4780606Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:25:20.4194322Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:20.4196222Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:20.7537675Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:20.7543487Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-dj9kb6vv 2024-08-20T22:25:20.7573771Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-dj9kb6vv 2024-08-20T22:25:21.5988171Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:25:21.6014764Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:22.1096068Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:22.2830808Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:25:22.2831935Z Running command git submodule update --init --recursive -q 2024-08-20T22:25:29.7537361Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:25:29.7572036Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:25:29.7575827Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:25:29.7578212Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:25:29.7581567Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:25:29.7584849Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:25:29.7611316Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:25:29.7615280Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:25:29.7618693Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:25:29.7622623Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:25:29.7651294Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:25:29.7655615Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:25:29.7657657Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:25:29.7659734Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:25:29.7663109Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:25:29.7668271Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:25:29.7682632Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:25:29.8261556Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:25:29.8328368Z Building wheels for collected packages: torchtext 2024-08-20T22:26:08.1160565Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T22:26:08.1195470Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045466 sha256=3832be628aa6dfb17fa80d0b66b9d5a9ab7bf37059741b98e728f7bd5882905b 2024-08-20T22:26:08.1197695Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:26:08.1219421Z Successfully built torchtext 2024-08-20T22:26:08.6010139Z Installing collected packages: torchtext 2024-08-20T22:26:08.7152996Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:26:08.8746863Z + install_torchvision 2024-08-20T22:26:08.8747382Z + local orig_preload 2024-08-20T22:26:08.8747808Z + local commit 2024-08-20T22:26:08.8750422Z ++ get_pinned_commit vision 2024-08-20T22:26:08.8750954Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:26:08.8771655Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:08.8772263Z + orig_preload= 2024-08-20T22:26:08.8772817Z + '[' -n '' ']' 2024-08-20T22:26:08.8773871Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:08.8778480Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:09.2162320Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:09.2167108Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-yh5z4kbp 2024-08-20T22:26:09.2195117Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-yh5z4kbp 2024-08-20T22:26:10.7893888Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:26:10.7920666Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.1412894Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:12.4432337Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:26:14.9903339Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:26:14.9938647Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:26:14.9941550Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:26:14.9947657Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:26:15.0019281Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:26:15.0024107Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:26:15.0027441Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:26:15.0031994Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:26:15.0035507Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:26:15.0042394Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:26:15.0059012Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:26:15.0541636Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:26:15.0607845Z Building wheels for collected packages: torchvision 2024-08-20T22:27:31.6031049Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:27:31.6063941Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116059 sha256=a7c01a7ceebe9e49ecff9e5e7c18bcbe077aabb31b69dc56099f87e71e52be8d 2024-08-20T22:27:31.6065592Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:27:31.6100466Z Successfully built torchvision 2024-08-20T22:27:32.0997133Z Installing collected packages: torchvision 2024-08-20T22:27:32.5129494Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:27:32.6539472Z + '[' -n '' ']' 2024-08-20T22:27:32.6539873Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:27:32.6540368Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:27:32.6541110Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:27:32.9902179Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:27:32.9906881Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-s0jwfop8 2024-08-20T22:27:32.9937785Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-s0jwfop8 2024-08-20T22:27:33.5577186Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:27:36.0175541Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:27:36.0210684Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:28:54.3777322Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / done 2024-08-20T22:28:54.3795198Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928016 sha256=66ca33b0dae7b0525a31c7637fc90615237a7af542d847c6f4ceacc7432f15e5 2024-08-20T22:28:54.3799474Z Stored in directory: /tmp/pip-ephem-wheel-cache-ys1f90ah/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:28:54.3827714Z Successfully built torchao 2024-08-20T22:28:54.9266457Z Installing collected packages: torchao 2024-08-20T22:28:55.1685514Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:28:55.2742523Z + id=0 2024-08-20T22:28:55.2745079Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:28:55.2746091Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:28:55.7421660Z Collecting opencv-python==4.8.0.74 2024-08-20T22:28:55.7587553Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:28:55.7666576Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:28:55.7731927Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:28:56.8395150Z Installing collected packages: opencv-python 2024-08-20T22:28:57.7000572Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:28:57.7978121Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:28:57.7978932Z + [[ dynamic_inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:28:57.7979642Z + [[ dynamic_inductor_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:28:57.7980213Z + checkout_install_torchbench 2024-08-20T22:28:57.7980653Z + local commit 2024-08-20T22:28:57.7984020Z ++ get_pinned_commit torchbench 2024-08-20T22:28:57.7984473Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:28:57.7999980Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:28:57.8000581Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:28:57.8017186Z Cloning into 'torchbench'... 2024-08-20T22:28:57.9532365Z remote: Enumerating objects: 31006, done. 2024-08-20T22:28:57.9533323Z remote: Counting objects: 0% (1/1746) 2024-08-20T22:28:57.9535889Z remote: Counting objects: 1% (18/1746) 2024-08-20T22:28:57.9537273Z remote: Counting objects: 2% (35/1746) 2024-08-20T22:28:57.9539175Z remote: Counting objects: 3% (53/1746) 2024-08-20T22:28:57.9539910Z remote: Counting objects: 4% (70/1746) 2024-08-20T22:28:57.9540646Z remote: Counting objects: 5% (88/1746) 2024-08-20T22:28:57.9541395Z remote: Counting objects: 6% (105/1746) 2024-08-20T22:28:57.9542001Z remote: Counting objects: 7% (123/1746) 2024-08-20T22:28:57.9542643Z remote: Counting objects: 8% (140/1746) 2024-08-20T22:28:57.9543278Z remote: Counting objects: 9% (158/1746) 2024-08-20T22:28:57.9543877Z remote: Counting objects: 10% (175/1746) 2024-08-20T22:28:57.9544602Z remote: Counting objects: 11% (193/1746) 2024-08-20T22:28:57.9545322Z remote: Counting objects: 12% (210/1746) 2024-08-20T22:28:57.9546249Z remote: Counting objects: 13% (227/1746) 2024-08-20T22:28:57.9546801Z remote: Counting objects: 14% (245/1746) 2024-08-20T22:28:57.9547424Z remote: Counting objects: 15% (262/1746) 2024-08-20T22:28:57.9547940Z remote: Counting objects: 16% (280/1746) 2024-08-20T22:28:57.9548836Z remote: Counting objects: 17% (297/1746) 2024-08-20T22:28:57.9549530Z remote: Counting objects: 18% (315/1746) 2024-08-20T22:28:57.9550215Z remote: Counting objects: 19% (332/1746) 2024-08-20T22:28:57.9550894Z remote: Counting objects: 20% (350/1746) 2024-08-20T22:28:57.9551513Z remote: Counting objects: 21% (367/1746) 2024-08-20T22:28:57.9552217Z remote: Counting objects: 22% (385/1746) 2024-08-20T22:28:57.9552766Z remote: Counting objects: 23% (402/1746) 2024-08-20T22:28:57.9553471Z remote: Counting objects: 24% (420/1746) 2024-08-20T22:28:57.9554185Z remote: Counting objects: 25% (437/1746) 2024-08-20T22:28:57.9554880Z remote: Counting objects: 26% (454/1746) 2024-08-20T22:28:57.9555453Z remote: Counting objects: 27% (472/1746) 2024-08-20T22:28:57.9557210Z remote: Counting objects: 28% (489/1746) 2024-08-20T22:28:57.9557899Z remote: Counting objects: 29% (507/1746) 2024-08-20T22:28:57.9558426Z remote: Counting objects: 30% (524/1746) 2024-08-20T22:28:57.9558966Z remote: Counting objects: 31% (542/1746) 2024-08-20T22:28:57.9559664Z remote: Counting objects: 32% (559/1746) 2024-08-20T22:28:57.9560340Z remote: Counting objects: 33% (577/1746) 2024-08-20T22:28:57.9561019Z remote: Counting objects: 34% (594/1746) 2024-08-20T22:28:57.9561553Z remote: Counting objects: 35% (612/1746) 2024-08-20T22:28:57.9562069Z remote: Counting objects: 36% (629/1746) 2024-08-20T22:28:57.9562621Z remote: Counting objects: 37% (647/1746) 2024-08-20T22:28:57.9563155Z remote: Counting objects: 38% (664/1746) 2024-08-20T22:28:57.9563673Z remote: Counting objects: 39% (681/1746) 2024-08-20T22:28:57.9564196Z remote: Counting objects: 40% (699/1746) 2024-08-20T22:28:57.9564707Z remote: Counting objects: 41% (716/1746) 2024-08-20T22:28:57.9565231Z remote: Counting objects: 42% (734/1746) 2024-08-20T22:28:57.9565759Z remote: Counting objects: 43% (751/1746) 2024-08-20T22:28:57.9566270Z remote: Counting objects: 44% (769/1746) 2024-08-20T22:28:57.9566917Z remote: Counting objects: 45% (786/1746) 2024-08-20T22:28:57.9567433Z remote: Counting objects: 46% (804/1746) 2024-08-20T22:28:57.9567945Z remote: Counting objects: 47% (821/1746) 2024-08-20T22:28:57.9568453Z remote: Counting objects: 48% (839/1746) 2024-08-20T22:28:57.9568954Z remote: Counting objects: 49% (856/1746) 2024-08-20T22:28:57.9569467Z remote: Counting objects: 50% (873/1746) 2024-08-20T22:28:57.9569985Z remote: Counting objects: 51% (891/1746) 2024-08-20T22:28:57.9570499Z remote: Counting objects: 52% (908/1746) 2024-08-20T22:28:57.9571002Z remote: Counting objects: 53% (926/1746) 2024-08-20T22:28:57.9571513Z remote: Counting objects: 54% (943/1746) 2024-08-20T22:28:57.9572032Z remote: Counting objects: 55% (961/1746) 2024-08-20T22:28:57.9572543Z remote: Counting objects: 56% (978/1746) 2024-08-20T22:28:57.9573054Z remote: Counting objects: 57% (996/1746) 2024-08-20T22:28:57.9573569Z remote: Counting objects: 58% (1013/1746) 2024-08-20T22:28:57.9574100Z remote: Counting objects: 59% (1031/1746) 2024-08-20T22:28:57.9574619Z remote: Counting objects: 60% (1048/1746) 2024-08-20T22:28:57.9575147Z remote: Counting objects: 61% (1066/1746) 2024-08-20T22:28:57.9575669Z remote: Counting objects: 62% (1083/1746) 2024-08-20T22:28:57.9576196Z remote: Counting objects: 63% (1100/1746) 2024-08-20T22:28:57.9576714Z remote: Counting objects: 64% (1118/1746) 2024-08-20T22:28:57.9577237Z remote: Counting objects: 65% (1135/1746) 2024-08-20T22:28:57.9577755Z remote: Counting objects: 66% (1153/1746) 2024-08-20T22:28:57.9578277Z remote: Counting objects: 67% (1170/1746) 2024-08-20T22:28:57.9578794Z remote: Counting objects: 68% (1188/1746) 2024-08-20T22:28:57.9579323Z remote: Counting objects: 69% (1205/1746) 2024-08-20T22:28:57.9579846Z remote: Counting objects: 70% (1223/1746) 2024-08-20T22:28:57.9580366Z remote: Counting objects: 71% (1240/1746) 2024-08-20T22:28:57.9580874Z remote: Counting objects: 72% (1258/1746) 2024-08-20T22:28:57.9581396Z remote: Counting objects: 73% (1275/1746) 2024-08-20T22:28:57.9581911Z remote: Counting objects: 74% (1293/1746) 2024-08-20T22:28:57.9582428Z remote: Counting objects: 75% (1310/1746) 2024-08-20T22:28:57.9582938Z remote: Counting objects: 76% (1327/1746) 2024-08-20T22:28:57.9583454Z remote: Counting objects: 77% (1345/1746) 2024-08-20T22:28:57.9583972Z remote: Counting objects: 78% (1362/1746) 2024-08-20T22:28:57.9584482Z remote: Counting objects: 79% (1380/1746) 2024-08-20T22:28:57.9584996Z remote: Counting objects: 80% (1397/1746) 2024-08-20T22:28:57.9585509Z remote: Counting objects: 81% (1415/1746) 2024-08-20T22:28:57.9586025Z remote: Counting objects: 82% (1432/1746) 2024-08-20T22:28:57.9586646Z remote: Counting objects: 83% (1450/1746) 2024-08-20T22:28:57.9587246Z remote: Counting objects: 84% (1467/1746) 2024-08-20T22:28:57.9587762Z remote: Counting objects: 85% (1485/1746) 2024-08-20T22:28:57.9588282Z remote: Counting objects: 86% (1502/1746) 2024-08-20T22:28:57.9588792Z remote: Counting objects: 87% (1520/1746) 2024-08-20T22:28:57.9589308Z remote: Counting objects: 88% (1537/1746) 2024-08-20T22:28:57.9589831Z remote: Counting objects: 89% (1554/1746) 2024-08-20T22:28:57.9590349Z remote: Counting objects: 90% (1572/1746) 2024-08-20T22:28:57.9590858Z remote: Counting objects: 91% (1589/1746) 2024-08-20T22:28:57.9591379Z remote: Counting objects: 92% (1607/1746) 2024-08-20T22:28:57.9591896Z remote: Counting objects: 93% (1624/1746) 2024-08-20T22:28:57.9592461Z remote: Counting objects: 94% (1642/1746) 2024-08-20T22:28:57.9592970Z remote: Counting objects: 95% (1659/1746) 2024-08-20T22:28:57.9593493Z remote: Counting objects: 96% (1677/1746) 2024-08-20T22:28:57.9594020Z remote: Counting objects: 97% (1694/1746) 2024-08-20T22:28:57.9594529Z remote: Counting objects: 98% (1712/1746) 2024-08-20T22:28:57.9595047Z remote: Counting objects: 99% (1729/1746) 2024-08-20T22:28:57.9595567Z remote: Counting objects: 100% (1746/1746) 2024-08-20T22:28:57.9596134Z remote: Counting objects: 100% (1746/1746), done. 2024-08-20T22:28:57.9596689Z remote: Compressing objects: 0% (1/901) 2024-08-20T22:28:57.9645121Z remote: Compressing objects: 1% (10/901) 2024-08-20T22:28:57.9761640Z remote: Compressing objects: 2% (19/901) 2024-08-20T22:28:57.9850842Z remote: Compressing objects: 3% (28/901) 2024-08-20T22:28:57.9899787Z remote: Compressing objects: 4% (37/901) 2024-08-20T22:28:57.9949893Z remote: Compressing objects: 5% (46/901) 2024-08-20T22:28:57.9983799Z remote: Compressing objects: 6% (55/901) 2024-08-20T22:28:58.0011617Z remote: Compressing objects: 7% (64/901) 2024-08-20T22:28:58.0044942Z remote: Compressing objects: 8% (73/901) 2024-08-20T22:28:58.0081925Z remote: Compressing objects: 9% (82/901) 2024-08-20T22:28:58.0107976Z remote: Compressing objects: 10% (91/901) 2024-08-20T22:28:58.0123726Z remote: Compressing objects: 11% (100/901) 2024-08-20T22:28:58.0135020Z remote: Compressing objects: 12% (109/901) 2024-08-20T22:28:58.0150421Z remote: Compressing objects: 13% (118/901) 2024-08-20T22:28:58.0166364Z remote: Compressing objects: 14% (127/901) 2024-08-20T22:28:58.0183863Z remote: Compressing objects: 15% (136/901) 2024-08-20T22:28:58.0192006Z remote: Compressing objects: 16% (145/901) 2024-08-20T22:28:58.0198492Z remote: Compressing objects: 17% (154/901) 2024-08-20T22:28:58.0205018Z remote: Compressing objects: 18% (163/901) 2024-08-20T22:28:58.0205754Z remote: Compressing objects: 19% (172/901) 2024-08-20T22:28:58.0211561Z remote: Compressing objects: 20% (181/901) 2024-08-20T22:28:58.0212309Z remote: Compressing objects: 21% (190/901) 2024-08-20T22:28:58.0212888Z remote: Compressing objects: 22% (199/901) 2024-08-20T22:28:58.0213440Z remote: Compressing objects: 23% (208/901) 2024-08-20T22:28:58.0217797Z remote: Compressing objects: 24% (217/901) 2024-08-20T22:28:58.0218528Z remote: Compressing objects: 25% (226/901) 2024-08-20T22:28:58.0219125Z remote: Compressing objects: 26% (235/901) 2024-08-20T22:28:58.0219673Z remote: Compressing objects: 27% (244/901) 2024-08-20T22:28:58.0223803Z remote: Compressing objects: 28% (253/901) 2024-08-20T22:28:58.0229954Z remote: Compressing objects: 29% (262/901) 2024-08-20T22:28:58.0237117Z remote: Compressing objects: 30% (271/901) 2024-08-20T22:28:58.0237847Z remote: Compressing objects: 31% (280/901) 2024-08-20T22:28:58.0242942Z remote: Compressing objects: 32% (289/901) 2024-08-20T22:28:58.0243667Z remote: Compressing objects: 33% (298/901) 2024-08-20T22:28:58.0244288Z remote: Compressing objects: 34% (307/901) 2024-08-20T22:28:58.0245324Z remote: Compressing objects: 35% (316/901) 2024-08-20T22:28:58.0246557Z remote: Compressing objects: 36% (325/901) 2024-08-20T22:28:58.0247124Z remote: Compressing objects: 37% (334/901) 2024-08-20T22:28:58.0248216Z remote: Compressing objects: 38% (343/901) 2024-08-20T22:28:58.0250506Z remote: Compressing objects: 39% (352/901) 2024-08-20T22:28:58.0251237Z remote: Compressing objects: 40% (361/901) 2024-08-20T22:28:58.0253043Z remote: Compressing objects: 41% (370/901) 2024-08-20T22:28:58.0253769Z remote: Compressing objects: 42% (379/901) 2024-08-20T22:28:58.0254366Z remote: Compressing objects: 43% (388/901) 2024-08-20T22:28:58.0255310Z remote: Compressing objects: 44% (397/901) 2024-08-20T22:28:58.0258359Z remote: Compressing objects: 45% (406/901) 2024-08-20T22:28:58.0259105Z remote: Compressing objects: 46% (415/901) 2024-08-20T22:28:58.0259771Z remote: Compressing objects: 47% (424/901) 2024-08-20T22:28:58.0260491Z remote: Compressing objects: 48% (433/901) 2024-08-20T22:28:58.0261048Z remote: Compressing objects: 49% (442/901) 2024-08-20T22:28:58.0261724Z remote: Compressing objects: 50% (451/901) 2024-08-20T22:28:58.0262404Z remote: Compressing objects: 51% (460/901) 2024-08-20T22:28:58.0262981Z remote: Compressing objects: 52% (469/901) 2024-08-20T22:28:58.0263708Z remote: Compressing objects: 53% (478/901) 2024-08-20T22:28:58.0264450Z remote: Compressing objects: 54% (487/901) 2024-08-20T22:28:58.0265154Z remote: Compressing objects: 55% (496/901) 2024-08-20T22:28:58.0265724Z remote: Compressing objects: 56% (505/901) 2024-08-20T22:28:58.0266273Z remote: Compressing objects: 57% (514/901) 2024-08-20T22:28:58.0266817Z remote: Compressing objects: 58% (523/901) 2024-08-20T22:28:58.0267544Z remote: Compressing objects: 59% (532/901) 2024-08-20T22:28:58.0268262Z remote: Compressing objects: 60% (541/901) 2024-08-20T22:28:58.0268879Z remote: Compressing objects: 61% (550/901) 2024-08-20T22:28:58.0269421Z remote: Compressing objects: 62% (559/901) 2024-08-20T22:28:58.0269951Z remote: Compressing objects: 63% (568/901) 2024-08-20T22:28:58.0270491Z remote: Compressing objects: 64% (577/901) 2024-08-20T22:28:58.0271033Z remote: Compressing objects: 65% (586/901) 2024-08-20T22:28:58.0271570Z remote: Compressing objects: 66% (595/901) 2024-08-20T22:28:58.0272120Z remote: Compressing objects: 67% (604/901) 2024-08-20T22:28:58.0272675Z remote: Compressing objects: 68% (613/901) 2024-08-20T22:28:58.0273209Z remote: Compressing objects: 69% (622/901) 2024-08-20T22:28:58.0273746Z remote: Compressing objects: 70% (631/901) 2024-08-20T22:28:58.0274269Z remote: Compressing objects: 71% (640/901) 2024-08-20T22:28:58.0274807Z remote: Compressing objects: 72% (649/901) 2024-08-20T22:28:58.0275345Z remote: Compressing objects: 73% (658/901) 2024-08-20T22:28:58.0276024Z remote: Compressing objects: 74% (667/901) 2024-08-20T22:28:58.0276762Z remote: Compressing objects: 75% (676/901) 2024-08-20T22:28:58.0277441Z remote: Compressing objects: 76% (685/901) 2024-08-20T22:28:58.0284361Z remote: Compressing objects: 77% (694/901) 2024-08-20T22:28:58.0289371Z remote: Compressing objects: 78% (703/901) 2024-08-20T22:28:58.0295024Z remote: Compressing objects: 79% (712/901) 2024-08-20T22:28:58.0301477Z remote: Compressing objects: 80% (721/901) 2024-08-20T22:28:58.0310189Z remote: Compressing objects: 81% (730/901) 2024-08-20T22:28:58.0317034Z remote: Compressing objects: 82% (739/901) 2024-08-20T22:28:58.0320556Z remote: Compressing objects: 83% (748/901) 2024-08-20T22:28:58.0324098Z remote: Compressing objects: 84% (757/901) 2024-08-20T22:28:58.0328133Z remote: Compressing objects: 85% (766/901) 2024-08-20T22:28:58.0333505Z remote: Compressing objects: 86% (775/901) 2024-08-20T22:28:58.0337163Z remote: Compressing objects: 87% (784/901) 2024-08-20T22:28:58.0341058Z remote: Compressing objects: 88% (793/901) 2024-08-20T22:28:58.0344525Z remote: Compressing objects: 89% (802/901) 2024-08-20T22:28:58.0348164Z remote: Compressing objects: 90% (811/901) 2024-08-20T22:28:58.0351156Z remote: Compressing objects: 91% (820/901) 2024-08-20T22:28:58.0354647Z remote: Compressing objects: 92% (829/901) 2024-08-20T22:28:58.0357427Z remote: Compressing objects: 93% (838/901) 2024-08-20T22:28:58.0360192Z remote: Compressing objects: 94% (847/901) 2024-08-20T22:28:58.0362559Z remote: Compressing objects: 95% (856/901) 2024-08-20T22:28:58.0365716Z remote: Compressing objects: 96% (865/901) 2024-08-20T22:28:58.0368531Z remote: Compressing objects: 97% (874/901) 2024-08-20T22:28:58.0370936Z remote: Compressing objects: 98% (883/901) 2024-08-20T22:28:58.0371656Z remote: Compressing objects: 99% (892/901) 2024-08-20T22:28:58.0373833Z remote: Compressing objects: 100% (901/901) 2024-08-20T22:28:58.0374605Z remote: Compressing objects: 100% (901/901), done. 2024-08-20T22:28:58.0451214Z Receiving objects: 0% (1/31006) 2024-08-20T22:28:58.0493183Z Receiving objects: 1% (311/31006) 2024-08-20T22:28:58.0532731Z Receiving objects: 2% (621/31006) 2024-08-20T22:28:58.0569695Z Receiving objects: 3% (931/31006) 2024-08-20T22:28:58.0602651Z Receiving objects: 4% (1241/31006) 2024-08-20T22:28:58.0635441Z Receiving objects: 5% (1551/31006) 2024-08-20T22:28:58.0673144Z Receiving objects: 6% (1861/31006) 2024-08-20T22:28:58.0711900Z Receiving objects: 7% (2171/31006) 2024-08-20T22:28:58.0742008Z Receiving objects: 8% (2481/31006) 2024-08-20T22:28:58.0775684Z Receiving objects: 9% (2791/31006) 2024-08-20T22:28:58.0844861Z Receiving objects: 10% (3101/31006) 2024-08-20T22:28:58.0894112Z Receiving objects: 11% (3411/31006) 2024-08-20T22:28:58.0960411Z Receiving objects: 12% (3721/31006) 2024-08-20T22:28:58.1090843Z Receiving objects: 13% (4031/31006) 2024-08-20T22:28:58.1235290Z Receiving objects: 14% (4341/31006) 2024-08-20T22:28:58.1303491Z Receiving objects: 15% (4651/31006) 2024-08-20T22:28:58.1378789Z Receiving objects: 16% (4961/31006) 2024-08-20T22:28:58.1434277Z Receiving objects: 17% (5272/31006) 2024-08-20T22:28:58.1465975Z Receiving objects: 18% (5582/31006) 2024-08-20T22:28:58.1522951Z Receiving objects: 19% (5892/31006) 2024-08-20T22:28:58.1558754Z Receiving objects: 20% (6202/31006) 2024-08-20T22:28:58.1573953Z Receiving objects: 21% (6512/31006) 2024-08-20T22:28:58.1619132Z Receiving objects: 22% (6822/31006) 2024-08-20T22:28:59.0398162Z Receiving objects: 23% (7132/31006) 2024-08-20T22:28:59.1456972Z Receiving objects: 23% (7393/31006), 75.43 MiB | 75.42 MiB/s 2024-08-20T22:28:59.7049182Z Receiving objects: 24% (7442/31006), 75.43 MiB | 75.42 MiB/s 2024-08-20T22:28:59.7771618Z Receiving objects: 25% (7752/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:28:59.8487895Z Receiving objects: 26% (8062/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:28:59.9203473Z Receiving objects: 27% (8372/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:28:59.9911622Z Receiving objects: 28% (8682/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:29:00.0396148Z Receiving objects: 29% (8992/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:29:00.0628888Z Receiving objects: 29% (9202/31006), 113.20 MiB | 75.51 MiB/s 2024-08-20T22:29:00.1350647Z Receiving objects: 30% (9302/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:00.2066165Z Receiving objects: 31% (9612/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:00.4415961Z Receiving objects: 32% (9922/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:00.5162631Z Receiving objects: 33% (10232/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:00.5286135Z Receiving objects: 34% (10543/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:00.7881436Z Receiving objects: 35% (10853/31006), 151.64 MiB | 75.85 MiB/s 2024-08-20T22:29:01.0375528Z Receiving objects: 36% (11163/31006), 184.91 MiB | 73.99 MiB/s 2024-08-20T22:29:01.0393472Z Receiving objects: 37% (11473/31006), 184.91 MiB | 73.99 MiB/s 2024-08-20T22:29:01.0396132Z Receiving objects: 38% (11783/31006), 184.91 MiB | 73.99 MiB/s 2024-08-20T22:29:01.0417203Z Receiving objects: 38% (11850/31006), 184.91 MiB | 73.99 MiB/s 2024-08-20T22:29:01.0435426Z Receiving objects: 39% (12093/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0451181Z Receiving objects: 40% (12403/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0462249Z Receiving objects: 41% (12713/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0474971Z Receiving objects: 42% (13023/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0490859Z Receiving objects: 43% (13333/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0507034Z Receiving objects: 44% (13643/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0514818Z Receiving objects: 45% (13953/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0519730Z Receiving objects: 46% (14263/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0534028Z Receiving objects: 47% (14573/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0555851Z Receiving objects: 48% (14883/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0568818Z Receiving objects: 49% (15193/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0769993Z Receiving objects: 50% (15503/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0776336Z Receiving objects: 51% (15814/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0783218Z Receiving objects: 52% (16124/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0789606Z Receiving objects: 53% (16434/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0795432Z Receiving objects: 54% (16744/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0812554Z Receiving objects: 55% (17054/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0822977Z Receiving objects: 56% (17364/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0829127Z Receiving objects: 57% (17674/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0847175Z Receiving objects: 58% (17984/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0856652Z Receiving objects: 59% (18294/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.0866790Z Receiving objects: 60% (18604/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1006110Z Receiving objects: 61% (18914/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1012313Z Receiving objects: 62% (19224/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1018061Z Receiving objects: 63% (19534/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1026093Z Receiving objects: 64% (19844/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1034188Z Receiving objects: 65% (20154/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1042875Z Receiving objects: 66% (20464/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1303420Z Receiving objects: 67% (20775/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1705390Z Receiving objects: 68% (21085/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1713426Z Receiving objects: 69% (21395/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1726797Z Receiving objects: 70% (21705/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1738453Z Receiving objects: 71% (22015/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1753677Z Receiving objects: 72% (22325/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1762955Z Receiving objects: 73% (22635/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1779602Z Receiving objects: 74% (22945/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1789380Z Receiving objects: 75% (23255/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1796861Z Receiving objects: 76% (23565/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.1805312Z Receiving objects: 77% (23875/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2019386Z Receiving objects: 78% (24185/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2053827Z Receiving objects: 79% (24495/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2062630Z Receiving objects: 80% (24805/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2075761Z Receiving objects: 81% (25115/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2087931Z Receiving objects: 82% (25425/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2136432Z Receiving objects: 83% (25735/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2278043Z Receiving objects: 84% (26046/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2284080Z Receiving objects: 85% (26356/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2290454Z Receiving objects: 86% (26666/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2295857Z Receiving objects: 87% (26976/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2303468Z Receiving objects: 88% (27286/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2308997Z Receiving objects: 89% (27596/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2420284Z Receiving objects: 90% (27906/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2434766Z Receiving objects: 91% (28216/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2447001Z Receiving objects: 92% (28526/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2464008Z Receiving objects: 93% (28836/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2497556Z Receiving objects: 94% (29146/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2509118Z Receiving objects: 95% (29456/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2514784Z Receiving objects: 96% (29766/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2522486Z Receiving objects: 97% (30076/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2598514Z Receiving objects: 98% (30386/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2616085Z Receiving objects: 99% (30696/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2617202Z remote: Total 31006 (delta 1017), reused 1477 (delta 826), pack-reused 29260 (from 1) 2024-08-20T22:29:01.2633563Z Receiving objects: 100% (31006/31006), 221.92 MiB | 74.02 MiB/s 2024-08-20T22:29:01.2634853Z Receiving objects: 100% (31006/31006), 232.50 MiB | 72.12 MiB/s, done. 2024-08-20T22:29:01.2665673Z Resolving deltas: 0% (0/16730) 2024-08-20T22:29:01.2679848Z Resolving deltas: 1% (168/16730) 2024-08-20T22:29:01.2690343Z Resolving deltas: 2% (393/16730) 2024-08-20T22:29:01.2713621Z Resolving deltas: 3% (504/16730) 2024-08-20T22:29:01.2729792Z Resolving deltas: 4% (673/16730) 2024-08-20T22:29:01.2739641Z Resolving deltas: 5% (902/16730) 2024-08-20T22:29:01.2749879Z Resolving deltas: 6% (1023/16730) 2024-08-20T22:29:01.2763235Z Resolving deltas: 7% (1172/16730) 2024-08-20T22:29:01.2772437Z Resolving deltas: 8% (1356/16730) 2024-08-20T22:29:01.2784645Z Resolving deltas: 9% (1518/16730) 2024-08-20T22:29:01.2794909Z Resolving deltas: 10% (1677/16730) 2024-08-20T22:29:01.2808332Z Resolving deltas: 11% (1841/16730) 2024-08-20T22:29:01.2821608Z Resolving deltas: 12% (2009/16730) 2024-08-20T22:29:01.2834575Z Resolving deltas: 13% (2198/16730) 2024-08-20T22:29:01.2848108Z Resolving deltas: 14% (2392/16730) 2024-08-20T22:29:01.2863940Z Resolving deltas: 15% (2510/16730) 2024-08-20T22:29:01.2887906Z Resolving deltas: 16% (2679/16730) 2024-08-20T22:29:01.2889764Z Resolving deltas: 17% (2994/16730) 2024-08-20T22:29:01.2900021Z Resolving deltas: 18% (3015/16730) 2024-08-20T22:29:01.2907651Z Resolving deltas: 19% (3204/16730) 2024-08-20T22:29:01.2943062Z Resolving deltas: 20% (3358/16730) 2024-08-20T22:29:01.2968738Z Resolving deltas: 21% (3525/16730) 2024-08-20T22:29:01.2974174Z Resolving deltas: 22% (3754/16730) 2024-08-20T22:29:01.2984329Z Resolving deltas: 23% (3860/16730) 2024-08-20T22:29:01.2994387Z Resolving deltas: 24% (4023/16730) 2024-08-20T22:29:01.3004749Z Resolving deltas: 25% (4198/16730) 2024-08-20T22:29:01.3018840Z Resolving deltas: 26% (4365/16730) 2024-08-20T22:29:01.3058284Z Resolving deltas: 27% (4518/16730) 2024-08-20T22:29:01.3074556Z Resolving deltas: 28% (4688/16730) 2024-08-20T22:29:01.3090265Z Resolving deltas: 29% (4853/16730) 2024-08-20T22:29:01.3106428Z Resolving deltas: 30% (5025/16730) 2024-08-20T22:29:01.3120357Z Resolving deltas: 31% (5195/16730) 2024-08-20T22:29:01.3135823Z Resolving deltas: 32% (5358/16730) 2024-08-20T22:29:01.3146335Z Resolving deltas: 33% (5545/16730) 2024-08-20T22:29:01.3157335Z Resolving deltas: 34% (5720/16730) 2024-08-20T22:29:01.3168699Z Resolving deltas: 35% (5895/16730) 2024-08-20T22:29:01.3189552Z Resolving deltas: 36% (6034/16730) 2024-08-20T22:29:01.3207915Z Resolving deltas: 37% (6197/16730) 2024-08-20T22:29:01.3218089Z Resolving deltas: 38% (6380/16730) 2024-08-20T22:29:01.3232665Z Resolving deltas: 39% (6549/16730) 2024-08-20T22:29:01.3246922Z Resolving deltas: 40% (6699/16730) 2024-08-20T22:29:01.3257331Z Resolving deltas: 41% (6870/16730) 2024-08-20T22:29:01.3272996Z Resolving deltas: 42% (7031/16730) 2024-08-20T22:29:01.3285205Z Resolving deltas: 43% (7208/16730) 2024-08-20T22:29:01.3293362Z Resolving deltas: 44% (7362/16730) 2024-08-20T22:29:01.3304436Z Resolving deltas: 45% (7540/16730) 2024-08-20T22:29:01.3320729Z Resolving deltas: 46% (7698/16730) 2024-08-20T22:29:01.3342052Z Resolving deltas: 47% (7865/16730) 2024-08-20T22:29:01.3356129Z Resolving deltas: 48% (8031/16730) 2024-08-20T22:29:01.3379277Z Resolving deltas: 49% (8216/16730) 2024-08-20T22:29:01.3390888Z Resolving deltas: 50% (8368/16730) 2024-08-20T22:29:01.3398905Z Resolving deltas: 51% (8535/16730) 2024-08-20T22:29:01.3410475Z Resolving deltas: 52% (8700/16730) 2024-08-20T22:29:01.3424560Z Resolving deltas: 53% (8874/16730) 2024-08-20T22:29:01.3435524Z Resolving deltas: 54% (9039/16730) 2024-08-20T22:29:01.3450743Z Resolving deltas: 55% (9203/16730) 2024-08-20T22:29:01.3458387Z Resolving deltas: 56% (9370/16730) 2024-08-20T22:29:01.3468129Z Resolving deltas: 57% (9566/16730) 2024-08-20T22:29:01.3472267Z Resolving deltas: 58% (9826/16730) 2024-08-20T22:29:01.3479253Z Resolving deltas: 59% (9913/16730) 2024-08-20T22:29:01.3485639Z Resolving deltas: 60% (10075/16730) 2024-08-20T22:29:01.3495186Z Resolving deltas: 61% (10231/16730) 2024-08-20T22:29:01.3500031Z Resolving deltas: 62% (10448/16730) 2024-08-20T22:29:01.3509415Z Resolving deltas: 63% (10560/16730) 2024-08-20T22:29:01.3528065Z Resolving deltas: 64% (10804/16730) 2024-08-20T22:29:01.3536136Z Resolving deltas: 66% (11082/16730) 2024-08-20T22:29:01.3554959Z Resolving deltas: 67% (11242/16730) 2024-08-20T22:29:01.3561612Z Resolving deltas: 68% (11381/16730) 2024-08-20T22:29:01.3573409Z Resolving deltas: 69% (11544/16730) 2024-08-20T22:29:01.3590302Z Resolving deltas: 70% (11711/16730) 2024-08-20T22:29:01.3602574Z Resolving deltas: 71% (11879/16730) 2024-08-20T22:29:01.3613018Z Resolving deltas: 72% (12062/16730) 2024-08-20T22:29:01.3639968Z Resolving deltas: 73% (12228/16730) 2024-08-20T22:29:01.3645831Z Resolving deltas: 75% (12620/16730) 2024-08-20T22:29:01.3659804Z Resolving deltas: 76% (12753/16730) 2024-08-20T22:29:01.3670636Z Resolving deltas: 77% (12902/16730) 2024-08-20T22:29:01.3687647Z Resolving deltas: 78% (13051/16730) 2024-08-20T22:29:01.3698618Z Resolving deltas: 79% (13221/16730) 2024-08-20T22:29:01.3711201Z Resolving deltas: 80% (13384/16730) 2024-08-20T22:29:01.3722822Z Resolving deltas: 81% (13572/16730) 2024-08-20T22:29:01.3734443Z Resolving deltas: 82% (13738/16730) 2024-08-20T22:29:01.3744788Z Resolving deltas: 83% (13886/16730) 2024-08-20T22:29:01.3753889Z Resolving deltas: 84% (14087/16730) 2024-08-20T22:29:01.3763359Z Resolving deltas: 85% (14233/16730) 2024-08-20T22:29:01.3773941Z Resolving deltas: 86% (14409/16730) 2024-08-20T22:29:01.3783174Z Resolving deltas: 87% (14567/16730) 2024-08-20T22:29:01.3795003Z Resolving deltas: 88% (14725/16730) 2024-08-20T22:29:01.3803791Z Resolving deltas: 89% (14904/16730) 2024-08-20T22:29:01.3814559Z Resolving deltas: 90% (15061/16730) 2024-08-20T22:29:01.3822629Z Resolving deltas: 91% (15226/16730) 2024-08-20T22:29:01.3828681Z Resolving deltas: 92% (15394/16730) 2024-08-20T22:29:01.3836478Z Resolving deltas: 93% (15560/16730) 2024-08-20T22:29:01.3846170Z Resolving deltas: 94% (15730/16730) 2024-08-20T22:29:01.3859356Z Resolving deltas: 95% (15920/16730) 2024-08-20T22:29:01.3861836Z Resolving deltas: 96% (16211/16730) 2024-08-20T22:29:01.3870929Z Resolving deltas: 97% (16244/16730) 2024-08-20T22:29:01.3922111Z Resolving deltas: 98% (16397/16730) 2024-08-20T22:29:01.3997270Z Resolving deltas: 99% (16568/16730) 2024-08-20T22:29:01.3998191Z Resolving deltas: 100% (16730/16730) 2024-08-20T22:29:01.3999030Z Resolving deltas: 100% (16730/16730), done. 2024-08-20T22:29:02.4013979Z + pushd torchbench 2024-08-20T22:29:02.4014377Z ~/workspace/torchbench ~/workspace 2024-08-20T22:29:02.4014882Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:29:02.4795974Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:29:02.4796522Z 2024-08-20T22:29:02.4797035Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:29:02.4797884Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:29:02.4798626Z state without impacting any branches by switching back to a branch. 2024-08-20T22:29:02.4799055Z 2024-08-20T22:29:02.4799386Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:29:02.4800181Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:29:02.4800594Z 2024-08-20T22:29:02.4800798Z git switch -c 2024-08-20T22:29:02.4801075Z 2024-08-20T22:29:02.4801212Z Or undo this operation with: 2024-08-20T22:29:02.4801462Z 2024-08-20T22:29:02.4801582Z git switch - 2024-08-20T22:29:02.4801758Z 2024-08-20T22:29:02.4802094Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:29:02.4802568Z 2024-08-20T22:29:02.4802729Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:29:02.4809922Z + '[' '' ']' 2024-08-20T22:29:02.4810308Z + python install.py --continue_on_fail 2024-08-20T22:29:04.7524149Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:29:23.0692417Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:29:26.1979519Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:29:28.1079415Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:29:29.7577806Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:29:29.7813560Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:29:37.4799913Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:29:40.2023715Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:29:42.9144098Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:29:45.6257420Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:29:45.6260646Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:29:46.6683054Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:29:49.3602420Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:29:49.3841260Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:30:46.0548131Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:30:54.4849261Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:31:01.5124862Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:31:08.2845243Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:31:16.7456140Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:31:23.8365242Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:31:30.1546074Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:31:37.0112579Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:31:44.1469685Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:31:51.3434679Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:31:58.0592195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:32:05.0288200Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:32:06.2415901Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:32:41.6598576Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:32:46.1980057Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:32:51.4205116Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:33:02.1503308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:33:03.3273770Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:33:04.7880374Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:33:11.2933428Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:33:18.3499160Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:33:23.7934711Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:33:31.4871468Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:33:37.3873179Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:33:42.4351530Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:33:48.6453308Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:34:05.0391476Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:34:11.2080787Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:34:15.1110155Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:34:19.8277939Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:34:26.5888793Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:34:26.5893663Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:34:39.6174225Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:34:43.5377804Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:34:43.5379660Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:34:51.0975525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:34:52.3203031Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:34:53.5573316Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:35:38.1829302Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:37:07.0782122Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:37:07.1077816Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:37:08.6645000Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:37:08.6885939Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:37:08.7124194Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:37:08.7365728Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:37:08.7600695Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:37:08.7842447Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:37:29.9228950Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:37:29.9231393Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:37:31.1627738Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:37:33.6495251Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:37:33.6738139Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:37:33.6974444Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:37:33.7213813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:37:33.7456846Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:37:33.7699195Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:37:40.3561343Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:37:41.7970289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:37:45.3595027Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:37:45.3835743Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:37:45.4077300Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:37:45.4315184Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:37:45.4556913Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:37:45.4795518Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:38:00.5085208Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:38:16.7259762Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:38:16.7501914Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:38:16.7505736Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:38:16.7508528Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:38:19.1296801Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:38:20.8850293Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:38:20.9094105Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:38:45.4374465Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:38:52.0027339Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:38:56.2747047Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:39:03.3798120Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:39:03.4043934Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:39:03.4292354Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:39:03.4529204Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:39:03.4770651Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:39:03.5009437Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:39:03.5251299Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:39:03.5491998Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:39:12.6398307Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:39:24.2512935Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:39:24.2765220Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:39:26.3773686Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:39:28.3544633Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:39:28.8418153Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:39:28.8418809Z Print all dependencies after TorchBench is installed 2024-08-20T22:39:28.8419513Z + python -mpip freeze 2024-08-20T22:39:29.3516674Z absl-py==2.1.0 2024-08-20T22:39:29.3517148Z accelerate==0.33.0 2024-08-20T22:39:29.3517625Z aiohappyeyeballs==2.4.0 2024-08-20T22:39:29.3518110Z aiohttp==3.10.5 2024-08-20T22:39:29.3518539Z aiosignal==1.3.1 2024-08-20T22:39:29.3519013Z annotated-types==0.7.0 2024-08-20T22:39:29.3519568Z antlr4-python3-runtime==4.9.3 2024-08-20T22:39:29.3520047Z anyascii==0.3.2 2024-08-20T22:39:29.3520431Z astroid==3.2.4 2024-08-20T22:39:29.3521338Z astunparse==1.6.3 2024-08-20T22:39:29.3521934Z async-timeout==4.0.3 2024-08-20T22:39:29.3522389Z attrs==23.1.0 2024-08-20T22:39:29.3522764Z audioread==3.0.1 2024-08-20T22:39:29.3523149Z babel==2.16.0 2024-08-20T22:39:29.3523497Z beautifulsoup4==4.12.3 2024-08-20T22:39:29.3524920Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:39:29.3526246Z bibtexparser==2.0.0b7 2024-08-20T22:39:29.3526789Z black==24.8.0 2024-08-20T22:39:29.3527168Z blinker==1.8.2 2024-08-20T22:39:29.3527530Z blis==0.7.11 2024-08-20T22:39:29.3527904Z blobfile==2.1.1 2024-08-20T22:39:29.3528277Z bokeh==3.5.1 2024-08-20T22:39:29.3528995Z boto3==1.19.12 2024-08-20T22:39:29.3529684Z botocore==1.22.12 2024-08-20T22:39:29.3530087Z bs4==0.0.2 2024-08-20T22:39:29.3530457Z cachetools==5.5.0 2024-08-20T22:39:29.3530842Z cardboardlint==1.3.1 2024-08-20T22:39:29.3531240Z catalogue==2.0.10 2024-08-20T22:39:29.3531602Z certifi==2024.7.4 2024-08-20T22:39:29.3531979Z cffi==1.17.0 2024-08-20T22:39:29.3532469Z charset-normalizer==3.3.2 2024-08-20T22:39:29.3532916Z click==8.1.7 2024-08-20T22:39:29.3533280Z clldutils==3.22.2 2024-08-20T22:39:29.3533685Z cloudpathlib==0.18.1 2024-08-20T22:39:29.3534092Z cloudpickle==3.0.0 2024-08-20T22:39:29.3534471Z colorama==0.4.6 2024-08-20T22:39:29.3534860Z colorlog==6.8.2 2024-08-20T22:39:29.3535234Z confection==0.1.5 2024-08-20T22:39:29.3535621Z contourpy==1.2.1 2024-08-20T22:39:29.3535989Z coremltools==5.0b5 2024-08-20T22:39:29.3536399Z cryptography==43.0.0 2024-08-20T22:39:29.3536822Z csvw==3.3.0 2024-08-20T22:39:29.3537172Z cycler==0.12.1 2024-08-20T22:39:29.3537540Z cymem==2.0.8 2024-08-20T22:39:29.3537897Z DALL-E==0.1 2024-08-20T22:39:29.3538268Z datasets==2.21.0 2024-08-20T22:39:29.3538648Z decorator==5.1.1 2024-08-20T22:39:29.3539031Z defusedxml==0.7.1 2024-08-20T22:39:29.3539419Z Deprecated==1.2.14 2024-08-20T22:39:29.3540332Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:39:29.3541318Z diffusers==0.20.2 2024-08-20T22:39:29.3541702Z dill==0.3.8 2024-08-20T22:39:29.3541992Z diskcache==5.6.3 2024-08-20T22:39:29.3542279Z distro==1.9.0 2024-08-20T22:39:29.3542559Z dlinfo==1.2.1 2024-08-20T22:39:29.3542908Z docker-pycreds==0.4.0 2024-08-20T22:39:29.3543242Z dominate==2.9.1 2024-08-20T22:39:29.3544068Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:39:29.3544822Z einops==0.8.0 2024-08-20T22:39:29.3545119Z exceptiongroup==1.2.2 2024-08-20T22:39:29.3545434Z execnet==2.1.1 2024-08-20T22:39:29.3546021Z expecttest==0.1.6 2024-08-20T22:39:29.3546326Z FastNLP==0.6.0 2024-08-20T22:39:29.3546675Z ffmpeg-python==0.2.0 2024-08-20T22:39:29.3546980Z filelock==3.13.1 2024-08-20T22:39:29.3547268Z Flask==3.0.3 2024-08-20T22:39:29.3547548Z flatbuffers==2.0 2024-08-20T22:39:29.3547842Z fonttools==4.53.1 2024-08-20T22:39:29.3548130Z frozenlist==1.4.1 2024-08-20T22:39:29.3548466Z fsspec==2024.6.1 2024-08-20T22:39:29.3548790Z ftfy==6.2.3 2024-08-20T22:39:29.3549377Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:39:29.3550073Z future==1.0.0 2024-08-20T22:39:29.3550370Z fvcore==0.1.5.post20221221 2024-08-20T22:39:29.3550708Z gdown==5.2.0 2024-08-20T22:39:29.3550978Z geojson==2.5.0 2024-08-20T22:39:29.3551258Z ghstack==0.8.0 2024-08-20T22:39:29.3551540Z gitdb==4.0.11 2024-08-20T22:39:29.3551813Z GitPython==3.1.43 2024-08-20T22:39:29.3552168Z google-auth==2.34.0 2024-08-20T22:39:29.3552534Z google-auth-oauthlib==1.0.0 2024-08-20T22:39:29.3552885Z grpcio==1.65.5 2024-08-20T22:39:29.3553163Z gym==0.26.2 2024-08-20T22:39:29.3553481Z gym-notices==0.0.8 2024-08-20T22:39:29.3553781Z h5py==3.11.0 2024-08-20T22:39:29.3554054Z higher==0.2.1 2024-08-20T22:39:29.3554390Z huggingface-hub==0.24.6 2024-08-20T22:39:29.3554752Z hydra-core==1.3.2 2024-08-20T22:39:29.3555050Z hypothesis==5.35.1 2024-08-20T22:39:29.3555337Z idna==3.7 2024-08-20T22:39:29.3555603Z imageio==2.35.1 2024-08-20T22:39:29.3555904Z importlib_metadata==8.4.0 2024-08-20T22:39:29.3556236Z inflect==7.3.1 2024-08-20T22:39:29.3556517Z iniconfig==2.0.0 2024-08-20T22:39:29.3556812Z iopath==0.1.9 2024-08-20T22:39:29.3557096Z isodate==0.6.1 2024-08-20T22:39:29.3557374Z isort==5.13.2 2024-08-20T22:39:29.3557656Z itsdangerous==2.2.0 2024-08-20T22:39:29.3557955Z Jinja2==3.1.4 2024-08-20T22:39:29.3558236Z jmespath==0.10.0 2024-08-20T22:39:29.3558520Z joblib==1.4.2 2024-08-20T22:39:29.3558799Z jsonpatch==1.33 2024-08-20T22:39:29.3559092Z jsonpointer==3.0.0 2024-08-20T22:39:29.3559387Z jsonschema==4.23.0 2024-08-20T22:39:29.3559780Z jsonschema-specifications==2023.12.1 2024-08-20T22:39:29.3560467Z junitparser==2.1.1 2024-08-20T22:39:29.3560934Z kaldi-io==0.9.8 2024-08-20T22:39:29.3561220Z kiwisolver==1.4.5 2024-08-20T22:39:29.3561513Z kornia==0.7.3 2024-08-20T22:39:29.3561799Z kornia_rs==0.1.5 2024-08-20T22:39:29.3562091Z lameenc==1.7.0 2024-08-20T22:39:29.3562376Z langcodes==3.4.0 2024-08-20T22:39:29.3562667Z langdetect==1.0.9 2024-08-20T22:39:29.3563007Z language-tags==1.2.0 2024-08-20T22:39:29.3563318Z language_data==1.2.0 2024-08-20T22:39:29.3563622Z lark==0.12.0 2024-08-20T22:39:29.3563902Z lazy_loader==0.4 2024-08-20T22:39:29.3564184Z libcst==1.4.0 2024-08-20T22:39:29.3564463Z librosa==0.9.2 2024-08-20T22:39:29.3564752Z lintrunner==0.12.5 2024-08-20T22:39:29.3565054Z llvmlite==0.38.1 2024-08-20T22:39:29.3565337Z lxml==4.9.4 2024-08-20T22:39:29.3565650Z marisa-trie==1.2.0 2024-08-20T22:39:29.3565948Z Markdown==3.7 2024-08-20T22:39:29.3566271Z markdown-it-py==3.0.0 2024-08-20T22:39:29.3566727Z MarkupSafe==2.1.5 2024-08-20T22:39:29.3567044Z matplotlib==3.8.4 2024-08-20T22:39:29.3567345Z mccabe==0.7.0 2024-08-20T22:39:29.3567621Z mdurl==0.1.2 2024-08-20T22:39:29.3567941Z ml-dtypes==0.4.0 2024-08-20T22:39:29.3568235Z MonkeyType==23.3.0 2024-08-20T22:39:29.3568574Z more-itertools==10.4.0 2024-08-20T22:39:29.3568893Z mpmath==1.3.0 2024-08-20T22:39:29.3569176Z msgpack==1.0.8 2024-08-20T22:39:29.3569460Z multidict==6.0.5 2024-08-20T22:39:29.3569752Z multiprocess==0.70.16 2024-08-20T22:39:29.3570071Z murmurhash==1.0.10 2024-08-20T22:39:29.3570369Z musdb==0.4.2 2024-08-20T22:39:29.3570647Z museval==0.4.1 2024-08-20T22:39:29.3570922Z mypy==1.10.0 2024-08-20T22:39:29.3571250Z mypy-extensions==1.0.0 2024-08-20T22:39:29.3571568Z networkx==2.8.8 2024-08-20T22:39:29.3571847Z ninja==1.10.2 2024-08-20T22:39:29.3572119Z nose==1.3.7 2024-08-20T22:39:29.3572391Z numba==0.55.2 2024-08-20T22:39:29.3572665Z numpy==1.21.2 2024-08-20T22:39:29.3573010Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:39:29.3573439Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:39:29.3573877Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:39:29.3574323Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:39:29.3574770Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:39:29.3575172Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:39:29.3575577Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:39:29.3575989Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:39:29.3576421Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:39:29.3576833Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:39:29.3577227Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:39:29.3577636Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:39:29.3577969Z oauthlib==3.2.2 2024-08-20T22:39:29.3578259Z omegaconf==2.3.0 2024-08-20T22:39:29.3578542Z onnx==1.16.1 2024-08-20T22:39:29.3578832Z onnxscript==0.1.0.dev20240817 2024-08-20T22:39:29.3579181Z opacus==1.5.2 2024-08-20T22:39:29.3579510Z opencv-python==4.8.0.74 2024-08-20T22:39:29.3579864Z opt-einsum==3.3.0 2024-08-20T22:39:29.3580162Z optree==0.12.1 2024-08-20T22:39:29.3580448Z packaging==24.1 2024-08-20T22:39:29.3580732Z pandas==2.0.3 2024-08-20T22:39:29.3581015Z patch==1.16 2024-08-20T22:39:29.3581294Z pathspec==0.12.1 2024-08-20T22:39:29.3581590Z phonemizer==3.3.0 2024-08-20T22:39:29.3581880Z pillow==10.3.0 2024-08-20T22:39:29.3582171Z platformdirs==4.2.2 2024-08-20T22:39:29.3582471Z pluggy==1.5.0 2024-08-20T22:39:29.3582743Z pooch==1.8.2 2024-08-20T22:39:29.3583026Z portalocker==2.10.1 2024-08-20T22:39:29.3583336Z preshed==3.0.9 2024-08-20T22:39:29.3583666Z prettytable==3.11.0 2024-08-20T22:39:29.3583963Z protobuf==3.20.2 2024-08-20T22:39:29.3584254Z psutil==6.0.0 2024-08-20T22:39:29.3584659Z py-cpuinfo==9.0.0 2024-08-20T22:39:29.3584956Z pyaml==24.7.0 2024-08-20T22:39:29.3585225Z pyarrow==17.0.0 2024-08-20T22:39:29.3585509Z pyasn1==0.6.0 2024-08-20T22:39:29.3585789Z pyasn1_modules==0.4.0 2024-08-20T22:39:29.3586114Z pyclipper==1.3.0.post5 2024-08-20T22:39:29.3586431Z pycocotools==2.0.8 2024-08-20T22:39:29.3586733Z pycparser==2.22 2024-08-20T22:39:29.3587033Z pycryptodomex==3.20.0 2024-08-20T22:39:29.3587344Z pydantic==2.8.2 2024-08-20T22:39:29.3587813Z pydantic_core==2.20.1 2024-08-20T22:39:29.3588213Z pydot==3.0.1 2024-08-20T22:39:29.3588492Z pygame==2.6.0 2024-08-20T22:39:29.3588766Z PyGithub==2.3.0 2024-08-20T22:39:29.3589052Z Pygments==2.15.0 2024-08-20T22:39:29.3589341Z PyJWT==2.9.0 2024-08-20T22:39:29.3589615Z pylatexenc==2.10 2024-08-20T22:39:29.3589918Z pylint==3.2.6 2024-08-20T22:39:29.3590194Z PyNaCl==1.5.0 2024-08-20T22:39:29.3590475Z pynvml==11.5.3 2024-08-20T22:39:29.3590755Z pyparsing==3.1.2 2024-08-20T22:39:29.3591050Z pypdfium2==4.30.0 2024-08-20T22:39:29.3591341Z pysbd==0.3.4 2024-08-20T22:39:29.3591612Z PySocks==1.7.1 2024-08-20T22:39:29.3591897Z pytest==7.3.2 2024-08-20T22:39:29.3592236Z pytest-benchmark==4.0.0 2024-08-20T22:39:29.3592601Z pytest-cpp==2.3.0 2024-08-20T22:39:29.3592952Z pytest-flakefinder==1.1.0 2024-08-20T22:39:29.3593351Z pytest-rerunfailures==14.0 2024-08-20T22:39:29.3593781Z pytest-xdist==3.3.1 2024-08-20T22:39:29.3594148Z python-dateutil==2.9.0.post0 2024-08-20T22:39:29.3594925Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:39:29.3596320Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:39:29.3597259Z pytz==2024.1 2024-08-20T22:39:29.3597541Z PyWavelets==1.4.1 2024-08-20T22:39:29.3597902Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:39:29.3598343Z rapidfuzz==3.9.6 2024-08-20T22:39:29.3598633Z rdflib==7.0.0 2024-08-20T22:39:29.3598906Z redis==5.0.8 2024-08-20T22:39:29.3599189Z referencing==0.35.1 2024-08-20T22:39:29.3599494Z regex==2024.7.24 2024-08-20T22:39:29.3599783Z requests==2.32.3 2024-08-20T22:39:29.3600126Z requests-oauthlib==2.0.0 2024-08-20T22:39:29.3600465Z resampy==0.4.3 2024-08-20T22:39:29.3600745Z rfc3986==1.5.0 2024-08-20T22:39:29.3601017Z rich==13.7.1 2024-08-20T22:39:29.3601296Z rockset==1.0.3 2024-08-20T22:39:29.3601622Z rpds-py==0.20.0 2024-08-20T22:39:29.3601904Z rsa==4.9 2024-08-20T22:39:29.3602162Z s3transfer==0.5.2 2024-08-20T22:39:29.3602471Z safetensors==0.4.4 2024-08-20T22:39:29.3602816Z scikit-image==0.21.0 2024-08-20T22:39:29.3603165Z scikit-learn==1.5.1 2024-08-20T22:39:29.3603509Z scipy==1.10.1 2024-08-20T22:39:29.3603793Z segments==2.2.1 2024-08-20T22:39:29.3604086Z sentencepiece==0.2.0 2024-08-20T22:39:29.3604429Z sentry-sdk==2.13.0 2024-08-20T22:39:29.3604734Z setproctitle==1.3.3 2024-08-20T22:39:29.3605034Z shapely==2.0.6 2024-08-20T22:39:29.3605311Z shellingham==1.5.4 2024-08-20T22:39:29.3605611Z simplejson==3.19.3 2024-08-20T22:39:29.3606000Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:39:29.3606495Z smart-open==7.0.4 2024-08-20T22:39:29.3606965Z smmap==5.0.1 2024-08-20T22:39:29.3607258Z sortedcontainers==2.4.0 2024-08-20T22:39:29.3607586Z soundfile==0.12.1 2024-08-20T22:39:29.3607882Z soupsieve==2.6 2024-08-20T22:39:29.3608157Z soxr==0.4.0 2024-08-20T22:39:29.3608424Z spacy==3.7.6 2024-08-20T22:39:29.3608745Z spacy-legacy==3.0.12 2024-08-20T22:39:29.3609089Z spacy-loggers==1.0.5 2024-08-20T22:39:29.3609404Z srsly==2.4.8 2024-08-20T22:39:29.3609685Z stempeg==0.2.3 2024-08-20T22:39:29.3609966Z submitit==1.5.1 2024-08-20T22:39:29.3610245Z sympy==1.13.1 2024-08-20T22:39:29.3610523Z tabulate==0.9.0 2024-08-20T22:39:29.3610859Z tb-nightly==2.13.0a20230426 2024-08-20T22:39:29.3611195Z tensorboard==2.13.0 2024-08-20T22:39:29.3611565Z tensorboard-data-server==0.7.2 2024-08-20T22:39:29.3611932Z tensorboardX==2.6.2.2 2024-08-20T22:39:29.3612249Z termcolor==2.4.0 2024-08-20T22:39:29.3612534Z thinc==8.2.5 2024-08-20T22:39:29.3612820Z threadpoolctl==3.5.0 2024-08-20T22:39:29.3613133Z tifffile==2024.8.10 2024-08-20T22:39:29.3613911Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:39:29.3614665Z tlparse==0.3.25 2024-08-20T22:39:29.3614954Z tokenizers==0.15.2 2024-08-20T22:39:29.3615246Z tomli==2.0.1 2024-08-20T22:39:29.3615513Z tomlkit==0.13.2 2024-08-20T22:39:29.3616893Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b444e33ace8c516878aca2862d707788a4c01d8c6190fb13e010fb9a131a57c5 2024-08-20T22:39:29.3618581Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:39:29.3619625Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:39:29.3620577Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:39:29.3621547Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:39:29.3622633Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:39:29.3623706Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:39:29.3624685Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:39:29.3625366Z tornado==6.4.1 2024-08-20T22:39:29.3625662Z tqdm==4.66.5 2024-08-20T22:39:29.3625946Z transformers==4.38.1 2024-08-20T22:39:29.3626263Z treetable==0.2.5 2024-08-20T22:39:29.3627056Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:39:29.3627825Z typeguard==4.3.0 2024-08-20T22:39:29.3628122Z typer==0.12.4 2024-08-20T22:39:29.3628410Z typing_extensions==4.12.2 2024-08-20T22:39:29.3628747Z tzdata==2024.1 2024-08-20T22:39:29.3629037Z Unidecode==1.3.8 2024-08-20T22:39:29.3629408Z unittest-xml-reporting==3.2.0 2024-08-20T22:39:29.3629770Z uritemplate==4.1.1 2024-08-20T22:39:29.3630071Z urllib3==1.26.19 2024-08-20T22:39:29.3630364Z visdom==0.2.4 2024-08-20T22:39:29.3630635Z wandb==0.17.7 2024-08-20T22:39:29.3630915Z wasabi==1.1.3 2024-08-20T22:39:29.3631193Z wcwidth==0.2.13 2024-08-20T22:39:29.3631477Z weasel==0.4.1 2024-08-20T22:39:29.3631806Z websocket-client==1.8.0 2024-08-20T22:39:29.3632133Z Werkzeug==3.0.3 2024-08-20T22:39:29.3632425Z wrapt==1.16.0 2024-08-20T22:39:29.3632710Z xdoctest==1.1.0 2024-08-20T22:39:29.3632992Z xxhash==3.5.0 2024-08-20T22:39:29.3633277Z xyzservices==2024.6.0 2024-08-20T22:39:29.3633589Z yacs==0.1.8 2024-08-20T22:39:29.3633849Z yarl==1.9.4 2024-08-20T22:39:29.3634165Z z3-solver==4.12.2.0 2024-08-20T22:39:29.3634470Z zipp==3.20.0 2024-08-20T22:39:29.3832501Z + popd 2024-08-20T22:39:29.3832833Z ~/workspace 2024-08-20T22:39:29.3833188Z + [[ dynamic_inductor_torchbench != *cpu* ]] 2024-08-20T22:39:29.3833638Z + install_torchrec_and_fbgemm 2024-08-20T22:39:29.3834005Z + local torchrec_commit 2024-08-20T22:39:29.3838269Z ++ get_pinned_commit torchrec 2024-08-20T22:39:29.3838686Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:39:29.3858287Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:39:29.3858802Z + local fbgemm_commit 2024-08-20T22:39:29.3862328Z ++ get_pinned_commit fbgemm 2024-08-20T22:39:29.3862732Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:39:29.3899089Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:29.3899853Z + pip_uninstall torchrec-nightly 2024-08-20T22:39:29.3900314Z + pip uninstall -y torchrec-nightly 2024-08-20T22:39:29.7654080Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:39:29.8008255Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:39:29.8008936Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:39:30.1661142Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:39:30.2004244Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:30.2005353Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:39:30.7166843Z Collecting setuptools-git-versioning 2024-08-20T22:39:30.7330641Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:39:30.8509239Z Collecting scikit-build 2024-08-20T22:39:30.8538319Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:39:30.9302620Z Collecting pyre-extensions 2024-08-20T22:39:30.9330860Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:39:30.9812757Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:39:30.9815168Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:39:31.0377325Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:39:31.0408497Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:39:31.0911026Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:39:31.0914753Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:39:31.0917974Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:39:31.1446746Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:39:31.1476330Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:39:31.1979699Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:39:31.2057862Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:39:31.2101870Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:39:31.2622647Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:39:31.3204641Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:39:31.3703637Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:39:31.4157145Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:39:32.3063667Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:39:32.6769072Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:39:32.7883260Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:39:32.7884726Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:32.7886661Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:39:33.1599213Z Collecting fbgemm-gpu 2024-08-20T22:39:33.1602648Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-ihsvtz8e/fbgemm-gpu_c823c0ea2e3c4557a4f52691c4379df8 2024-08-20T22:39:33.1632442Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-ihsvtz8e/fbgemm-gpu_c823c0ea2e3c4557a4f52691c4379df8 2024-08-20T22:39:34.2387945Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:39:34.2416946Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:34.7469417Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:35.2849402Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:39:35.2850703Z Running command git submodule update --init --recursive -q 2024-08-20T22:39:45.0185778Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:39:45.0204597Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:39:45.0211710Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:24:16.2932896Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T23:24:16.4761641Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252619785 sha256=5f957bb389a724aa6d731af1cce90a9e9a8e68ecb1cb134046e2ef327970bb91 2024-08-20T23:24:16.4763292Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:24:16.4783202Z Successfully built fbgemm-gpu 2024-08-20T23:24:17.3279234Z Installing collected packages: fbgemm-gpu 2024-08-20T23:24:20.7756606Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:24:21.6807883Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:21.6810695Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:22.0493487Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:22.0497336Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-c8dzr1xd 2024-08-20T23:24:22.0526709Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-c8dzr1xd 2024-08-20T23:24:22.8901197Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:24:22.8928294Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:23.2053409Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:23.5622815Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:24:23.8330642Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:24:23.8366347Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:24:23.8369916Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:24:23.8372571Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:24:23.8890113Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:24:23.9084297Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:24:23.9492736Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:24:23.9494565Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:24:23.9504264Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:24:23.9513573Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:24:23.9782473Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:24:23.9785987Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:24:23.9789246Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:24:23.9811664Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:24:23.9815351Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:24:23.9975699Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:24:24.0010751Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:24:24.0108340Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:24:24.0111209Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:24:24.0130351Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:24:24.0163617Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:24:24.0167193Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:24:24.0171397Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:24:24.0173604Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:24:24.0179085Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:24:24.0193736Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:24:24.0609682Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:24:24.0791786Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:24:24.1008156Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:24:24.1111810Z Building wheels for collected packages: torchrec 2024-08-20T23:24:24.6265138Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:24:24.6272954Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=55f13712285198a5726e1465d08ccebfbca8c2047b2cb5e388c287a21e684c52 2024-08-20T23:24:24.6274427Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:24:24.6295372Z Successfully built torchrec 2024-08-20T23:24:25.4181842Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:24:26.0304675Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:24:26.1684414Z ++ pwd 2024-08-20T23:24:26.1686583Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:24:26.1687344Z + test_dynamo_benchmark torchbench 0 2024-08-20T23:24:26.1692467Z ++ pwd 2024-08-20T23:24:26.1694907Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:24:26.1695610Z + local suite=torchbench 2024-08-20T23:24:26.1695958Z + shift 2024-08-20T23:24:26.1696316Z + local shard_id=0 2024-08-20T23:24:26.1696734Z + shift 2024-08-20T23:24:26.1697127Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:24:26.1698952Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-08-20T23:24:26.1699459Z + [[ dynamic_inductor_torchbench == *cpu* ]] 2024-08-20T23:24:26.1699977Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:24:26.1700746Z + test_single_dynamo_benchmark inference torchbench 0 --inference --bfloat16 2024-08-20T23:24:26.1707039Z ++ pwd 2024-08-20T23:24:26.1707753Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:24:26.1708533Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:24:26.1770746Z + local name=inference 2024-08-20T23:24:26.1771185Z + shift 2024-08-20T23:24:26.1771557Z + local suite=torchbench 2024-08-20T23:24:26.1771899Z + shift 2024-08-20T23:24:26.1772171Z + local shard_id=0 2024-08-20T23:24:26.1772472Z + shift 2024-08-20T23:24:26.1774954Z + partition_flags=() 2024-08-20T23:24:26.1775319Z + local partition_flags 2024-08-20T23:24:26.1775881Z + [[ -n 2 ]] 2024-08-20T23:24:26.1776297Z + [[ -n 0 ]] 2024-08-20T23:24:26.1776977Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:24:26.1777682Z + [[ dynamic_inductor_torchbench == *perf_compare* ]] 2024-08-20T23:24:26.1778205Z + [[ dynamic_inductor_torchbench == *perf* ]] 2024-08-20T23:24:26.1778713Z + [[ dynamic_inductor_torchbench == *aot_inductor* ]] 2024-08-20T23:24:26.1779219Z + [[ dynamic_inductor_torchbench == *_avx2* ]] 2024-08-20T23:24:26.1779700Z + [[ dynamic_inductor_torchbench == *_avx512* ]] 2024-08-20T23:24:26.1781558Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:24:32.1449718Z 2024-08-20T23:24:32.8985528Z loading model: 0it [00:00, ?it/s]/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/optim/apply_optimizer_in_backward.py:47: DeprecationWarning: `TorchScript` support for functional optimizers is deprecated and will be removed in a future PyTorch release. Consider using the `torch.compile` optimizer instead. 2024-08-20T23:24:32.8987613Z from torch.distributed.optim import _apply_optimizer_in_backward 2024-08-20T23:24:33.3075253Z 2024-08-20T23:24:33.3075973Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:24:33.3076447Z cuda eval torchrec_dlrm 2024-08-20T23:24:33.3104809Z Traceback (most recent call last): 2024-08-20T23:24:33.3105664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:24:33.3106671Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:24:33.3108038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 434, in forward_pass 2024-08-20T23:24:33.3109209Z return mod(*inputs) 2024-08-20T23:24:33.3110282Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:24:33.3111168Z return self._call_impl(*args, **kwargs) 2024-08-20T23:24:33.3112104Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:24:33.3112928Z return forward_call(*args, **kwargs) 2024-08-20T23:24:33.3113796Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 893, in forward 2024-08-20T23:24:33.3114713Z logits = self.model(batch.dense_features, batch.sparse_features) 2024-08-20T23:24:33.3115810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:24:33.3116694Z return self._call_impl(*args, **kwargs) 2024-08-20T23:24:33.3117622Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:24:33.3118457Z return forward_call(*args, **kwargs) 2024-08-20T23:24:33.3119331Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 572, in forward 2024-08-20T23:24:33.3120123Z concatenated_dense = self.inter_arch( 2024-08-20T23:24:33.3121095Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:24:33.3121976Z return self._call_impl(*args, **kwargs) 2024-08-20T23:24:33.3122901Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:24:33.3123722Z return forward_call(*args, **kwargs) 2024-08-20T23:24:33.3124596Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/models/dlrm.py", line 284, in forward 2024-08-20T23:24:33.3125523Z return self.crossnet(combined_values.reshape([B, -1])) 2024-08-20T23:24:33.3126573Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:24:33.3127568Z return self._call_impl(*args, **kwargs) 2024-08-20T23:24:33.3128498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:24:33.3129325Z return forward_call(*args, **kwargs) 2024-08-20T23:24:33.3130227Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchrec/modules/crossnet.py", line 175, in forward 2024-08-20T23:24:33.3131152Z x_l_v = torch.nn.functional.linear(x_l, self.V_kernels[layer]) 2024-08-20T23:24:33.3131980Z RuntimeError: expected mat1 and mat2 to have the same dtype, but got: float != c10::BFloat16 2024-08-20T23:24:33.3132531Z 2024-08-20T23:24:33.3132833Z The above exception was the direct cause of the following exception: 2024-08-20T23:24:33.3133283Z 2024-08-20T23:24:33.3133429Z Traceback (most recent call last): 2024-08-20T23:24:33.3134079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:24:33.3134742Z ) = runner.load_model( 2024-08-20T23:24:33.3135392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:24:33.3136150Z self.validate_model(model, example_inputs) 2024-08-20T23:24:33.3136900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:24:33.3137667Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:24:33.3138132Z RuntimeError: Eager run failed 2024-08-20T23:24:33.3138395Z 2024-08-20T23:24:33.3138515Z eager_fail_to_run 2024-08-20T23:24:36.9589134Z 2024-08-20T23:24:39.1783837Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:24:39.1784441Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:24:39.1841828Z cuda eval BERT_pytorch 2024-08-20T23:25:41.2390031Z pass 2024-08-20T23:25:41.2423351Z TIMING: code_gen:4.85013 inductor_compile:16.91872 backend_compile:47.06572 entire_frame_compile:59.40744 total_wall_time:59.40744 2024-08-20T23:25:41.2425106Z STATS: call_* op count: 1076 | FakeTensor.__torch_dispatch__:2319 | FakeTensorMode.__torch_dispatch__:52289 | attempt fast:2879 | fast is_contiguous:2879 | ProxyTorchDispatchMode.__torch_dispatch__:12905 2024-08-20T23:25:41.2426418Z Dynamo produced 1 graphs covering 1076 ops with 0 graph breaks (0 unique) 2024-08-20T23:25:48.1297240Z 2024-08-20T23:25:51.0861414Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:51.0861895Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:25:51.0909054Z cuda eval Background_Matting 2024-08-20T23:25:51.0912273Z pass_due_to_skip 2024-08-20T23:25:51.1440800Z TIMING: 2024-08-20T23:25:51.1441116Z STATS: call_* op count: 0 2024-08-20T23:25:51.1441822Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:25:54.7229951Z 2024-08-20T23:25:57.6218130Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:25:57.6218614Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:25:57.6242338Z cuda eval LearningToPaint 2024-08-20T23:26:06.7462311Z pass 2024-08-20T23:26:06.7472924Z TIMING: code_gen:1.88506 inductor_compile:3.14965 backend_compile:5.69745 entire_frame_compile:7.60673 total_wall_time:7.60673 2024-08-20T23:26:06.7474713Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:8401 | ProxyTorchDispatchMode.__torch_dispatch__:2930 | FakeTensor.__torch_dispatch__:1094 2024-08-20T23:26:06.7476123Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:26:10.9201454Z 2024-08-20T23:26:12.3339454Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth 2024-08-20T23:26:12.3514413Z 2024-08-20T23:26:12.3514626Z 2024-08-20T23:26:12.4517296Z 0% 0.00/528M [00:00 will be ignored 2024-08-21T00:11:28.1230185Z pass 2024-08-21T00:11:28.1489838Z TIMING: entire_frame_compile:197.26109 code_gen:38.48698 inductor_compile:145.56495 backend_compile:176.05573 total_wall_time:219.17094 2024-08-21T00:11:28.1493504Z STATS: call_* op count: 671 | FakeTensorMode.__torch_dispatch__:56931 | attempt fast:2462 | fast is_contiguous:2236 | slow no contiguity match:226 | ProxyTorchDispatchMode.__torch_dispatch__:17204 | FakeTensor.__torch_dispatch__:10237 2024-08-21T00:11:28.1495001Z Dynamo produced 3 graphs covering 671 ops with 7 graph breaks (5 unique) 2024-08-21T00:11:36.9612942Z 2024-08-21T00:11:38.5024187Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:11:38.5024673Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:11:38.5025108Z cuda train alexnet 2024-08-21T00:11:59.1983660Z pass 2024-08-21T00:11:59.2190627Z TIMING: entire_frame_compile:14.33512 code_gen:9.34941 inductor_compile:11.41335 backend_compile:13.49574 total_wall_time:22.52865 2024-08-21T00:11:59.2193751Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:3969 | attempt fast:118 | fast is_contiguous:118 | ProxyTorchDispatchMode.__torch_dispatch__:950 | FakeTensor.__torch_dispatch__:332 2024-08-21T00:11:59.2196315Z Dynamo produced 2 graphs covering 26 ops with 6 graph breaks (5 unique) 2024-08-21T00:12:03.3577699Z 2024-08-21T00:12:05.3127646Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:05.3128522Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:05.3129199Z cuda train basic_gnn_edgecnn 2024-08-21T00:12:08.7685623Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:10.7866965Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:12:10.7867826Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 190, in propagate 2024-08-21T00:12:10.7868452Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-08-21T00:12:10.7869068Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 128, in _collect 2024-08-21T00:12:10.7869608Z x_j = self._lift(x_j, edge_def, j) 2024-08-21T00:12:10.7870147Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 88, in _lift 2024-08-21T00:12:10.7870704Z return src.index_select(self.node_dim, index) 2024-08-21T00:12:10.7871038Z 2024-08-21T00:12:10.8020280Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:12:10.8021451Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 190, in propagate 2024-08-21T00:12:10.8022672Z kwargs = self._collect(edge_index, the_size, in_kwargs) 2024-08-21T00:12:10.8023873Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 128, in _collect 2024-08-21T00:12:10.8024923Z x_j = self._lift(x_j, edge_def, j) 2024-08-21T00:12:10.8025939Z File "/tmp/jenkins_pyg/tmpdrslkjki.py", line 88, in _lift 2024-08-21T00:12:10.8027014Z return src.index_select(self.node_dim, index) 2024-08-21T00:12:10.8027652Z 2024-08-21T00:12:10.9278935Z W0821 00:12:10.926000 46417 torch/_logging/_internal.py:1058] [24/0] Profiler function will be ignored 2024-08-21T00:12:16.0555738Z pass 2024-08-21T00:12:16.0565901Z TIMING: entire_frame_compile:7.81208 inductor_compile:5.44953 backend_compile:5.9469 code_gen:4.06449 total_wall_time:8.5971 2024-08-21T00:12:16.0567323Z STATS: call_* op count: 102 | FakeTensorMode.__torch_dispatch__:4593 | ProxyTorchDispatchMode.__torch_dispatch__:1787 | FakeTensor.__torch_dispatch__:987 2024-08-21T00:12:16.0568422Z Dynamo produced 8 graphs covering 102 ops with 20 graph breaks (6 unique) 2024-08-21T00:12:20.0904771Z 2024-08-21T00:12:22.0489584Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:22.0490070Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:22.0490506Z cuda train basic_gnn_gcn 2024-08-21T00:12:24.8281180Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:25.4131644Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:25.8808705Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:26.1125249Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:26.6215364Z W0821 00:12:26.620000 46633 torch/_logging/_internal.py:1058] [18/0] Profiler function will be ignored 2024-08-21T00:12:30.0681006Z pass 2024-08-21T00:12:30.0686632Z TIMING: entire_frame_compile:5.57473 inductor_compile:4.05387 backend_compile:4.57702 code_gen:2.97078 total_wall_time:5.96899 2024-08-21T00:12:30.0688044Z STATS: call_* op count: 98 | FakeTensorMode.__torch_dispatch__:2799 | FakeTensor.__torch_dispatch__:538 | ProxyTorchDispatchMode.__torch_dispatch__:1004 2024-08-21T00:12:30.0689135Z Dynamo produced 10 graphs covering 98 ops with 13 graph breaks (6 unique) 2024-08-21T00:12:34.0635560Z 2024-08-21T00:12:35.9773500Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:35.9773981Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:35.9774414Z cuda train basic_gnn_gin 2024-08-21T00:12:39.0617031Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:39.9998767Z W0821 00:12:39.999000 46853 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:12:44.1152159Z pass 2024-08-21T00:12:44.1178606Z TIMING: entire_frame_compile:6.36784 code_gen:3.13383 inductor_compile:4.71911 backend_compile:5.16129 total_wall_time:7.10141 2024-08-21T00:12:44.1179967Z STATS: call_* op count: 109 | FakeTensorMode.__torch_dispatch__:4058 | ProxyTorchDispatchMode.__torch_dispatch__:1747 | FakeTensor.__torch_dispatch__:954 2024-08-21T00:12:44.1181079Z Dynamo produced 3 graphs covering 109 ops with 7 graph breaks (5 unique) 2024-08-21T00:12:48.1377960Z 2024-08-21T00:12:50.0433423Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:12:50.0434306Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:12:50.0435153Z cuda train basic_gnn_sage 2024-08-21T00:12:52.8033188Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:12:53.6179516Z W0821 00:12:53.617000 47037 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:12:57.1384353Z pass 2024-08-21T00:12:57.1410936Z TIMING: entire_frame_compile:5.45295 code_gen:2.67929 inductor_compile:4.08311 backend_compile:4.45018 total_wall_time:6.0977 2024-08-21T00:12:57.1412286Z STATS: call_* op count: 106 | FakeTensorMode.__torch_dispatch__:3437 | ProxyTorchDispatchMode.__torch_dispatch__:1396 | FakeTensor.__torch_dispatch__:731 2024-08-21T00:12:57.1413388Z Dynamo produced 3 graphs covering 106 ops with 7 graph breaks (5 unique) 2024-08-21T00:13:01.0960567Z 2024-08-21T00:13:01.6866694Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:01.6867162Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:01.6867590Z cuda train dcgan 2024-08-21T00:13:09.1021280Z pass 2024-08-21T00:13:09.1039763Z TIMING: entire_frame_compile:5.84049 code_gen:1.43249 inductor_compile:3.13628 backend_compile:4.77975 total_wall_time:7.29324 2024-08-21T00:13:09.1041591Z STATS: call_* op count: 17 | FakeTensorMode.__torch_dispatch__:2713 | attempt fast:121 | fast is_contiguous:121 | FakeTensor.__torch_dispatch__:455 | ProxyTorchDispatchMode.__torch_dispatch__:830 2024-08-21T00:13:09.1042988Z Dynamo produced 2 graphs covering 17 ops with 6 graph breaks (5 unique) 2024-08-21T00:13:13.0581918Z 2024-08-21T00:13:16.7310748Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:13:16.7311230Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:13:16.7311664Z cuda train demucs 2024-08-21T00:14:15.0233764Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Error while creating guard: 2024-08-21T00:14:15.0236578Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Name: '' 2024-08-21T00:14:15.0237900Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Source: shape_env 2024-08-21T00:14:15.0238975Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Create Function: SHAPE_ENV 2024-08-21T00:14:15.0239992Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Guard Types: None 2024-08-21T00:14:15.0246009Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Code List: None 2024-08-21T00:14:15.0247379Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Object Weakref: None 2024-08-21T00:14:15.0248469Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Guarded Class Weakref: None 2024-08-21T00:14:15.0249613Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] Traceback (most recent call last): 2024-08-21T00:14:15.0251491Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create 2024-08-21T00:14:15.0253073Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] return self.create_fn(builder, self) 2024-08-21T00:14:15.0254619Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV 2024-08-21T00:14:15.0255949Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] guards = output_graph.shape_env.produce_guards( 2024-08-21T00:14:15.0257707Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards 2024-08-21T00:14:15.0259304Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] raise ConstraintViolationError( 2024-08-21T00:14:15.0260993Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-08-21T00:14:15.0263093Z E0821 00:14:15.021000 47443 torch/_guards.py:282] [4/0_1] - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). 2024-08-21T00:14:15.0264341Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] Created at: 2024-08-21T00:14:15.0265675Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 603, in transform 2024-08-21T00:14:15.0266960Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] tracer = InstructionTranslator( 2024-08-21T00:14:15.0268409Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2654, in __init__ 2024-08-21T00:14:15.0269636Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] output=OutputGraph( 2024-08-21T00:14:15.0270993Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 317, in __init__ 2024-08-21T00:14:15.0272225Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] self.init_ambient_guards() 2024-08-21T00:14:15.0273663Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 463, in init_ambient_guards 2024-08-21T00:14:15.0275201Z E0821 00:14:15.022000 47443 torch/_guards.py:284] [4/0_1] self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV)) 2024-08-21T00:14:15.0276036Z ERROR:common: 2024-08-21T00:14:15.0276359Z Traceback (most recent call last): 2024-08-21T00:14:15.0277057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2819, in check_accuracy 2024-08-21T00:14:15.0277890Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-08-21T00:14:15.0278883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-21T00:14:15.0279648Z return fn(*args, **kwargs) 2024-08-21T00:14:15.0280333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-21T00:14:15.0281123Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-21T00:14:15.0282158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 437, in forward_and_backward_pass 2024-08-21T00:14:15.0283051Z cloned_inputs = clone_inputs(inputs) 2024-08-21T00:14:15.0284013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 438, in torch_dynamo_resume_in_forward_and_backward_pass_at_437 2024-08-21T00:14:15.0284952Z self.optimizer_zero_grad(mod) 2024-08-21T00:14:15.0285871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:14:15.0286932Z pred = mod(*cloned_inputs) 2024-08-21T00:14:15.0287862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:14:15.0288741Z return self._call_impl(*args, **kwargs) 2024-08-21T00:14:15.0289657Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:14:15.0290488Z return forward_call(*args, **kwargs) 2024-08-21T00:14:15.0291407Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-21T00:14:15.0292235Z return self._torchdynamo_orig_callable( 2024-08-21T00:14:15.0293144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1039, in __call__ 2024-08-21T00:14:15.0293948Z result = self._inner_convert( 2024-08-21T00:14:15.0294810Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-21T00:14:15.0295590Z return _compile( 2024-08-21T00:14:15.0296396Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-21T00:14:15.0297311Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-21T00:14:15.0298363Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-21T00:14:15.0299285Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-21T00:14:15.0300278Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-21T00:14:15.0301079Z return function(*args, **kwargs) 2024-08-21T00:14:15.0301997Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 783, in _compile_inner 2024-08-21T00:14:15.0302837Z check_fn = CheckFunctionManager( 2024-08-21T00:14:15.0303698Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2200, in __init__ 2024-08-21T00:14:15.0304445Z guard.create(builder) 2024-08-21T00:14:15.0305194Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create 2024-08-21T00:14:15.0305934Z return self.create_fn(builder, self) 2024-08-21T00:14:15.0306818Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV 2024-08-21T00:14:15.0307647Z guards = output_graph.shape_env.produce_guards( 2024-08-21T00:14:15.0308721Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards 2024-08-21T00:14:15.0309638Z raise ConstraintViolationError( 2024-08-21T00:14:15.0310776Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['streams'].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-08-21T00:14:15.0312370Z - Not all values of RelaxedUnspecConstraint(L['streams'].size()[0]) are valid because L['streams'].size()[0] was inferred to be a constant (4). 2024-08-21T00:14:15.0313134Z 2024-08-21T00:14:15.0313140Z 2024-08-21T00:14:15.0313422Z You can suppress this exception and fall back to eager by setting: 2024-08-21T00:14:15.0313978Z import torch._dynamo 2024-08-21T00:14:15.0314483Z torch._dynamo.config.suppress_errors = True 2024-08-21T00:14:15.0314886Z 2024-08-21T00:14:15.0315183Z TorchDynamo optimized model failed to run because of following error 2024-08-21T00:14:15.0315724Z fail_to_run 2024-08-21T00:14:15.0565799Z TIMING: entire_frame_compile:0.85406 inductor_compile:1.44438 backend_compile:2.0668 code_gen:0.54086 total_wall_time:2.29794 2024-08-21T00:14:15.0567709Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:1251 | attempt fast:30 | fast is_contiguous:12 | slow no contiguity match:12 | slow both tensors nontrivially broadcast:6 | ProxyTorchDispatchMode.__torch_dispatch__:140 | FakeTensor.__torch_dispatch__:36 2024-08-21T00:14:15.0569332Z Dynamo produced 2 graphs covering 26 ops with 4 graph breaks (3 unique) 2024-08-21T00:14:18.9685567Z 2024-08-21T00:14:21.2550661Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:14:21.2551144Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:14:21.2551571Z cuda train densenet121 2024-08-21T00:17:44.1444623Z pass 2024-08-21T00:17:44.1522257Z TIMING: entire_frame_compile:186.18656 code_gen:60.74356 inductor_compile:104.6645 backend_compile:161.6898 total_wall_time:259.33706 2024-08-21T00:17:44.1524012Z STATS: call_* op count: 435 | FakeTensorMode.__torch_dispatch__:95948 | attempt fast:4258 | fast is_contiguous:3196 | ProxyTorchDispatchMode.__torch_dispatch__:30351 | slow no contiguity match:1062 | FakeTensor.__torch_dispatch__:14947 2024-08-21T00:17:44.1525481Z Dynamo produced 2 graphs covering 435 ops with 6 graph breaks (5 unique) 2024-08-21T00:18:00.5949453Z 2024-08-21T00:18:05.8843342Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:05.8843993Z loading model: 0it [00:05, ?it/s] 2024-08-21T00:18:05.8844615Z cuda train detectron2_maskrcnn_r_50_c4 2024-08-21T00:18:05.8858884Z Traceback (most recent call last): 2024-08-21T00:18:05.8859656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-21T00:18:05.8860405Z self.model_iter_fn(model, example_inputs) 2024-08-21T00:18:05.8861267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-21T00:18:05.8862088Z pred = mod(*cloned_inputs) 2024-08-21T00:18:05.8863219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:18:05.8864103Z return self._call_impl(*args, **kwargs) 2024-08-21T00:18:05.8865021Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:18:05.8865838Z return forward_call(*args, **kwargs) 2024-08-21T00:18:05.8866793Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward 2024-08-21T00:18:05.8867858Z return self.inference(batched_inputs) 2024-08-21T00:18:05.8869108Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 213, in inference 2024-08-21T00:18:05.8870080Z results, _ = self.roi_heads(images, features, proposals, None) 2024-08-21T00:18:05.8871160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:18:05.8872082Z return self._call_impl(*args, **kwargs) 2024-08-21T00:18:05.8873249Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:18:05.8874061Z return forward_call(*args, **kwargs) 2024-08-21T00:18:05.8875044Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 477, in forward 2024-08-21T00:18:05.8875985Z box_features = self._shared_roi_transform( 2024-08-21T00:18:05.8877059Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/roi_heads/roi_heads.py", line 456, in _shared_roi_transform 2024-08-21T00:18:05.8877995Z x = self.pooler(features, boxes) 2024-08-21T00:18:05.8879966Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:18:05.8881050Z return self._call_impl(*args, **kwargs) 2024-08-21T00:18:05.8882005Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:18:05.8882822Z return forward_call(*args, **kwargs) 2024-08-21T00:18:05.8883720Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/modeling/poolers.py", line 246, in forward 2024-08-21T00:18:05.8884595Z return self.level_poolers[0](x[0], pooler_fmt_boxes) 2024-08-21T00:18:05.8885620Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-21T00:18:05.8886492Z return self._call_impl(*args, **kwargs) 2024-08-21T00:18:05.8887498Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-21T00:18:05.8888318Z return forward_call(*args, **kwargs) 2024-08-21T00:18:05.8889220Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/detectron2/layers/roi_align.py", line 58, in forward 2024-08-21T00:18:05.8890001Z return roi_align( 2024-08-21T00:18:05.8890823Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align 2024-08-21T00:18:05.8891915Z return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned) 2024-08-21T00:18:05.8893106Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 168, in _roi_align 2024-08-21T00:18:05.8894171Z val = _bilinear_interpolate(input, roi_batch_ind, y, x, ymask, xmask) # [K, C, PH, PW, IY, IX] 2024-08-21T00:18:05.8895389Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 62, in _bilinear_interpolate 2024-08-21T00:18:05.8896241Z v1 = masked_index(y_low, x_low) 2024-08-21T00:18:05.8897150Z File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 55, in masked_index 2024-08-21T00:18:05.8897947Z return input[ 2024-08-21T00:18:05.8901016Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 21890.87 GiB. GPU 0 has a total capacity of 21.98 GiB of which 16.54 GiB is free. Process 105913 has 5.42 GiB memory in use. Of the allocated memory 4.58 GiB is allocated by PyTorch, and 558.89 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-21T00:18:05.8903715Z 2024-08-21T00:18:05.8904014Z The above exception was the direct cause of the following exception: 2024-08-21T00:18:05.8904459Z 2024-08-21T00:18:05.8904606Z Traceback (most recent call last): 2024-08-21T00:18:05.8905257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:18:05.8905957Z ) = runner.load_model( 2024-08-21T00:18:05.8906612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-21T00:18:05.8907369Z self.validate_model(model, example_inputs) 2024-08-21T00:18:05.8908116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-21T00:18:05.8908870Z raise RuntimeError("Eager run failed") from e 2024-08-21T00:18:05.8909341Z RuntimeError: Eager run failed 2024-08-21T00:18:05.8909598Z 2024-08-21T00:18:05.8909717Z eager_fail_to_run 2024-08-21T00:18:09.5922441Z 2024-08-21T00:18:18.4658160Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:18.4658638Z loading model: 0it [00:08, ?it/s] 2024-08-21T00:18:18.4659059Z cuda train dlrm 2024-08-21T00:18:27.1008978Z skipping cudagraphs due to sparsity not handled. Please file issue for sparse inference weights. Found from : 2024-08-21T00:18:27.1011347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:18:27.1012924Z pred = mod(*cloned_inputs) 2024-08-21T00:18:27.1014078Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 355, in forward 2024-08-21T00:18:27.1015324Z return self.sequential_forward(dense_x, lS_o, lS_i) 2024-08-21T00:18:27.1016307Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 367, in sequential_forward 2024-08-21T00:18:27.1017260Z ly = self.apply_emb(lS_o, lS_i, self.emb_l) 2024-08-21T00:18:27.1018152Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm/dlrm_s_pytorch.py", line 308, in apply_emb 2024-08-21T00:18:27.1019097Z V = E(sparse_index_group_batch, sparse_offset_group_batch) 2024-08-21T00:18:27.1019473Z 2024-08-21T00:18:27.5784142Z pass 2024-08-21T00:18:27.6172822Z TIMING: entire_frame_compile:6.42668 code_gen:1.2372 inductor_compile:2.02871 backend_compile:5.69429 total_wall_time:7.36302 2024-08-21T00:18:27.6174193Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:3620 | ProxyTorchDispatchMode.__torch_dispatch__:1379 | FakeTensor.__torch_dispatch__:387 2024-08-21T00:18:27.6175282Z Dynamo produced 2 graphs covering 40 ops with 6 graph breaks (5 unique) 2024-08-21T00:18:31.5114375Z 2024-08-21T00:18:32.9005380Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:32.9005877Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:18:32.9006308Z cuda train drq 2024-08-21T00:18:40.6726286Z W0821 00:18:40.671000 49776 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:18:48.6247511Z pass 2024-08-21T00:18:48.6278263Z TIMING: entire_frame_compile:13.33301 code_gen:6.31499 inductor_compile:7.82773 backend_compile:11.23505 total_wall_time:14.50589 2024-08-21T00:18:48.6281326Z STATS: call_* op count: 136 | FakeTensorMode.__torch_dispatch__:6973 | ProxyTorchDispatchMode.__torch_dispatch__:3284 | FakeTensor.__torch_dispatch__:1487 2024-08-21T00:18:48.6283561Z Dynamo produced 5 graphs covering 136 ops with 7 graph breaks (5 unique) 2024-08-21T00:18:52.8398181Z 2024-08-21T00:18:54.7325917Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:18:54.7326922Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:18:54.7327784Z cuda train fastNLP_Bert 2024-08-21T00:19:01.2947762Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break from `Tensor.item()`, consider setting: 2024-08-21T00:19:01.2950263Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-21T00:19:01.2951375Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] or: 2024-08-21T00:19:01.2952347Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-21T00:19:01.2953644Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] to include these operations in the captured graph. 2024-08-21T00:19:01.2954622Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:19:01.2955508Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] Graph break: from user code at: 2024-08-21T00:19:01.2957104Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:19:01.2958810Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] pred = mod(*cloned_inputs) 2024-08-21T00:19:01.2960729Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/models/bert.py", line 265, in forward 2024-08-21T00:19:01.2962487Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] sequence_output = self.bert(words) 2024-08-21T00:19:01.2964291Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 137, in forward 2024-08-21T00:19:01.2965752Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] outputs = self.model(words) 2024-08-21T00:19:01.2967473Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/embeddings/bert_embedding.py", line 445, in forward 2024-08-21T00:19:01.2969484Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] max_word_piece_length = batch_word_pieces_length.sum(dim=-1).max().item() # 表示word piece的长度(包括padding) 2024-08-21T00:19:01.2972293Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:19:01.2973042Z W0821 00:19:01.293000 50081 torch/_dynamo/variables/tensor.py:776] [3/0] 2024-08-21T00:19:37.2524892Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:19:37.2527404Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 512, in forward 2024-08-21T00:19:37.2528386Z embedding_output = self.embeddings(input_ids, token_type_ids) 2024-08-21T00:19:37.2529574Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/fastNLP/modules/encoder/bert.py", line 236, in forward 2024-08-21T00:19:37.2530466Z words_embeddings = self.word_embeddings(input_ids) 2024-08-21T00:19:37.2530816Z 2024-08-21T00:19:40.1852574Z pass 2024-08-21T00:19:40.2515273Z TIMING: entire_frame_compile:25.14242 inductor_compile:20.55166 backend_compile:21.29168 code_gen:12.66805 total_wall_time:35.75417 2024-08-21T00:19:40.2518074Z STATS: call_* op count: 448 | FakeTensorMode.__torch_dispatch__:38676 | ProxyTorchDispatchMode.__torch_dispatch__:19109 | FakeTensor.__torch_dispatch__:5968 2024-08-21T00:19:40.2520311Z Dynamo produced 7 graphs covering 448 ops with 10 graph breaks (6 unique) 2024-08-21T00:19:45.4211017Z 2024-08-21T00:19:46.3382380Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:46.3382884Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:19:46.3383363Z cuda train functorch_dp_cifar10 2024-08-21T00:20:21.7745484Z W0821 00:20:21.773000 50445 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:20:39.8140098Z pass 2024-08-21T00:20:39.8190935Z TIMING: entire_frame_compile:49.87575 code_gen:16.80744 inductor_compile:28.78412 backend_compile:42.87445 total_wall_time:62.89012 2024-08-21T00:20:39.8194946Z STATS: call_* op count: 276 | FakeTensorMode.__torch_dispatch__:25662 | attempt fast:1468 | fast is_contiguous:1428 | ProxyTorchDispatchMode.__torch_dispatch__:7938 | slow both tensors nontrivially broadcast:40 | FakeTensor.__torch_dispatch__:6771 2024-08-21T00:20:39.8196503Z Dynamo produced 3 graphs covering 276 ops with 7 graph breaks (5 unique) 2024-08-21T00:20:45.5130246Z 2024-08-21T00:20:46.1541341Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:20:46.1541806Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:20:46.1544958Z cuda train functorch_maml_omniglot 2024-08-21T00:20:54.1812563Z W0821 00:20:54.180000 50942 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:20:59.8352569Z pass 2024-08-21T00:20:59.8353581Z TIMING: entire_frame_compile:8.79788 code_gen:6.68999 inductor_compile:8.02954 backend_compile:7.50291 total_wall_time:11.85534 2024-08-21T00:20:59.8355010Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:3923 | ProxyTorchDispatchMode.__torch_dispatch__:1433 | FakeTensor.__torch_dispatch__:1195 2024-08-21T00:20:59.8356118Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-21T00:21:03.9039085Z 2024-08-21T00:21:05.2549823Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:21:05.2550274Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:21:05.2550714Z cuda train hf_Albert 2024-08-21T00:22:19.3367914Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:22:19.3369651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:22:19.3371091Z pred = mod(*cloned_inputs) 2024-08-21T00:22:19.3372883Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 986, in forward 2024-08-21T00:22:19.3374348Z outputs = self.albert( 2024-08-21T00:22:19.3375952Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 729, in forward 2024-08-21T00:22:19.3377456Z embedding_output = self.embeddings( 2024-08-21T00:22:19.3379160Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/albert/modeling_albert.py", line 250, in forward 2024-08-21T00:22:19.3380690Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:22:19.3381253Z 2024-08-21T00:22:20.8865767Z W0821 00:22:20.885000 51229 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:22:30.9327031Z pass 2024-08-21T00:22:30.9480184Z TIMING: entire_frame_compile:75.359 code_gen:13.969 inductor_compile:32.93508 backend_compile:63.77476 total_wall_time:93.1021 2024-08-21T00:22:30.9483324Z STATS: call_* op count: 542 | FakeTensorMode.__torch_dispatch__:64854 | attempt fast:3509 | fast is_contiguous:3509 | ProxyTorchDispatchMode.__torch_dispatch__:21516 | FakeTensor.__torch_dispatch__:7421 2024-08-21T00:22:30.9485916Z Dynamo produced 3 graphs covering 542 ops with 6 graph breaks (5 unique) 2024-08-21T00:22:38.0805974Z 2024-08-21T00:22:42.0960248Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:22:42.0960903Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:22:42.0961449Z cuda train hf_Bart 2024-08-21T00:24:08.8456359Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:24:08.8457535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:24:08.8458490Z pred = mod(*cloned_inputs) 2024-08-21T00:24:08.8459429Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 69, in forward 2024-08-21T00:24:08.8460550Z return self.model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) 2024-08-21T00:24:08.8461912Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1731, in forward 2024-08-21T00:24:08.8462798Z outputs = self.model( 2024-08-21T00:24:08.8463740Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1599, in forward 2024-08-21T00:24:08.8464638Z encoder_outputs = self.encoder( 2024-08-21T00:24:08.8465613Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bart/modeling_bart.py", line 1152, in forward 2024-08-21T00:24:08.8466620Z inputs_embeds = self.embed_tokens(input_ids) * self.embed_scale 2024-08-21T00:24:08.8467035Z 2024-08-21T00:24:11.7903499Z W0821 00:24:11.789000 51607 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:25:10.4949690Z pass 2024-08-21T00:25:10.5423938Z TIMING: entire_frame_compile:132.76556 code_gen:32.67978 inductor_compile:63.72959 backend_compile:107.83346 total_wall_time:149.78901 2024-08-21T00:25:10.5426115Z STATS: call_* op count: 1249 | FakeTensorMode.__torch_dispatch__:93338 | attempt fast:3417 | fast is_contiguous:3413 | slow no contiguity match:4 | FakeTensor.__torch_dispatch__:20594 | ProxyTorchDispatchMode.__torch_dispatch__:32328 2024-08-21T00:25:10.5427803Z Dynamo produced 3 graphs covering 1249 ops with 6 graph breaks (5 unique) 2024-08-21T00:25:20.7172163Z 2024-08-21T00:25:23.3401464Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:25:23.3401943Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:25:23.3402436Z cuda train hf_Bert 2024-08-21T00:26:39.2774009Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:26:39.2776678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:26:39.2777682Z pred = mod(*cloned_inputs) 2024-08-21T00:26:39.2779039Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-21T00:26:39.2780126Z outputs = self.bert( 2024-08-21T00:26:39.2781105Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-21T00:26:39.2782020Z embedding_output = self.embeddings( 2024-08-21T00:26:39.2783116Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-21T00:26:39.2784039Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:26:39.2784389Z 2024-08-21T00:26:42.1177521Z W0821 00:26:42.116000 51929 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:27:27.9118064Z pass 2024-08-21T00:27:27.9528604Z TIMING: entire_frame_compile:111.77763 code_gen:26.01734 inductor_compile:49.75253 backend_compile:90.38791 total_wall_time:126.96203 2024-08-21T00:27:27.9531345Z STATS: call_* op count: 994 | FakeTensorMode.__torch_dispatch__:81925 | attempt fast:3121 | fast is_contiguous:3121 | ProxyTorchDispatchMode.__torch_dispatch__:27774 | FakeTensor.__torch_dispatch__:15918 2024-08-21T00:27:27.9532711Z Dynamo produced 3 graphs covering 994 ops with 6 graph breaks (5 unique) 2024-08-21T00:27:37.0650300Z 2024-08-21T00:27:41.9121287Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:27:41.9121752Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:27:41.9122176Z cuda train hf_Bert_large 2024-08-21T00:30:13.5219894Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:30:13.5221067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:30:13.5225738Z pred = mod(*cloned_inputs) 2024-08-21T00:30:13.5226985Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1360, in forward 2024-08-21T00:30:13.5227864Z outputs = self.bert( 2024-08-21T00:30:13.5228788Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 1006, in forward 2024-08-21T00:30:13.5229817Z embedding_output = self.embeddings( 2024-08-21T00:30:13.5231065Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/bert/modeling_bert.py", line 232, in forward 2024-08-21T00:30:13.5231994Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:30:13.5232336Z 2024-08-21T00:30:18.7426130Z W0821 00:30:18.741000 52155 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:31:50.5849075Z pass 2024-08-21T00:31:50.6991126Z TIMING: entire_frame_compile:219.72068 code_gen:52.3875 inductor_compile:98.28173 backend_compile:177.81176 total_wall_time:248.73998 2024-08-21T00:31:50.6995710Z STATS: call_* op count: 1918 | FakeTensorMode.__torch_dispatch__:159680 | attempt fast:6015 | fast is_contiguous:6015 | ProxyTorchDispatchMode.__torch_dispatch__:54414 | FakeTensor.__torch_dispatch__:30990 2024-08-21T00:31:50.6997795Z Dynamo produced 3 graphs covering 1918 ops with 6 graph breaks (5 unique) 2024-08-21T00:32:05.2355674Z 2024-08-21T00:32:08.8759921Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:32:08.8760517Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:32:08.8760997Z cuda train hf_BigBird 2024-08-21T00:32:20.1578859Z WARNING:common:fp64 golden ref were not generated for hf_BigBird. Setting accuracy check to cosine 2024-08-21T00:33:16.2635805Z W0821 00:33:16.262000 52504 torch/_inductor/utils.py:1391] [3/0_1] DeviceCopy in input program 2024-08-21T00:33:49.0005659Z skipping cudagraphs due to skipping cudagraphs due to cpu device (cat_1). Found from : 2024-08-21T00:33:49.0007130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:33:49.0009363Z pred = mod(*cloned_inputs) 2024-08-21T00:33:49.0010871Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2450, in forward 2024-08-21T00:33:49.0012021Z outputs = self.bert( 2024-08-21T00:33:49.0013114Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 2133, in forward 2024-08-21T00:33:49.0014054Z encoder_outputs = self.encoder( 2024-08-21T00:33:49.0015693Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1631, in forward 2024-08-21T00:33:49.0016771Z layer_outputs = layer_module( 2024-08-21T00:33:49.0017802Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1488, in forward 2024-08-21T00:33:49.0018756Z self_attention_outputs = self.attention( 2024-08-21T00:33:49.0019811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 1401, in forward 2024-08-21T00:33:49.0020722Z self_outputs = self.self( 2024-08-21T00:33:49.0021700Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 472, in forward 2024-08-21T00:33:49.0022769Z context_layer, attention_probs = self.bigbird_block_sparse_attention( 2024-08-21T00:33:49.0024064Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 593, in bigbird_block_sparse_attention 2024-08-21T00:33:49.0025100Z rand_attn = np.stack(rand_attn, axis=0) 2024-08-21T00:33:49.0025414Z 2024-08-21T00:33:49.7309837Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Error while creating guard: 2024-08-21T00:33:49.7311093Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Name: '' 2024-08-21T00:33:49.7311813Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Source: shape_env 2024-08-21T00:33:49.7312633Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Create Function: SHAPE_ENV 2024-08-21T00:33:49.7313444Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Guard Types: None 2024-08-21T00:33:49.7314224Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Code List: None 2024-08-21T00:33:49.7315184Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Object Weakref: None 2024-08-21T00:33:49.7316166Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Guarded Class Weakref: None 2024-08-21T00:33:49.7317355Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] Traceback (most recent call last): 2024-08-21T00:33:49.7318821Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create 2024-08-21T00:33:49.7320246Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] return self.create_fn(builder, self) 2024-08-21T00:33:49.7321838Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV 2024-08-21T00:33:49.7323437Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] guards = output_graph.shape_env.produce_guards( 2024-08-21T00:33:49.7325255Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards 2024-08-21T00:33:49.7326732Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] raise ConstraintViolationError( 2024-08-21T00:33:49.7328473Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-08-21T00:33:49.7330712Z E0821 00:33:49.729000 52504 torch/_guards.py:282] [3/0_1] - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4). 2024-08-21T00:33:49.7332018Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] Created at: 2024-08-21T00:33:49.7333338Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 603, in transform 2024-08-21T00:33:49.7334620Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] tracer = InstructionTranslator( 2024-08-21T00:33:49.7336062Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/symbolic_convert.py", line 2654, in __init__ 2024-08-21T00:33:49.7337292Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] output=OutputGraph( 2024-08-21T00:33:49.7338651Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 317, in __init__ 2024-08-21T00:33:49.7339887Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] self.init_ambient_guards() 2024-08-21T00:33:49.7341333Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/output_graph.py", line 463, in init_ambient_guards 2024-08-21T00:33:49.7342815Z E0821 00:33:49.730000 52504 torch/_guards.py:284] [3/0_1] self.guards.add(ShapeEnvSource().make_guard(GuardBuilder.SHAPE_ENV)) 2024-08-21T00:33:49.7343607Z ERROR:common: 2024-08-21T00:33:49.7343930Z Traceback (most recent call last): 2024-08-21T00:33:49.7344627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2819, in check_accuracy 2024-08-21T00:33:49.7345455Z new_result = optimized_model_iter_fn(model_copy, example_inputs) 2024-08-21T00:33:49.7346862Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 469, in _fn 2024-08-21T00:33:49.7347616Z return fn(*args, **kwargs) 2024-08-21T00:33:49.7348294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-21T00:33:49.7349092Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-21T00:33:49.7349982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 437, in forward_and_backward_pass 2024-08-21T00:33:49.7350795Z cloned_inputs = clone_inputs(inputs) 2024-08-21T00:33:49.7351753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 438, in torch_dynamo_resume_in_forward_and_backward_pass_at_437 2024-08-21T00:33:49.7352723Z self.optimizer_zero_grad(mod) 2024-08-21T00:33:49.7353606Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1238, in __call__ 2024-08-21T00:33:49.7354437Z return self._torchdynamo_orig_callable( 2024-08-21T00:33:49.7355349Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 1039, in __call__ 2024-08-21T00:33:49.7356142Z result = self._inner_convert( 2024-08-21T00:33:49.7357213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 514, in __call__ 2024-08-21T00:33:49.7357998Z return _compile( 2024-08-21T00:33:49.7358934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 902, in _compile 2024-08-21T00:33:49.7359848Z guarded_code = compile_inner(code, one_graph, hooks, transform) 2024-08-21T00:33:49.7360886Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 653, in compile_inner 2024-08-21T00:33:49.7361798Z return _compile_inner(code, one_graph, hooks, transform) 2024-08-21T00:33:49.7362790Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_utils_internal.py", line 87, in wrapper_function 2024-08-21T00:33:49.7363598Z return function(*args, **kwargs) 2024-08-21T00:33:49.7364514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/convert_frame.py", line 783, in _compile_inner 2024-08-21T00:33:49.7365343Z check_fn = CheckFunctionManager( 2024-08-21T00:33:49.7366213Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 2200, in __init__ 2024-08-21T00:33:49.7367082Z guard.create(builder) 2024-08-21T00:33:49.7367828Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_guards.py", line 280, in create 2024-08-21T00:33:49.7368606Z return self.create_fn(builder, self) 2024-08-21T00:33:49.7369484Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_dynamo/guards.py", line 1782, in SHAPE_ENV 2024-08-21T00:33:49.7370309Z guards = output_graph.shape_env.produce_guards( 2024-08-21T00:33:49.7371374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/experimental/symbolic_shapes.py", line 4185, in produce_guards 2024-08-21T00:33:49.7372286Z raise ConstraintViolationError( 2024-08-21T00:33:49.7373469Z torch.fx.experimental.symbolic_shapes.ConstraintViolationError: Constraints violated (L['cloned_inputs'][0].size()[0])! For more information, run with TORCH_LOGS="+dynamic". 2024-08-21T00:33:49.7375172Z - Not all values of RelaxedUnspecConstraint(L['cloned_inputs'][0].size()[0]) are valid because L['cloned_inputs'][0].size()[0] was inferred to be a constant (4). 2024-08-21T00:33:49.7376113Z 2024-08-21T00:33:49.7376121Z 2024-08-21T00:33:49.7376429Z You can suppress this exception and fall back to eager by setting: 2024-08-21T00:33:49.7377104Z import torch._dynamo 2024-08-21T00:33:49.7377618Z torch._dynamo.config.suppress_errors = True 2024-08-21T00:33:49.7377961Z 2024-08-21T00:33:49.7378255Z TorchDynamo optimized model failed to run because of following error 2024-08-21T00:33:49.7379329Z fail_to_run 2024-08-21T00:33:49.7997800Z TIMING: entire_frame_compile:0.02454 code_gen:27.44621 inductor_compile:37.15771 backend_compile:71.89094 total_wall_time:37.18225 2024-08-21T00:33:49.7999957Z STATS: call_* op count: 3361 | FakeTensorMode.__torch_dispatch__:152071 | attempt fast:102 | slow both tensors nontrivially broadcast:2 | fast is_contiguous:100 | ProxyTorchDispatchMode.__torch_dispatch__:74931 | FakeTensor.__torch_dispatch__:6438 2024-08-21T00:33:49.8001574Z Dynamo produced 1 graphs covering 3361 ops with 3 graph breaks (3 unique) 2024-08-21T00:33:56.6524966Z 2024-08-21T00:33:58.7936503Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:33:58.7937062Z loading model: 0it [00:02, ?it/s] 2024-08-21T00:33:58.7937506Z cuda train hf_DistilBert 2024-08-21T00:34:43.2409290Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:34:43.2411898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:34:43.2413643Z pred = mod(*cloned_inputs) 2024-08-21T00:34:43.2415002Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 905, in forward 2024-08-21T00:34:43.2416252Z dlbrt_output = self.distilbert( 2024-08-21T00:34:43.2417647Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 814, in forward 2024-08-21T00:34:43.2418958Z embeddings = self.embeddings(input_ids, inputs_embeds) # (bs, seq_length, dim) 2024-08-21T00:34:43.2420219Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 141, in forward 2024-08-21T00:34:43.2421337Z input_embeds = self.word_embeddings(input_ids) # (bs, max_seq_length, dim) 2024-08-21T00:34:43.2421804Z 2024-08-21T00:34:44.8009350Z W0821 00:34:44.800000 52828 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:35:09.7910876Z pass 2024-08-21T00:35:09.8163880Z TIMING: entire_frame_compile:63.19519 code_gen:16.52251 inductor_compile:32.08433 backend_compile:52.28631 total_wall_time:73.40192 2024-08-21T00:35:09.8167014Z STATS: call_* op count: 540 | FakeTensorMode.__torch_dispatch__:43832 | attempt fast:1715 | fast is_contiguous:1679 | slow no contiguity match:36 | ProxyTorchDispatchMode.__torch_dispatch__:14156 | FakeTensor.__torch_dispatch__:8621 2024-08-21T00:35:09.8168502Z Dynamo produced 3 graphs covering 540 ops with 6 graph breaks (5 unique) 2024-08-21T00:35:16.3872765Z 2024-08-21T00:35:19.6993011Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:35:19.6993487Z loading model: 0it [00:03, ?it/s] 2024-08-21T00:35:19.6993916Z cuda train hf_GPT2 2024-08-21T00:36:29.2578717Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:36:29.2579873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in torch_dynamo_resume_in_forward_and_backward_pass_at_438 2024-08-21T00:36:29.2581224Z pred = mod(*cloned_inputs) 2024-08-21T00:36:29.2582382Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward 2024-08-21T00:36:29.2586389Z transformer_outputs = self.transformer( 2024-08-21T00:36:29.2587524Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 837, in forward 2024-08-21T00:36:29.2588418Z inputs_embeds = self.wte(input_ids) 2024-08-21T00:36:29.2588723Z 2024-08-21T00:36:31.6374032Z W0821 00:36:31.636000 53085 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:37:06.1675855Z pass 2024-08-21T00:37:06.2452641Z TIMING: entire_frame_compile:97.25434 code_gen:23.58958 inductor_compile:45.69784 backend_compile:79.84372 total_wall_time:112.36093 2024-08-21T00:37:06.2454396Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:66243 | attempt fast:2910 | fast is_contiguous:2886 | ProxyTorchDispatchMode.__torch_dispatch__:21916 | slow no contiguity match:24 | FakeTensor.__torch_dispatch__:13227 2024-08-21T00:37:06.2455869Z Dynamo produced 3 graphs covering 1087 ops with 6 graph breaks (5 unique) 2024-08-21T00:37:14.5137048Z 2024-08-21T00:37:27.9467478Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:37:27.9467950Z loading model: 0it [00:13, ?it/s] 2024-08-21T00:37:27.9470997Z cuda train hf_GPT2_large 2024-08-21T00:37:27.9471609Z pass_due_to_skip 2024-08-21T00:37:28.4508149Z TIMING: 2024-08-21T00:37:28.4508996Z STATS: call_* op count: 0 2024-08-21T00:37:28.4510114Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:37:32.0076852Z 2024-08-21T00:37:33.6636333Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:37:33.6637052Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:37:33.6637714Z cuda train hf_Reformer 2024-08-21T00:37:48.5491158Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-21T00:37:48.5492884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 2094, in torch_dynamo_resume_in_forward_at_2066 2024-08-21T00:37:48.5494213Z embedding_output = self.embeddings( 2024-08-21T00:37:48.5496014Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/reformer/modeling_reformer.py", line 254, in forward 2024-08-21T00:37:48.5497206Z inputs_embeds = self.word_embeddings(input_ids) 2024-08-21T00:37:48.5497545Z 2024-08-21T00:37:54.1960197Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:37:57.0979153Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:37:58.5908572Z skipping cudagraphs due to ['incompatible ops'] 2024-08-21T00:38:00.4764819Z W0821 00:38:00.475000 53484 torch/_logging/_internal.py:1058] [32/0] Profiler function will be ignored 2024-08-21T00:38:25.2800530Z pass 2024-08-21T00:38:25.2926253Z TIMING: entire_frame_compile:35.36629 inductor_compile:21.45792 backend_compile:28.02281 code_gen:15.77922 total_wall_time:37.14157 2024-08-21T00:38:25.2929776Z STATS: call_* op count: 688 | FakeTensorMode.__torch_dispatch__:22576 | ProxyTorchDispatchMode.__torch_dispatch__:8320 | FakeTensor.__torch_dispatch__:4848 | attempt fast:53 | fast is_contiguous:44 | slow no contiguity match:9 2024-08-21T00:38:25.2932682Z Dynamo produced 17 graphs covering 688 ops with 25 graph breaks (9 unique) 2024-08-21T00:38:30.3163441Z 2024-08-21T00:38:34.6901898Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:38:34.6902617Z loading model: 0it [00:04, ?it/s] 2024-08-21T00:38:34.6903270Z cuda train hf_T5_base 2024-08-21T00:38:37.8380295Z WARNING:common:fp64 golden ref were not generated for hf_T5_base. Setting accuracy check to cosine 2024-08-21T00:38:39.9501286Z ERROR:common: 2024-08-21T00:38:39.9501700Z Traceback (most recent call last): 2024-08-21T00:38:39.9502439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2753, in check_accuracy 2024-08-21T00:38:39.9503194Z correct_rerun_result = self.run_n_iterations( 2024-08-21T00:38:39.9504017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2545, in run_n_iterations 2024-08-21T00:38:39.9504969Z self.model_iter_fn(mod, inputs, collect_outputs=False) 2024-08-21T00:38:39.9506125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-21T00:38:39.9507214Z self.grad_scaler.scale(loss).backward() 2024-08-21T00:38:39.9509918Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-21T00:38:39.9510703Z torch.autograd.backward( 2024-08-21T00:38:39.9511568Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-21T00:38:39.9512328Z _engine_run_backward( 2024-08-21T00:38:39.9513203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-21T00:38:39.9514336Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-21T00:38:39.9517873Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 192.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 88.44 MiB is free. Process 111015 has 21.88 GiB memory in use. Of the allocated memory 20.95 GiB is allocated by PyTorch, and 595.70 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-21T00:38:40.0353820Z eager_2nd_run_OOM 2024-08-21T00:38:40.0385350Z TIMING: 2024-08-21T00:38:40.0385670Z STATS: call_* op count: 0 2024-08-21T00:38:40.0386529Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:38:43.6754459Z 2024-08-21T00:38:54.0371850Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:38:54.0372452Z loading model: 0it [00:10, ?it/s] 2024-08-21T00:38:54.0372988Z cuda train hf_T5_large 2024-08-21T00:38:54.0376138Z pass_due_to_skip 2024-08-21T00:38:54.5101987Z TIMING: 2024-08-21T00:38:54.5102606Z STATS: call_* op count: 0 2024-08-21T00:38:54.5103313Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-21T00:38:58.1293128Z 2024-08-21T00:38:59.2991721Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:38:59.2992320Z loading model: 0it [00:01, ?it/s] 2024-08-21T00:38:59.2992781Z cuda train hf_Whisper 2024-08-21T00:39:29.9079705Z W0821 00:39:29.907000 53980 torch/_logging/_internal.py:1058] [5/0] Profiler function will be ignored 2024-08-21T00:39:48.4612300Z pass 2024-08-21T00:39:48.4678453Z TIMING: entire_frame_compile:43.03841 code_gen:14.89681 inductor_compile:23.25064 backend_compile:35.94091 total_wall_time:49.76348 2024-08-21T00:39:48.4682118Z STATS: call_* op count: 346 | FakeTensorMode.__torch_dispatch__:25761 | attempt fast:975 | fast is_contiguous:919 | slow no contiguity match:56 | FakeTensor.__torch_dispatch__:5928 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2024-08-21T00:39:48.4683608Z Dynamo produced 3 graphs covering 346 ops with 6 graph breaks (5 unique) 2024-08-21T00:39:53.9747232Z 2024-08-21T00:39:54.2771248Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:39:54.2771894Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:39:54.2772512Z cuda train hf_distil_whisper 2024-08-21T00:39:54.2791280Z Traceback (most recent call last): 2024-08-21T00:39:54.2792164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-21T00:39:54.2792966Z ) = runner.load_model( 2024-08-21T00:39:54.2793626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-21T00:39:54.2794859Z benchmark = benchmark_cls( 2024-08-21T00:39:54.2795675Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-21T00:39:54.2796470Z obj = type.__call__(cls, *args, **kwargs) 2024-08-21T00:39:54.2797990Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper/__init__.py", line 12, in __init__ 2024-08-21T00:39:54.2799011Z raise NotImplementedError("Training is not implemented.") 2024-08-21T00:39:54.2799676Z NotImplementedError: Training is not implemented. 2024-08-21T00:39:54.2800057Z 2024-08-21T00:39:54.2800176Z model_fail_to_load 2024-08-21T00:39:57.7201063Z 2024-08-21T00:39:58.1962780Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:39:58.1963411Z loading model: 0it [00:00, ?it/s] 2024-08-21T00:39:58.1964100Z cuda train lennard_jones 2024-08-21T00:40:02.7479376Z W0821 00:40:02.746000 54387 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-21T00:40:06.2679042Z pass 2024-08-21T00:40:06.2680205Z TIMING: entire_frame_compile:6.38604 code_gen:2.41653 inductor_compile:3.92307 backend_compile:5.42561 total_wall_time:6.98361 2024-08-21T00:40:06.2681737Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:3628 | attempt fast:82 | fast is_contiguous:82 | ProxyTorchDispatchMode.__torch_dispatch__:1221 | FakeTensor.__torch_dispatch__:796 2024-08-21T00:40:06.2683016Z Dynamo produced 3 graphs covering 60 ops with 7 graph breaks (5 unique) 2024-08-21T00:40:07.4179554Z accuracy pass_rate=72.73% 2024-08-21T00:40:07.4180260Z calls_captured gmean=0.00x mean=427.030x 2024-08-21T00:40:07.4181133Z unique_graphs gmean=0.00x mean=3.091x 2024-08-21T00:40:07.4184753Z graph_breaks gmean=0.00x mean=6.212x 2024-08-21T00:40:07.4188069Z unique_graph_breaks gmean=0.00x mean=4.121x 2024-08-21T00:40:07.4191360Z autograd_captures gmean=0.00x mean=0.000x 2024-08-21T00:40:07.4194689Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-21T00:40:07.4197911Z cudagraph_skips gmean=0.00x mean=0.727x 2024-08-21T00:40:08.6017616Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv 2024-08-21T00:40:08.9363680Z torchrec_dlrm XFAIL 2024-08-21T00:40:08.9367538Z BERT_pytorch PASS 2024-08-21T00:40:08.9371594Z Background_Matting XFAIL 2024-08-21T00:40:08.9376212Z LearningToPaint PASS 2024-08-21T00:40:08.9380972Z Super_SloMo PASS 2024-08-21T00:40:08.9385788Z alexnet PASS 2024-08-21T00:40:08.9390633Z basic_gnn_edgecnn PASS 2024-08-21T00:40:08.9395283Z basic_gnn_gcn PASS 2024-08-21T00:40:08.9399943Z basic_gnn_gin PASS 2024-08-21T00:40:08.9404567Z basic_gnn_sage PASS 2024-08-21T00:40:08.9409576Z dcgan PASS 2024-08-21T00:40:08.9414382Z demucs XFAIL 2024-08-21T00:40:08.9419112Z densenet121 PASS 2024-08-21T00:40:08.9423893Z detectron2_maskrcnn_r_50_c4 XFAIL 2024-08-21T00:40:08.9428656Z dlrm PASS 2024-08-21T00:40:08.9433398Z drq PASS 2024-08-21T00:40:08.9438278Z fastNLP_Bert PASS 2024-08-21T00:40:08.9443082Z functorch_dp_cifar10 PASS 2024-08-21T00:40:08.9448287Z functorch_maml_omniglot PASS 2024-08-21T00:40:08.9453263Z hf_Albert PASS 2024-08-21T00:40:08.9458117Z hf_Bart PASS 2024-08-21T00:40:08.9462936Z hf_Bert PASS 2024-08-21T00:40:08.9467686Z hf_Bert_large PASS 2024-08-21T00:40:08.9472640Z hf_BigBird XFAIL 2024-08-21T00:40:08.9477419Z hf_DistilBert PASS 2024-08-21T00:40:08.9482219Z hf_GPT2 PASS 2024-08-21T00:40:08.9487004Z hf_GPT2_large XFAIL 2024-08-21T00:40:08.9492027Z hf_Reformer PASS 2024-08-21T00:40:08.9496706Z hf_T5_base XFAIL 2024-08-21T00:40:08.9501482Z hf_T5_large XFAIL 2024-08-21T00:40:08.9506168Z hf_Whisper PASS 2024-08-21T00:40:08.9510996Z hf_distil_whisper XFAIL 2024-08-21T00:40:08.9515712Z lennard_jones PASS 2024-08-21T00:40:09.0031514Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_inductor_torchbench_training.csv 2024-08-21T00:40:09.3413145Z torchrec_dlrm PASS 2024-08-21T00:40:09.3415832Z BERT_pytorch PASS 2024-08-21T00:40:09.3420489Z Background_Matting PASS 2024-08-21T00:40:09.3425039Z LearningToPaint PASS 2024-08-21T00:40:09.3429646Z Super_SloMo PASS 2024-08-21T00:40:09.3434390Z alexnet PASS 2024-08-21T00:40:09.3439008Z basic_gnn_edgecnn PASS 2024-08-21T00:40:09.3443600Z basic_gnn_gcn PASS 2024-08-21T00:40:09.3450078Z basic_gnn_gin PASS 2024-08-21T00:40:09.3455277Z basic_gnn_sage PASS 2024-08-21T00:40:09.3459667Z dcgan PASS 2024-08-21T00:40:09.3464047Z demucs PASS 2024-08-21T00:40:09.3468584Z densenet121 PASS 2024-08-21T00:40:09.3472949Z detectron2_maskrcnn_r_50_c4 PASS 2024-08-21T00:40:09.3477431Z dlrm PASS 2024-08-21T00:40:09.3482055Z drq PASS 2024-08-21T00:40:09.3486447Z fastNLP_Bert PASS 2024-08-21T00:40:09.3491307Z functorch_dp_cifar10 PASS 2024-08-21T00:40:09.3495461Z functorch_maml_omniglot PASS 2024-08-21T00:40:09.3500086Z hf_Albert PASS 2024-08-21T00:40:09.3504605Z hf_Bart PASS 2024-08-21T00:40:09.3509530Z hf_Bert PASS 2024-08-21T00:40:09.3513363Z hf_Bert_large PASS 2024-08-21T00:40:09.3517929Z hf_BigBird PASS 2024-08-21T00:40:09.3522391Z hf_DistilBert PASS 2024-08-21T00:40:09.3527035Z hf_GPT2 PASS 2024-08-21T00:40:09.3532115Z hf_GPT2_large PASS 2024-08-21T00:40:09.3536469Z hf_Reformer PASS 2024-08-21T00:40:09.3541095Z hf_T5_base PASS 2024-08-21T00:40:09.3545890Z hf_T5_large PASS 2024-08-21T00:40:09.3550883Z hf_Whisper PASS 2024-08-21T00:40:09.3555399Z hf_distil_whisper PASS 2024-08-21T00:40:09.3560248Z lennard_jones PASS 2024-08-21T00:40:09.4084412Z + cleanup_workspace 2024-08-21T00:40:09.4085564Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-21T00:40:09.4086654Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-21T00:40:09.4087681Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-21T00:40:09.4088308Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-21T00:40:09.4089103Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-21T00:40:09.4089973Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-21T00:40:09.4090666Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-21T00:40:10.1756563Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-21T00:40:10.1757079Z cat test/**/*_toprint.log || true 2024-08-21T00:40:10.1770774Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.1771280Z env: 2024-08-21T00:40:10.1771548Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.1771994Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.1772721Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.1773371Z ##[endgroup] 2024-08-21T00:40:10.1865493Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-21T00:40:10.1897026Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:40:10.1897492Z kill "$MONITOR_SCRIPT_PID" 2024-08-21T00:40:10.1905830Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.1906324Z env: 2024-08-21T00:40:10.1906596Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.1907043Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.1907761Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.1908411Z MONITOR_SCRIPT_PID: 52311 2024-08-21T00:40:10.1908765Z ##[endgroup] 2024-08-21T00:40:10.2108171Z Prepare all required actions 2024-08-21T00:40:10.2108614Z Getting action download info 2024-08-21T00:40:10.4144512Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-21T00:40:10.5876741Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-21T00:40:10.5877216Z with: 2024-08-21T00:40:10.5877825Z file-suffix: test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448 2024-08-21T00:40:10.5878556Z s3-bucket: gha-artifacts 2024-08-21T00:40:10.5878904Z env: 2024-08-21T00:40:10.5879178Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.5879624Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.5880357Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.5880999Z ##[endgroup] 2024-08-21T00:40:10.5907873Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-21T00:40:10.5908526Z # Remove any previous test jsons if they exist 2024-08-21T00:40:10.5909037Z rm -f test-jsons-*.zip 2024-08-21T00:40:10.5909557Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-21T00:40:10.5919287Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.5919977Z env: 2024-08-21T00:40:10.5920270Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.5920729Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.5921474Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.5922458Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448 2024-08-21T00:40:10.5923183Z ##[endgroup] 2024-08-21T00:40:10.6341955Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-21T00:40:10.6392985Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-21T00:40:10.6393709Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-21T00:40:10.6409679Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-21T00:40:10.6426730Z adding: test/slow_tests.json (deflated 82%) 2024-08-21T00:40:10.6460772Z ##[group]Run # Remove any previous test reports if they exist 2024-08-21T00:40:10.6461424Z # Remove any previous test reports if they exist 2024-08-21T00:40:10.6461947Z rm -f test-reports-*.zip 2024-08-21T00:40:10.6462536Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-21T00:40:10.6471610Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.6472109Z env: 2024-08-21T00:40:10.6472393Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.6472841Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.6473565Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.6474552Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448 2024-08-21T00:40:10.6475266Z ##[endgroup] 2024-08-21T00:40:10.6737339Z adding: test/test-reports/inference_torchbench.csv (deflated 73%) 2024-08-21T00:40:10.6741967Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-21T00:40:10.6743265Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 79%) 2024-08-21T00:40:10.6744127Z adding: test/test-reports/training_torchbench.csv (deflated 63%) 2024-08-21T00:40:10.6751035Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-21T00:40:10.6752209Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%) 2024-08-21T00:40:10.6785325Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-21T00:40:10.6785947Z # Remove any previous usage logs if they exist 2024-08-21T00:40:10.6786455Z rm -f logs-*.zip 2024-08-21T00:40:10.6787334Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-21T00:40:10.6788111Z # so check to see if the file exists first 2024-08-21T00:40:10.6788629Z if [ -f 'usage_log.txt' ]; then 2024-08-21T00:40:10.6789151Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-21T00:40:10.6789648Z fi 2024-08-21T00:40:10.6790022Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-21T00:40:10.6790604Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-21T00:40:10.6791097Z fi 2024-08-21T00:40:10.6799844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.6800358Z env: 2024-08-21T00:40:10.6800647Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.6801098Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.6801841Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.6802829Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448 2024-08-21T00:40:10.6803548Z ##[endgroup] 2024-08-21T00:40:10.6925476Z adding: usage_log.txt (deflated 92%) 2024-08-21T00:40:10.7015309Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-21T00:40:10.7016200Z # Remove any previous debugging artifacts if they exist 2024-08-21T00:40:10.7016749Z rm -f debug-*.zip 2024-08-21T00:40:10.7017131Z if [ -d 'test/debug' ]; then 2024-08-21T00:40:10.7017627Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-21T00:40:10.7018102Z fi 2024-08-21T00:40:10.7026500Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:10.7026995Z env: 2024-08-21T00:40:10.7027272Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.7027727Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.7028458Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.7029460Z FILE_SUFFIX: test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448 2024-08-21T00:40:10.7030182Z ##[endgroup] 2024-08-21T00:40:10.7126610Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:40:10.7127177Z with: 2024-08-21T00:40:10.7127482Z s3-bucket: gha-artifacts 2024-08-21T00:40:10.7127917Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:10.7128387Z retention-days: 14 2024-08-21T00:40:10.7128727Z if-no-files-found: warn 2024-08-21T00:40:10.7129092Z path: test-jsons-*.zip 2024-08-21T00:40:10.7129442Z name: artifact 2024-08-21T00:40:10.7129751Z region: us-east-1 2024-08-21T00:40:10.7130059Z env: 2024-08-21T00:40:10.7130343Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:10.7130794Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:10.7131519Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:10.7132159Z ##[endgroup] 2024-08-21T00:40:11.1009487Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:40:11.1010722Z With the provided path, there will be 1 file uploaded 2024-08-21T00:40:11.1011372Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:11.2296026Z Starting upload of test-jsons-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:11.3820630Z Finished upload of test-jsons-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:11.4111314Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:40:11.4111761Z with: 2024-08-21T00:40:11.4112060Z s3-bucket: gha-artifacts 2024-08-21T00:40:11.4112504Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:11.4112975Z retention-days: 14 2024-08-21T00:40:11.4113323Z if-no-files-found: error 2024-08-21T00:40:11.4113697Z path: test-reports-*.zip 2024-08-21T00:40:11.4114054Z name: artifact 2024-08-21T00:40:11.4114363Z region: us-east-1 2024-08-21T00:40:11.4114888Z env: 2024-08-21T00:40:11.4115176Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:11.4115632Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:11.4116370Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:11.4117030Z ##[endgroup] 2024-08-21T00:40:11.7651219Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:40:11.7652082Z With the provided path, there will be 1 file uploaded 2024-08-21T00:40:11.7652900Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:11.7710435Z Starting upload of test-reports-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:11.8618020Z Finished upload of test-reports-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:11.8905753Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:40:11.8906205Z with: 2024-08-21T00:40:11.8906514Z s3-bucket: gha-artifacts 2024-08-21T00:40:11.8906956Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:11.8907432Z retention-days: 14 2024-08-21T00:40:11.8907771Z if-no-files-found: ignore 2024-08-21T00:40:11.8908137Z path: logs-*.zip 2024-08-21T00:40:11.8908636Z name: artifact 2024-08-21T00:40:11.8908954Z region: us-east-1 2024-08-21T00:40:11.8909261Z env: 2024-08-21T00:40:11.8909548Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:11.8909995Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:11.8910735Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:11.8911423Z ##[endgroup] 2024-08-21T00:40:12.2450563Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-21T00:40:12.2451266Z With the provided path, there will be 1 file uploaded 2024-08-21T00:40:12.2452041Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:12.2507089Z Starting upload of logs-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:12.3710315Z Finished upload of logs-test-dynamic_inductor_torchbench-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026529448.zip 2024-08-21T00:40:12.3992851Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-21T00:40:12.3993308Z with: 2024-08-21T00:40:12.3993595Z s3-bucket: gha-artifacts 2024-08-21T00:40:12.3994029Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-21T00:40:12.3994499Z retention-days: 14 2024-08-21T00:40:12.3994835Z if-no-files-found: ignore 2024-08-21T00:40:12.3995199Z path: debug-*.zip 2024-08-21T00:40:12.3995511Z name: artifact 2024-08-21T00:40:12.3995819Z region: us-east-1 2024-08-21T00:40:12.3996121Z env: 2024-08-21T00:40:12.3996399Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:12.3996847Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:12.3997581Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:12.3998216Z ##[endgroup] 2024-08-21T00:40:12.7475088Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-21T00:40:12.7762220Z ##[group]Run # shellcheck disable=SC2156 2024-08-21T00:40:12.7762710Z # shellcheck disable=SC2156 2024-08-21T00:40:12.7763510Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-21T00:40:12.7772843Z shell: /usr/bin/bash -e {0} 2024-08-21T00:40:12.7773203Z env: 2024-08-21T00:40:12.7773487Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:12.7773937Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:12.7774659Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:12.7775298Z ##[endgroup] 2024-08-21T00:40:13.0449069Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-21T00:40:13.0449614Z with: 2024-08-21T00:40:13.0449879Z env: 2024-08-21T00:40:13.0450155Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:13.0450598Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:13.0451330Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:13.0451968Z ##[endgroup] 2024-08-21T00:40:13.0471375Z ##[group]Run set -eou pipefail 2024-08-21T00:40:13.0471787Z set -eou pipefail 2024-08-21T00:40:13.0472137Z  2024-08-21T00:40:13.0472638Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-21T00:40:13.0473271Z for _ in $(seq 1440); do 2024-08-21T00:40:13.0473736Z  # Break if no ssh session exists anymore 2024-08-21T00:40:13.0474236Z  if [ "$(who)" = "" ]; then 2024-08-21T00:40:13.0474649Z  break 2024-08-21T00:40:13.0474991Z  fi 2024-08-21T00:40:13.0475294Z  echo "." 2024-08-21T00:40:13.0475621Z  sleep 5 2024-08-21T00:40:13.0475944Z done 2024-08-21T00:40:13.0484744Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:13.0485232Z env: 2024-08-21T00:40:13.0485509Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:13.0485961Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:13.0486955Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:13.0487591Z ##[endgroup] 2024-08-21T00:40:13.0521062Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-21T00:40:13.0608180Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:40:13.0608940Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-21T00:40:13.0609530Z # shellcheck disable=SC2046 2024-08-21T00:40:13.0609999Z docker stop $(docker ps -q) || true 2024-08-21T00:40:13.0610478Z # Prune all of the docker images 2024-08-21T00:40:13.0610927Z docker system prune -af 2024-08-21T00:40:13.0619816Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:13.0620312Z env: 2024-08-21T00:40:13.0620593Z GIT_DEFAULT_BRANCH: main 2024-08-21T00:40:13.0621049Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-21T00:40:13.0621785Z DOCKER_CONTAINER_ID: 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:13.0622442Z ##[endgroup] 2024-08-21T00:40:13.7180812Z 586d7babfd6e 2024-08-21T00:40:17.6801017Z Deleted Containers: 2024-08-21T00:40:17.6801730Z 586d7babfd6e740949061ec1eb3e108dfbdb4c26942f41878a6f7ea3e2ff4ceb 2024-08-21T00:40:17.6802202Z 2024-08-21T00:40:26.6400648Z Deleted Images: 2024-08-21T00:40:26.6402450Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-21T00:40:26.6405390Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-21T00:40:26.6407164Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-21T00:40:26.6408006Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-21T00:40:26.6408859Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-21T00:40:26.6409662Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-21T00:40:26.6410458Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-21T00:40:26.6411255Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-21T00:40:26.6412058Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-21T00:40:26.6413292Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-21T00:40:26.6414227Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-21T00:40:26.6415226Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-21T00:40:26.6416043Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-21T00:40:26.6416842Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-21T00:40:26.6417650Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-21T00:40:26.6418465Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-21T00:40:26.6419273Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-21T00:40:26.6420090Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-21T00:40:26.6420911Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-21T00:40:26.6421727Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-21T00:40:26.6422538Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-21T00:40:26.6423344Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-21T00:40:26.6424152Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-21T00:40:26.6425207Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-21T00:40:26.6426031Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-21T00:40:26.6426848Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-21T00:40:26.6427656Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-21T00:40:26.6428480Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-21T00:40:26.6429308Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-21T00:40:26.6430135Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-21T00:40:26.6430950Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-21T00:40:26.6431765Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-21T00:40:26.6432823Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-21T00:40:26.6433927Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-21T00:40:26.6434986Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-21T00:40:26.6436041Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-21T00:40:26.6437104Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-21T00:40:26.6438175Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-21T00:40:26.6439255Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-21T00:40:26.6440329Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-21T00:40:26.6441429Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-21T00:40:26.6442563Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-21T00:40:26.6443669Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-21T00:40:26.6444819Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-21T00:40:26.6446169Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-21T00:40:26.6447308Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-21T00:40:26.6448585Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-21T00:40:26.6449672Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-21T00:40:26.6450739Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-21T00:40:26.6451770Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-21T00:40:26.6452834Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-21T00:40:26.6453959Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-21T00:40:26.6455061Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-21T00:40:26.6456157Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-21T00:40:26.6457246Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-21T00:40:26.6458364Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-21T00:40:26.6459437Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-21T00:40:26.6460295Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-21T00:40:26.6461111Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-21T00:40:26.6461916Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-21T00:40:26.6463962Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-21T00:40:26.6464837Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-21T00:40:26.6465658Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-21T00:40:26.6466482Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-21T00:40:26.6467304Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-21T00:40:26.6468095Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-21T00:40:26.6468902Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-21T00:40:26.6469714Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-21T00:40:26.6470522Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-21T00:40:26.6471339Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-21T00:40:26.6472159Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-21T00:40:26.6472964Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-21T00:40:26.6473749Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-21T00:40:26.6474546Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-21T00:40:26.6475409Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-21T00:40:26.6476216Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-21T00:40:26.6477032Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-21T00:40:26.6477855Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-21T00:40:26.6478671Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-21T00:40:26.6479492Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-21T00:40:26.6480294Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-21T00:40:26.6480771Z 2024-08-21T00:40:26.6480915Z Total reclaimed space: 49.69GB 2024-08-21T00:40:26.6547571Z Post job cleanup. 2024-08-21T00:40:26.6597346Z Post job cleanup. 2024-08-21T00:40:26.7511269Z [command]/usr/bin/git version 2024-08-21T00:40:26.7590688Z git version 2.40.1 2024-08-21T00:40:26.7628268Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/e29b5ec9-2c12-4d6d-8802-7643a646edc1' before making global git config changes 2024-08-21T00:40:26.7629488Z Adding repository directory to the temporary git global config as a safe directory 2024-08-21T00:40:26.7633354Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-21T00:40:26.7682273Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-21T00:40:26.7726968Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-21T00:40:26.8118738Z Entering 'android/libs/fbjni' 2024-08-21T00:40:26.8187352Z Entering 'third_party/FP16' 2024-08-21T00:40:26.8255125Z Entering 'third_party/FXdiv' 2024-08-21T00:40:26.8325801Z Entering 'third_party/NNPACK' 2024-08-21T00:40:26.8395121Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:40:26.8462420Z Entering 'third_party/XNNPACK' 2024-08-21T00:40:26.8544898Z Entering 'third_party/benchmark' 2024-08-21T00:40:26.8619360Z Entering 'third_party/cpp-httplib' 2024-08-21T00:40:26.8690382Z Entering 'third_party/cpuinfo' 2024-08-21T00:40:26.8761709Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:40:26.8829385Z Entering 'third_party/cutlass' 2024-08-21T00:40:26.8906133Z Entering 'third_party/eigen' 2024-08-21T00:40:26.8979344Z Entering 'third_party/fbgemm' 2024-08-21T00:40:26.9045252Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:40:26.9110717Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:40:26.9176700Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:40:26.9251082Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:40:26.9317997Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:40:26.9391744Z Entering 'third_party/flatbuffers' 2024-08-21T00:40:26.9460937Z Entering 'third_party/fmt' 2024-08-21T00:40:26.9528206Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:40:26.9595384Z Entering 'third_party/gloo' 2024-08-21T00:40:26.9662888Z Entering 'third_party/googletest' 2024-08-21T00:40:26.9729070Z Entering 'third_party/ideep' 2024-08-21T00:40:26.9794243Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:40:26.9870824Z Entering 'third_party/ittapi' 2024-08-21T00:40:26.9938954Z Entering 'third_party/kineto' 2024-08-21T00:40:27.0007555Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:40:27.0072753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:40:27.0142435Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:40:27.0214009Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:40:27.0282096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:40:27.0349289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:40:27.0419703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:40:27.0487789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:40:27.0560866Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:40:27.0629806Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:40:27.0701575Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:40:27.0767672Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:40:27.0837047Z Entering 'third_party/mimalloc' 2024-08-21T00:40:27.0908060Z Entering 'third_party/nccl/nccl' 2024-08-21T00:40:27.0975977Z Entering 'third_party/nlohmann' 2024-08-21T00:40:27.1044420Z Entering 'third_party/onnx' 2024-08-21T00:40:27.1125936Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:40:27.1200743Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:40:27.1273246Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:40:27.1340482Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:40:27.1408904Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:40:27.1476977Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:40:27.1542499Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:40:27.1611650Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:40:27.1678842Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:40:27.1747672Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:40:27.1811049Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:40:27.1882792Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:40:27.1956016Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:40:27.2042986Z Entering 'third_party/pocketfft' 2024-08-21T00:40:27.2116304Z Entering 'third_party/protobuf' 2024-08-21T00:40:27.2185499Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:40:27.2253468Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:40:27.2324150Z Entering 'third_party/psimd' 2024-08-21T00:40:27.2391324Z Entering 'third_party/pthreadpool' 2024-08-21T00:40:27.2459490Z Entering 'third_party/pybind11' 2024-08-21T00:40:27.2527395Z Entering 'third_party/python-peachpy' 2024-08-21T00:40:27.2594385Z Entering 'third_party/sleef' 2024-08-21T00:40:27.2671200Z Entering 'third_party/tensorpipe' 2024-08-21T00:40:27.2736512Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:40:27.2803934Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:40:27.2871449Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:40:27.2937527Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:40:27.3002347Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:40:27.3101580Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-21T00:40:27.3137354Z http.https://github.com/.extraheader 2024-08-21T00:40:27.3148656Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-21T00:40:27.3196309Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-21T00:40:27.3560758Z Entering 'android/libs/fbjni' 2024-08-21T00:40:27.3605091Z http.https://github.com/.extraheader 2024-08-21T00:40:27.3650933Z Entering 'third_party/FP16' 2024-08-21T00:40:27.3695971Z http.https://github.com/.extraheader 2024-08-21T00:40:27.3740145Z Entering 'third_party/FXdiv' 2024-08-21T00:40:27.3784754Z http.https://github.com/.extraheader 2024-08-21T00:40:27.3832717Z Entering 'third_party/NNPACK' 2024-08-21T00:40:27.3876940Z http.https://github.com/.extraheader 2024-08-21T00:40:27.3920068Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-21T00:40:27.3964317Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4011994Z Entering 'third_party/XNNPACK' 2024-08-21T00:40:27.4058404Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4118523Z Entering 'third_party/benchmark' 2024-08-21T00:40:27.4162363Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4206384Z Entering 'third_party/cpp-httplib' 2024-08-21T00:40:27.4251071Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4294645Z Entering 'third_party/cpuinfo' 2024-08-21T00:40:27.4338763Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4384324Z Entering 'third_party/cudnn_frontend' 2024-08-21T00:40:27.4427766Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4472151Z Entering 'third_party/cutlass' 2024-08-21T00:40:27.4516592Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4567402Z Entering 'third_party/eigen' 2024-08-21T00:40:27.4611535Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4657023Z Entering 'third_party/fbgemm' 2024-08-21T00:40:27.4700589Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4742968Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-21T00:40:27.4785866Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4827910Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-21T00:40:27.4871592Z http.https://github.com/.extraheader 2024-08-21T00:40:27.4916983Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-21T00:40:27.4963162Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5016779Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-21T00:40:27.5063895Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5107709Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-21T00:40:27.5153487Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5199065Z Entering 'third_party/flatbuffers' 2024-08-21T00:40:27.5242480Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5289284Z Entering 'third_party/fmt' 2024-08-21T00:40:27.5332879Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5376483Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-21T00:40:27.5420131Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5465240Z Entering 'third_party/gloo' 2024-08-21T00:40:27.5507945Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5551246Z Entering 'third_party/googletest' 2024-08-21T00:40:27.5594938Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5639379Z Entering 'third_party/ideep' 2024-08-21T00:40:27.5687138Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5729386Z Entering 'third_party/ideep/mkl-dnn' 2024-08-21T00:40:27.5771768Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5822987Z Entering 'third_party/ittapi' 2024-08-21T00:40:27.5866573Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5909900Z Entering 'third_party/kineto' 2024-08-21T00:40:27.5954829Z http.https://github.com/.extraheader 2024-08-21T00:40:27.5998235Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-21T00:40:27.6040752Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6084425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-21T00:40:27.6127133Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6172859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-21T00:40:27.6216307Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6261140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-21T00:40:27.6304243Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6350013Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-21T00:40:27.6393130Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6436030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-21T00:40:27.6479007Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6526177Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-21T00:40:27.6570980Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6614801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-21T00:40:27.6659187Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6704153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-21T00:40:27.6747342Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6793170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-21T00:40:27.6837481Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6886255Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-21T00:40:27.6933436Z http.https://github.com/.extraheader 2024-08-21T00:40:27.6976537Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-21T00:40:27.7019397Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7066971Z Entering 'third_party/mimalloc' 2024-08-21T00:40:27.7112215Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7159935Z Entering 'third_party/nccl/nccl' 2024-08-21T00:40:27.7204873Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7250328Z Entering 'third_party/nlohmann' 2024-08-21T00:40:27.7293789Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7339069Z Entering 'third_party/onnx' 2024-08-21T00:40:27.7383159Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7447221Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-21T00:40:27.7490282Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7534308Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-21T00:40:27.7577845Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7625266Z Entering 'third_party/opentelemetry-cpp' 2024-08-21T00:40:27.7669744Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7714603Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-21T00:40:27.7761282Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7804320Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-21T00:40:27.7846735Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7889425Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-21T00:40:27.7931364Z http.https://github.com/.extraheader 2024-08-21T00:40:27.7979335Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-21T00:40:27.8021442Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8065559Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-21T00:40:27.8112483Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8156345Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-21T00:40:27.8203242Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8244981Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-21T00:40:27.8292333Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8333448Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-21T00:40:27.8376546Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8423221Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-21T00:40:27.8466340Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8512954Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-21T00:40:27.8561624Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8625451Z Entering 'third_party/pocketfft' 2024-08-21T00:40:27.8670197Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8714164Z Entering 'third_party/protobuf' 2024-08-21T00:40:27.8758455Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8804785Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-21T00:40:27.8847862Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8890131Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-21T00:40:27.8932948Z http.https://github.com/.extraheader 2024-08-21T00:40:27.8980120Z Entering 'third_party/psimd' 2024-08-21T00:40:27.9024126Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9072309Z Entering 'third_party/pthreadpool' 2024-08-21T00:40:27.9116589Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9163739Z Entering 'third_party/pybind11' 2024-08-21T00:40:27.9207584Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9251454Z Entering 'third_party/python-peachpy' 2024-08-21T00:40:27.9295682Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9340666Z Entering 'third_party/sleef' 2024-08-21T00:40:27.9385638Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9429914Z Entering 'third_party/tensorpipe' 2024-08-21T00:40:27.9474441Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9517586Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-21T00:40:27.9563742Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9605853Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-21T00:40:27.9653550Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9695338Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-21T00:40:27.9737702Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9782213Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-21T00:40:27.9824897Z http.https://github.com/.extraheader 2024-08-21T00:40:27.9867778Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-21T00:40:27.9911089Z http.https://github.com/.extraheader 2024-08-21T00:40:28.0072921Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-21T00:40:28.0094769Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-21T00:40:28.0103259Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-21T00:40:28.0103773Z ##[endgroup] 2024-08-21T00:40:34.9378974Z Cleaning up orphan processes