2024-08-20T22:05:01.8913125Z Current runner version: '2.319.1' 2024-08-20T22:05:01.8919363Z Runner name: 'i-0f9bea9b4b5dbaef9' 2024-08-20T22:05:01.8920172Z Runner group name: 'Default' 2024-08-20T22:05:01.8921111Z Machine name: 'ip-10-0-78-138' 2024-08-20T22:05:01.8938182Z Testing runner upgrade compatibility 2024-08-20T22:05:02.0872041Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:05:02.0874029Z Actions: read 2024-08-20T22:05:02.0874530Z Attestations: read 2024-08-20T22:05:02.0874990Z Checks: read 2024-08-20T22:05:02.0875396Z Contents: read 2024-08-20T22:05:02.0875868Z Deployments: read 2024-08-20T22:05:02.0876302Z Discussions: read 2024-08-20T22:05:02.0876738Z Issues: read 2024-08-20T22:05:02.0877197Z Metadata: read 2024-08-20T22:05:02.0877592Z Packages: read 2024-08-20T22:05:02.0878128Z Pages: read 2024-08-20T22:05:02.0878586Z PullRequests: read 2024-08-20T22:05:02.0879046Z RepositoryProjects: read 2024-08-20T22:05:02.0879475Z SecurityEvents: read 2024-08-20T22:05:02.0879978Z Statuses: read 2024-08-20T22:05:02.0880567Z ##[endgroup] 2024-08-20T22:05:02.0883530Z Secret source: Actions 2024-08-20T22:05:02.0884062Z Prepare workflow directory 2024-08-20T22:05:02.1796924Z Prepare all required actions 2024-08-20T22:05:02.1962417Z Getting action download info 2024-08-20T22:05:02.3657744Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:05:02.8506280Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:05:06.9664894Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:05:07.1668145Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:05:07.4788125Z Getting action download info 2024-08-20T22:05:07.5666819Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:05:07.7789999Z Getting action download info 2024-08-20T22:05:07.8759559Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:05:08.0525747Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:05:08.0528374Z ##[group] Inputs 2024-08-20T22:05:08.0528902Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:05:08.0535916Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:05:08.0543784Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:05:08.0544958Z sync-tag: 2024-08-20T22:05:08.0545766Z timeout-minutes: 240 2024-08-20T22:05:08.0546099Z use-gha: 2024-08-20T22:05:08.0546379Z dashboard-tag: 2024-08-20T22:05:08.0546690Z s3-bucket: gha-artifacts 2024-08-20T22:05:08.0547040Z aws-role-to-assume: 2024-08-20T22:05:08.0547374Z ##[endgroup] 2024-08-20T22:05:08.0548398Z Complete job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:05:08.1266050Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:05:08.1410396Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:05:08.1422152Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:08.1422702Z ##[endgroup] 2024-08-20T22:05:09.3216674Z Runner Type: linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:05:09.3217233Z Instance Type: g5.4xlarge 2024-08-20T22:05:09.3217917Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:05:09.3218492Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:05:15.1121628Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:05:15.1122170Z with: 2024-08-20T22:05:15.1122874Z github-secret: *** 2024-08-20T22:05:15.1123801Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:15.1124779Z activate-with-label: false 2024-08-20T22:05:15.1125140Z label: with-ssh 2024-08-20T22:05:15.1125459Z remove-existing-keys: true 2024-08-20T22:05:15.1125820Z fail-silently: true 2024-08-20T22:05:15.1126580Z env: 2024-08-20T22:05:15.1126847Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:15.1127209Z ##[endgroup] 2024-08-20T22:05:15.2018767Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:05:15.2021605Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:05:15.5096951Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:05:15.5949917Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:05:15.5950545Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:05:15.6667068Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:05:15.6683166Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:05:15.6707172Z Login using: ssh ec2-user@ec2-44-200-86-2.compute-1.amazonaws.com 2024-08-20T22:05:15.6707932Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:05:15.6708738Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:05:15.6840168Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:05:15.6840726Z with: 2024-08-20T22:05:15.6841010Z submodules: recursive 2024-08-20T22:05:15.6841345Z fetch-depth: 0 2024-08-20T22:05:15.6841644Z env: 2024-08-20T22:05:15.6841920Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:15.6842258Z ##[endgroup] 2024-08-20T22:05:15.7040705Z ##[group]Run retry () { 2024-08-20T22:05:15.7041093Z retry () { 2024-08-20T22:05:15.7041610Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:05:15.7042163Z } 2024-08-20T22:05:15.7042475Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:05:15.7042904Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:05:15.7043395Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:15.7043857Z else 2024-08-20T22:05:15.7044196Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:05:15.7044626Z fi 2024-08-20T22:05:15.7045164Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:05:15.7058167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:05:15.7058676Z env: 2024-08-20T22:05:15.7058955Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:15.7059298Z NO_SUDO: 2024-08-20T22:05:15.7059573Z ##[endgroup] 2024-08-20T22:05:15.7092902Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:15.9068178Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:05:15.9068622Z with: 2024-08-20T22:05:15.9068949Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:05:15.9069395Z fetch-depth: 0 2024-08-20T22:05:15.9069714Z submodules: recursive 2024-08-20T22:05:15.9070054Z quiet-checkout: true 2024-08-20T22:05:15.9070412Z repository: pytorch/pytorch 2024-08-20T22:05:15.9070892Z token: *** 2024-08-20T22:05:15.9071187Z ssh-strict: true 2024-08-20T22:05:15.9071512Z persist-credentials: true 2024-08-20T22:05:15.9071875Z clean: true 2024-08-20T22:05:15.9072210Z sparse-checkout-cone-mode: true 2024-08-20T22:05:15.9072607Z lfs: false 2024-08-20T22:05:15.9072905Z set-safe-directory: true 2024-08-20T22:05:15.9073247Z env: 2024-08-20T22:05:15.9073523Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:05:15.9073988Z ##[endgroup] 2024-08-20T22:05:16.0051608Z Syncing repository: pytorch/pytorch 2024-08-20T22:05:16.0053092Z ##[group]Getting Git version info 2024-08-20T22:05:16.0053829Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:16.0054846Z [command]/usr/bin/git version 2024-08-20T22:05:16.0055210Z git version 2.40.1 2024-08-20T22:05:16.0057451Z ##[endgroup] 2024-08-20T22:05:16.0073022Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/f4d8b605-fd28-467e-a9a3-f77cf3fd5281' before making global git config changes 2024-08-20T22:05:16.0074267Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:05:16.0075911Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:16.0128668Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:05:16.0132311Z ##[group]Initializing the repository 2024-08-20T22:05:16.0135119Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:05:16.0178285Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:05:16.0179256Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:05:16.0180060Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:05:16.0180624Z hint: 2024-08-20T22:05:16.0181075Z hint: git config --global init.defaultBranch 2024-08-20T22:05:16.0181543Z hint: 2024-08-20T22:05:16.0182062Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:05:16.0182916Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:05:16.0183778Z hint: 2024-08-20T22:05:16.0184112Z hint: git branch -m 2024-08-20T22:05:16.0185056Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:05:16.0193103Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:05:16.0235402Z ##[endgroup] 2024-08-20T22:05:16.0236002Z ##[group]Disabling automatic garbage collection 2024-08-20T22:05:16.0238574Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:05:16.0279749Z ##[endgroup] 2024-08-20T22:05:16.0280550Z ##[group]Setting up auth 2024-08-20T22:05:16.0286954Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:05:16.0329214Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:05:16.0696528Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:05:16.0739405Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:05:16.1092757Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:05:16.1149398Z ##[endgroup] 2024-08-20T22:05:16.1150214Z ##[group]Fetching the repository 2024-08-20T22:05:16.1156955Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:05:19.0556932Z remote: Enumerating objects: 1008509 2024-08-20T22:05:19.0557496Z remote: Enumerating objects: 1010208, done. 2024-08-20T22:05:19.0559652Z remote: Counting objects: 0% (1/1699) 2024-08-20T22:05:19.0560274Z remote: Counting objects: 1% (17/1699) 2024-08-20T22:05:19.0560817Z remote: Counting objects: 2% (34/1699) 2024-08-20T22:05:19.0561326Z remote: Counting objects: 3% (51/1699) 2024-08-20T22:05:19.0562158Z remote: Counting objects: 4% (68/1699) 2024-08-20T22:05:19.0562867Z remote: Counting objects: 5% (85/1699) 2024-08-20T22:05:19.0563543Z remote: Counting objects: 6% (102/1699) 2024-08-20T22:05:19.0564067Z remote: Counting objects: 7% (119/1699) 2024-08-20T22:05:19.0564569Z remote: Counting objects: 8% (136/1699) 2024-08-20T22:05:19.0565080Z remote: Counting objects: 9% (153/1699) 2024-08-20T22:05:19.0565594Z remote: Counting objects: 10% (170/1699) 2024-08-20T22:05:19.0566175Z remote: Counting objects: 11% (187/1699) 2024-08-20T22:05:19.0566842Z remote: Counting objects: 12% (204/1699) 2024-08-20T22:05:19.0567492Z remote: Counting objects: 13% (221/1699) 2024-08-20T22:05:19.0568175Z remote: Counting objects: 14% (238/1699) 2024-08-20T22:05:19.0568855Z remote: Counting objects: 15% (255/1699) 2024-08-20T22:05:19.0569593Z remote: Counting objects: 16% (272/1699) 2024-08-20T22:05:19.0570346Z remote: Counting objects: 17% (289/1699) 2024-08-20T22:05:19.0570985Z remote: Counting objects: 18% (306/1699) 2024-08-20T22:05:19.0571723Z remote: Counting objects: 19% (323/1699) 2024-08-20T22:05:19.0572507Z remote: Counting objects: 20% (340/1699) 2024-08-20T22:05:19.0573090Z remote: Counting objects: 21% (357/1699) 2024-08-20T22:05:19.0573663Z remote: Counting objects: 22% (374/1699) 2024-08-20T22:05:19.0574393Z remote: Counting objects: 23% (391/1699) 2024-08-20T22:05:19.0575157Z remote: Counting objects: 24% (408/1699) 2024-08-20T22:05:19.0575900Z remote: Counting objects: 25% (425/1699) 2024-08-20T22:05:19.0576472Z remote: Counting objects: 26% (442/1699) 2024-08-20T22:05:19.0576989Z remote: Counting objects: 27% (459/1699) 2024-08-20T22:05:19.0578063Z remote: Counting objects: 28% (476/1699) 2024-08-20T22:05:19.0578865Z remote: Counting objects: 29% (493/1699) 2024-08-20T22:05:19.0579646Z remote: Counting objects: 30% (510/1699) 2024-08-20T22:05:19.0580180Z remote: Counting objects: 31% (527/1699) 2024-08-20T22:05:19.0580830Z remote: Counting objects: 32% (544/1699) 2024-08-20T22:05:19.0581492Z remote: Counting objects: 33% (561/1699) 2024-08-20T22:05:19.0582197Z remote: Counting objects: 34% (578/1699) 2024-08-20T22:05:19.0582713Z remote: Counting objects: 35% (595/1699) 2024-08-20T22:05:19.0583214Z remote: Counting objects: 36% (612/1699) 2024-08-20T22:05:19.0583726Z remote: Counting objects: 37% (629/1699) 2024-08-20T22:05:19.0584237Z remote: Counting objects: 38% (646/1699) 2024-08-20T22:05:19.0584743Z remote: Counting objects: 39% (663/1699) 2024-08-20T22:05:19.0585245Z remote: Counting objects: 40% (680/1699) 2024-08-20T22:05:19.0585977Z remote: Counting objects: 41% (697/1699) 2024-08-20T22:05:19.0586480Z remote: Counting objects: 42% (714/1699) 2024-08-20T22:05:19.0586982Z remote: Counting objects: 43% (731/1699) 2024-08-20T22:05:19.0587479Z remote: Counting objects: 44% (748/1699) 2024-08-20T22:05:19.0587993Z remote: Counting objects: 45% (765/1699) 2024-08-20T22:05:19.0588502Z remote: Counting objects: 46% (782/1699) 2024-08-20T22:05:19.0589008Z remote: Counting objects: 47% (799/1699) 2024-08-20T22:05:19.0589508Z remote: Counting objects: 48% (816/1699) 2024-08-20T22:05:19.0590009Z remote: Counting objects: 49% (833/1699) 2024-08-20T22:05:19.0590512Z remote: Counting objects: 50% (850/1699) 2024-08-20T22:05:19.0591039Z remote: Counting objects: 51% (867/1699) 2024-08-20T22:05:19.0591560Z remote: Counting objects: 52% (884/1699) 2024-08-20T22:05:19.0592063Z remote: Counting objects: 53% (901/1699) 2024-08-20T22:05:19.0592589Z remote: Counting objects: 54% (918/1699) 2024-08-20T22:05:19.0593116Z remote: Counting objects: 55% (935/1699) 2024-08-20T22:05:19.0593614Z remote: Counting objects: 56% (952/1699) 2024-08-20T22:05:19.0594112Z remote: Counting objects: 57% (969/1699) 2024-08-20T22:05:19.0594624Z remote: Counting objects: 58% (986/1699) 2024-08-20T22:05:19.0595145Z remote: Counting objects: 59% (1003/1699) 2024-08-20T22:05:19.0595684Z remote: Counting objects: 60% (1020/1699) 2024-08-20T22:05:19.0596204Z remote: Counting objects: 61% (1037/1699) 2024-08-20T22:05:19.0596734Z remote: Counting objects: 62% (1054/1699) 2024-08-20T22:05:19.0597273Z remote: Counting objects: 63% (1071/1699) 2024-08-20T22:05:19.0597899Z remote: Counting objects: 64% (1088/1699) 2024-08-20T22:05:19.0598435Z remote: Counting objects: 65% (1105/1699) 2024-08-20T22:05:19.0598960Z remote: Counting objects: 66% (1122/1699) 2024-08-20T22:05:19.0599493Z remote: Counting objects: 67% (1139/1699) 2024-08-20T22:05:19.0600019Z remote: Counting objects: 68% (1156/1699) 2024-08-20T22:05:19.0600546Z remote: Counting objects: 69% (1173/1699) 2024-08-20T22:05:19.0601068Z remote: Counting objects: 70% (1190/1699) 2024-08-20T22:05:19.0601595Z remote: Counting objects: 71% (1207/1699) 2024-08-20T22:05:19.0602120Z remote: Counting objects: 72% (1224/1699) 2024-08-20T22:05:19.0602658Z remote: Counting objects: 73% (1241/1699) 2024-08-20T22:05:19.0603180Z remote: Counting objects: 74% (1258/1699) 2024-08-20T22:05:19.0603711Z remote: Counting objects: 75% (1275/1699) 2024-08-20T22:05:19.0604236Z remote: Counting objects: 76% (1292/1699) 2024-08-20T22:05:19.0604762Z remote: Counting objects: 77% (1309/1699) 2024-08-20T22:05:19.0605279Z remote: Counting objects: 78% (1326/1699) 2024-08-20T22:05:19.0605900Z remote: Counting objects: 79% (1343/1699) 2024-08-20T22:05:19.0606432Z remote: Counting objects: 80% (1360/1699) 2024-08-20T22:05:19.0606948Z remote: Counting objects: 81% (1377/1699) 2024-08-20T22:05:19.0607476Z remote: Counting objects: 82% (1394/1699) 2024-08-20T22:05:19.0608001Z remote: Counting objects: 83% (1411/1699) 2024-08-20T22:05:19.0608530Z remote: Counting objects: 84% (1428/1699) 2024-08-20T22:05:19.0609053Z remote: Counting objects: 85% (1445/1699) 2024-08-20T22:05:19.0609578Z remote: Counting objects: 86% (1462/1699) 2024-08-20T22:05:19.0610106Z remote: Counting objects: 87% (1479/1699) 2024-08-20T22:05:19.0610629Z remote: Counting objects: 88% (1496/1699) 2024-08-20T22:05:19.0611147Z remote: Counting objects: 89% (1513/1699) 2024-08-20T22:05:19.0611676Z remote: Counting objects: 90% (1530/1699) 2024-08-20T22:05:19.0612198Z remote: Counting objects: 91% (1547/1699) 2024-08-20T22:05:19.0612823Z remote: Counting objects: 92% (1564/1699) 2024-08-20T22:05:19.0613338Z remote: Counting objects: 93% (1581/1699) 2024-08-20T22:05:19.0613865Z remote: Counting objects: 94% (1598/1699) 2024-08-20T22:05:19.0614390Z remote: Counting objects: 95% (1615/1699) 2024-08-20T22:05:19.0614921Z remote: Counting objects: 96% (1632/1699) 2024-08-20T22:05:19.0615457Z remote: Counting objects: 97% (1649/1699) 2024-08-20T22:05:19.0615996Z remote: Counting objects: 98% (1666/1699) 2024-08-20T22:05:19.0616534Z remote: Counting objects: 99% (1683/1699) 2024-08-20T22:05:19.0617064Z remote: Counting objects: 100% (1699/1699) 2024-08-20T22:05:19.0617635Z remote: Counting objects: 100% (1699/1699), done. 2024-08-20T22:05:19.0736189Z remote: Compressing objects: 0% (1/826) 2024-08-20T22:05:19.0997336Z remote: Compressing objects: 1% (9/826) 2024-08-20T22:05:19.1316168Z remote: Compressing objects: 2% (17/826) 2024-08-20T22:05:19.1856473Z remote: Compressing objects: 3% (25/826) 2024-08-20T22:05:19.2763192Z remote: Compressing objects: 4% (34/826) 2024-08-20T22:05:19.3189029Z remote: Compressing objects: 5% (42/826) 2024-08-20T22:05:19.3862903Z remote: Compressing objects: 6% (50/826) 2024-08-20T22:05:19.4294319Z remote: Compressing objects: 7% (58/826) 2024-08-20T22:05:19.4583343Z remote: Compressing objects: 8% (67/826) 2024-08-20T22:05:19.4818849Z remote: Compressing objects: 9% (75/826) 2024-08-20T22:05:19.5075796Z remote: Compressing objects: 10% (83/826) 2024-08-20T22:05:19.5301865Z remote: Compressing objects: 11% (91/826) 2024-08-20T22:05:19.5513469Z remote: Compressing objects: 12% (100/826) 2024-08-20T22:05:19.5677283Z remote: Compressing objects: 13% (108/826) 2024-08-20T22:05:19.5815996Z remote: Compressing objects: 14% (116/826) 2024-08-20T22:05:19.5913255Z remote: Compressing objects: 15% (124/826) 2024-08-20T22:05:19.6010536Z remote: Compressing objects: 16% (133/826) 2024-08-20T22:05:19.6080252Z remote: Compressing objects: 17% (141/826) 2024-08-20T22:05:19.6121823Z remote: Compressing objects: 18% (149/826) 2024-08-20T22:05:19.6172329Z remote: Compressing objects: 19% (157/826) 2024-08-20T22:05:19.6203567Z remote: Compressing objects: 20% (166/826) 2024-08-20T22:05:19.6225344Z remote: Compressing objects: 21% (174/826) 2024-08-20T22:05:19.6231282Z remote: Compressing objects: 22% (182/826) 2024-08-20T22:05:19.6237278Z remote: Compressing objects: 23% (190/826) 2024-08-20T22:05:19.6245389Z remote: Compressing objects: 24% (199/826) 2024-08-20T22:05:19.6250403Z remote: Compressing objects: 25% (207/826) 2024-08-20T22:05:19.6262852Z remote: Compressing objects: 26% (215/826) 2024-08-20T22:05:19.6272887Z remote: Compressing objects: 27% (224/826) 2024-08-20T22:05:19.6302606Z remote: Compressing objects: 28% (232/826) 2024-08-20T22:05:19.6307571Z remote: Compressing objects: 29% (240/826) 2024-08-20T22:05:19.6320015Z remote: Compressing objects: 30% (248/826) 2024-08-20T22:05:19.6329658Z remote: Compressing objects: 31% (257/826) 2024-08-20T22:05:19.6335455Z remote: Compressing objects: 32% (265/826) 2024-08-20T22:05:19.6344497Z remote: Compressing objects: 33% (273/826) 2024-08-20T22:05:19.6352437Z remote: Compressing objects: 34% (281/826) 2024-08-20T22:05:19.6359656Z remote: Compressing objects: 35% (290/826) 2024-08-20T22:05:19.6365839Z remote: Compressing objects: 36% (298/826) 2024-08-20T22:05:19.6377614Z remote: Compressing objects: 37% (306/826) 2024-08-20T22:05:19.6385812Z remote: Compressing objects: 38% (314/826) 2024-08-20T22:05:19.6393749Z remote: Compressing objects: 39% (323/826) 2024-08-20T22:05:19.6401369Z remote: Compressing objects: 40% (331/826) 2024-08-20T22:05:19.6406863Z remote: Compressing objects: 41% (339/826) 2024-08-20T22:05:19.6417250Z remote: Compressing objects: 42% (347/826) 2024-08-20T22:05:19.6420975Z remote: Compressing objects: 43% (356/826) 2024-08-20T22:05:19.6428780Z remote: Compressing objects: 44% (364/826) 2024-08-20T22:05:19.6432604Z remote: Compressing objects: 45% (372/826) 2024-08-20T22:05:19.6438242Z remote: Compressing objects: 46% (380/826) 2024-08-20T22:05:19.6441641Z remote: Compressing objects: 47% (389/826) 2024-08-20T22:05:19.6444383Z remote: Compressing objects: 48% (397/826) 2024-08-20T22:05:19.6448142Z remote: Compressing objects: 49% (405/826) 2024-08-20T22:05:19.6450574Z remote: Compressing objects: 50% (413/826) 2024-08-20T22:05:19.6453842Z remote: Compressing objects: 51% (422/826) 2024-08-20T22:05:19.6456033Z remote: Compressing objects: 52% (430/826) 2024-08-20T22:05:19.6458134Z remote: Compressing objects: 53% (438/826) 2024-08-20T22:05:19.6459363Z remote: Compressing objects: 54% (447/826) 2024-08-20T22:05:19.6460802Z remote: Compressing objects: 55% (455/826) 2024-08-20T22:05:19.6462079Z remote: Compressing objects: 56% (463/826) 2024-08-20T22:05:19.6463439Z remote: Compressing objects: 57% (471/826) 2024-08-20T22:05:19.6465064Z remote: Compressing objects: 58% (480/826) 2024-08-20T22:05:19.6465693Z remote: Compressing objects: 59% (488/826) 2024-08-20T22:05:19.6466325Z remote: Compressing objects: 60% (496/826) 2024-08-20T22:05:19.6466938Z remote: Compressing objects: 61% (504/826) 2024-08-20T22:05:19.6467769Z remote: Compressing objects: 62% (513/826) 2024-08-20T22:05:19.6468348Z remote: Compressing objects: 63% (521/826) 2024-08-20T22:05:19.6469338Z remote: Compressing objects: 64% (529/826) 2024-08-20T22:05:19.6476361Z remote: Compressing objects: 65% (537/826) 2024-08-20T22:05:19.6481786Z remote: Compressing objects: 66% (546/826) 2024-08-20T22:05:19.6486338Z remote: Compressing objects: 67% (554/826) 2024-08-20T22:05:19.6490164Z remote: Compressing objects: 68% (562/826) 2024-08-20T22:05:19.6494800Z remote: Compressing objects: 69% (570/826) 2024-08-20T22:05:19.6499210Z remote: Compressing objects: 70% (579/826) 2024-08-20T22:05:19.6501836Z remote: Compressing objects: 71% (587/826) 2024-08-20T22:05:19.6504683Z remote: Compressing objects: 72% (595/826) 2024-08-20T22:05:19.6509049Z remote: Compressing objects: 73% (603/826) 2024-08-20T22:05:19.6511524Z remote: Compressing objects: 74% (612/826) 2024-08-20T22:05:19.6514747Z remote: Compressing objects: 75% (620/826) 2024-08-20T22:05:19.6517752Z remote: Compressing objects: 76% (628/826) 2024-08-20T22:05:19.6520376Z remote: Compressing objects: 77% (637/826) 2024-08-20T22:05:19.6522069Z remote: Compressing objects: 78% (645/826) 2024-08-20T22:05:19.6524060Z remote: Compressing objects: 79% (653/826) 2024-08-20T22:05:19.6526818Z remote: Compressing objects: 80% (661/826) 2024-08-20T22:05:19.6528811Z remote: Compressing objects: 81% (670/826) 2024-08-20T22:05:19.6530096Z remote: Compressing objects: 82% (678/826) 2024-08-20T22:05:19.6532863Z remote: Compressing objects: 83% (686/826) 2024-08-20T22:05:19.6534080Z remote: Compressing objects: 84% (694/826) 2024-08-20T22:05:19.6536086Z remote: Compressing objects: 85% (703/826) 2024-08-20T22:05:19.6537617Z remote: Compressing objects: 86% (711/826) 2024-08-20T22:05:19.6539513Z remote: Compressing objects: 87% (719/826) 2024-08-20T22:05:19.6541882Z remote: Compressing objects: 88% (727/826) 2024-08-20T22:05:19.6544263Z remote: Compressing objects: 89% (736/826) 2024-08-20T22:05:19.6545096Z remote: Compressing objects: 90% (744/826) 2024-08-20T22:05:19.6546877Z remote: Compressing objects: 91% (752/826) 2024-08-20T22:05:19.6547768Z remote: Compressing objects: 92% (760/826) 2024-08-20T22:05:19.6548381Z remote: Compressing objects: 93% (769/826) 2024-08-20T22:05:19.6549205Z remote: Compressing objects: 94% (777/826) 2024-08-20T22:05:19.6551040Z remote: Compressing objects: 95% (785/826) 2024-08-20T22:05:19.6552941Z remote: Compressing objects: 96% (793/826) 2024-08-20T22:05:19.6553504Z remote: Compressing objects: 97% (802/826) 2024-08-20T22:05:19.6554788Z remote: Compressing objects: 98% (810/826) 2024-08-20T22:05:19.6556644Z remote: Compressing objects: 99% (818/826) 2024-08-20T22:05:19.6557279Z remote: Compressing objects: 100% (826/826) 2024-08-20T22:05:19.6557913Z remote: Compressing objects: 100% (826/826), done. 2024-08-20T22:05:43.2134119Z remote: Total 1010208 (delta 1106), reused 1335 (delta 870), pack-reused 1008509 (from 1) 2024-08-20T22:06:06.2097706Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:06:06.2135396Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:06:06.2141056Z ##[endgroup] 2024-08-20T22:06:06.2143763Z ##[group]Determining the checkout info 2024-08-20T22:06:06.2145367Z ##[endgroup] 2024-08-20T22:06:06.2145931Z ##[group]Checking out the ref 2024-08-20T22:06:06.2146964Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:06:07.9051882Z ##[endgroup] 2024-08-20T22:06:07.9053165Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:06:07.9056214Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:06:07.9115766Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:06:07.9155417Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:06:07.9196488Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:06:07.9234481Z ##[endgroup] 2024-08-20T22:06:07.9235252Z ##[group]Fetching submodules 2024-08-20T22:06:07.9239312Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:06:07.9625203Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:06:07.9991995Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:06:07.9993580Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:06:07.9997393Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:06:08.0000870Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:06:08.0004979Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:06:08.0007528Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:06:08.0011204Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:06:08.0014840Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:06:08.0018290Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:06:08.0022176Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:06:08.0026260Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:06:08.0032421Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:06:08.0036423Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:06:08.0040754Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:06:08.0044611Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:06:08.0048762Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:06:08.0052722Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:06:08.0058251Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:06:08.0062478Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:06:08.0067419Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:06:08.0071719Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:06:08.0076271Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:06:08.0080959Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:06:08.0086163Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:06:08.0090798Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:06:08.0096138Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:06:08.0100911Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:06:08.0108995Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:06:08.0113967Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:06:08.0119240Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:06:08.0124118Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:06:08.0130121Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:06:08.0134996Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:06:08.0140561Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:06:08.0174212Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:06:08.3169805Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:06:08.5011760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:06:08.6683421Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:06:08.9434855Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:06:10.8715807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:06:22.0745669Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:06:22.4788621Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:06:22.9150056Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:06:23.4985248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:06:24.7167578Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:06:26.8711413Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:06:32.7275771Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:06:34.2462182Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:06:35.9643851Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:06:37.2551800Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:06:37.6766354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:06:38.0321798Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:06:39.1413994Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:06:39.5483415Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:06:39.8701036Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:06:41.4601957Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:06:42.1539862Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:06:42.8506370Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:06:48.9064215Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:06:50.8869923Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:06:55.1200643Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:06:55.3486301Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:07:04.6275697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:07:04.8116576Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:07:05.0131030Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:07:05.9294404Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:07:06.2221306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:07:06.8675361Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:07:07.3078817Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:07:07.3232095Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:07:07.3347985Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:07:07.3659141Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:07:07.4120399Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:07:08.6001065Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:07:08.6291833Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:07:08.6823135Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:07:08.7965740Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:07:08.8371174Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:07:09.4639999Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:07:09.7461373Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:07:09.8414076Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:07:09.8433827Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:07:09.8437330Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:07:09.8440888Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:07:09.8444430Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:07:09.8448126Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:07:09.8480322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:07:10.8508283Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:07:11.8490020Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:07:13.9428220Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:07:14.9807129Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:07:15.3126850Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:07:15.4232993Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:07:15.9384237Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:07:16.0091499Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:07:16.0241866Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:07:16.1783402Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:07:16.2242195Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:07:16.2704563Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:07:16.3035894Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:07:16.3583770Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:07:16.3743950Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:07:16.3763797Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:07:16.3793248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:07:29.5108865Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:07:29.5332475Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:07:29.6345124Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:07:29.6367676Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:07:29.6370238Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:07:29.6374103Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:07:29.6404905Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:07:30.2577677Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:07:31.4603018Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:07:32.6292128Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:07:32.6313155Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:07:32.6316490Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:07:32.6320505Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:07:32.6324341Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:07:32.6328696Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:07:32.6332409Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:07:32.6336780Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:07:32.6340752Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:07:32.6373028Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:07:33.5389874Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:07:33.9091354Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:07:35.1921094Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:07:35.4915461Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:07:36.0074278Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:07:37.0631713Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:07:44.5116459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:07:44.9351908Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:07:44.9588325Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:07:45.0012763Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:07:45.0179909Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:07:45.0198557Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:07:45.0230269Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:07:45.3141506Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:07:45.3370391Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:07:45.3856338Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:07:45.5052188Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:07:45.5253984Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:07:45.5711452Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:07:45.6377217Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:07:45.6842710Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:07:45.7146842Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:07:45.8411478Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:07:46.3551417Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:07:46.3589374Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:07:46.3591858Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:07:46.3623777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:07:46.7966760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:07:47.7607462Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:07:47.8016407Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:07:47.8933049Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:07:47.8956081Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:07:47.8960507Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:07:47.8962635Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:07:47.8965049Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:07:47.8969016Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:07:47.8972704Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:07:47.8976319Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:07:47.8979922Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:07:47.9012445Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:07:48.3112374Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:07:49.3497455Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:07:49.6713640Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:07:55.8113710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:07:56.0855236Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:07:56.2962166Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:07:56.6153121Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:08:03.5641694Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:08:03.6115208Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:08:03.6302486Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:08:03.7540103Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:08:03.7702734Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:08:03.7890115Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:08:03.8094555Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:08:03.8113740Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:03.8117266Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:03.8148134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:08:05.7764650Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:08:07.0895696Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:08:07.1432728Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:08:07.7943716Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:08:07.8089415Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:08:08.1275950Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:08:08.1302330Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:08.1305653Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:08.1337401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:08:08.5737350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:08:09.6454258Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:08:09.7280523Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:08:09.7401107Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:08:09.7553487Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:08:09.7999672Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:08:09.8342740Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:08:09.8839648Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:08:09.9183743Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:08:09.9203246Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:09.9206486Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:09.9210795Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:09.9214012Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:09.9247710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:08:10.9807398Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:08:11.1994011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:08:12.3806598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:08:13.3601819Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:08:13.3798532Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:08:13.4584943Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:08:13.4928070Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:08:13.4947145Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:13.4979587Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:08:13.7141887Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:08:13.7188627Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:08:13.7555747Z Entering 'android/libs/fbjni' 2024-08-20T22:08:13.7607697Z Entering 'third_party/FP16' 2024-08-20T22:08:13.7659260Z Entering 'third_party/FXdiv' 2024-08-20T22:08:13.7710116Z Entering 'third_party/NNPACK' 2024-08-20T22:08:13.7762423Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:13.7813366Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:13.7882733Z Entering 'third_party/benchmark' 2024-08-20T22:08:13.7933742Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:13.7984398Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:13.8036654Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:13.8087736Z Entering 'third_party/cutlass' 2024-08-20T22:08:13.8146717Z Entering 'third_party/eigen' 2024-08-20T22:08:13.8199928Z Entering 'third_party/fbgemm' 2024-08-20T22:08:13.8250963Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:13.8303218Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:13.8355334Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:13.8414138Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:13.8466876Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:13.8519502Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:13.8576887Z Entering 'third_party/fmt' 2024-08-20T22:08:13.8627623Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:13.8679987Z Entering 'third_party/gloo' 2024-08-20T22:08:13.8731396Z Entering 'third_party/googletest' 2024-08-20T22:08:13.8782540Z Entering 'third_party/ideep' 2024-08-20T22:08:13.8832672Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:13.8894726Z Entering 'third_party/ittapi' 2024-08-20T22:08:13.8946117Z Entering 'third_party/kineto' 2024-08-20T22:08:13.8996634Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:13.9053834Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:13.9105614Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:13.9156055Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:13.9208824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:13.9257099Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:13.9311153Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:13.9361217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:13.9410550Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:13.9462525Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:13.9515789Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:13.9568604Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:13.9626038Z Entering 'third_party/mimalloc' 2024-08-20T22:08:13.9678548Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:13.9730029Z Entering 'third_party/nlohmann' 2024-08-20T22:08:13.9782591Z Entering 'third_party/onnx' 2024-08-20T22:08:13.9847908Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:13.9898898Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:13.9955830Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:14.0007705Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:14.0059366Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:14.0110275Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:14.0166804Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:14.0217585Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:14.0267657Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:14.0317090Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:14.0366071Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:14.0419719Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:14.0473372Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:14.0547518Z Entering 'third_party/pocketfft' 2024-08-20T22:08:14.0598029Z Entering 'third_party/protobuf' 2024-08-20T22:08:14.0652849Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:14.0701935Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:14.0754538Z Entering 'third_party/psimd' 2024-08-20T22:08:14.0804688Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:14.0854650Z Entering 'third_party/pybind11' 2024-08-20T22:08:14.0904550Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:14.0954513Z Entering 'third_party/sleef' 2024-08-20T22:08:14.1004831Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:14.1055605Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:14.1105527Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:14.1157151Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:14.1207967Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:14.1256765Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:14.1334061Z ##[endgroup] 2024-08-20T22:08:14.1335211Z ##[group]Persisting credentials for submodules 2024-08-20T22:08:14.1340388Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:08:14.1711464Z Entering 'android/libs/fbjni' 2024-08-20T22:08:14.1781138Z Entering 'third_party/FP16' 2024-08-20T22:08:14.1850927Z Entering 'third_party/FXdiv' 2024-08-20T22:08:14.1917823Z Entering 'third_party/NNPACK' 2024-08-20T22:08:14.1986785Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:14.2055107Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:14.2138224Z Entering 'third_party/benchmark' 2024-08-20T22:08:14.2205998Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:14.2275692Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:14.2344894Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:14.2413440Z Entering 'third_party/cutlass' 2024-08-20T22:08:14.2490689Z Entering 'third_party/eigen' 2024-08-20T22:08:14.2560868Z Entering 'third_party/fbgemm' 2024-08-20T22:08:14.2626814Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:14.2694040Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:14.2762121Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:14.2835094Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:14.2901119Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:14.2970490Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:14.3042236Z Entering 'third_party/fmt' 2024-08-20T22:08:14.3112871Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:14.3181171Z Entering 'third_party/gloo' 2024-08-20T22:08:14.3249377Z Entering 'third_party/googletest' 2024-08-20T22:08:14.3321895Z Entering 'third_party/ideep' 2024-08-20T22:08:14.3388667Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:14.3464794Z Entering 'third_party/ittapi' 2024-08-20T22:08:14.3533028Z Entering 'third_party/kineto' 2024-08-20T22:08:14.3598222Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:14.3664362Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:14.3732933Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:14.3800704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:14.3874282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:14.3941943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:14.4012699Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:14.4080321Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:14.4148322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:14.4215557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:14.4285901Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:14.4351582Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:14.4420689Z Entering 'third_party/mimalloc' 2024-08-20T22:08:14.4491978Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:14.4561374Z Entering 'third_party/nlohmann' 2024-08-20T22:08:14.4629753Z Entering 'third_party/onnx' 2024-08-20T22:08:14.4718036Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:14.4787022Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:14.4860893Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:14.4927971Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:14.4996084Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:14.5063032Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:14.5129052Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:14.5195965Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:14.5261679Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:14.5327193Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:14.5388817Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:14.5457844Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:14.5530048Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:14.5616779Z Entering 'third_party/pocketfft' 2024-08-20T22:08:14.5683752Z Entering 'third_party/protobuf' 2024-08-20T22:08:14.5754143Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:14.5821596Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:14.5895222Z Entering 'third_party/psimd' 2024-08-20T22:08:14.5962498Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:14.6031218Z Entering 'third_party/pybind11' 2024-08-20T22:08:14.6105544Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:14.6173894Z Entering 'third_party/sleef' 2024-08-20T22:08:14.6241874Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:14.6308592Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:14.6381065Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:14.6446125Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:14.6512409Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:14.6577549Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:14.6671840Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:08:14.7035370Z Entering 'android/libs/fbjni' 2024-08-20T22:08:14.7097652Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:08:14.7119513Z Entering 'third_party/FP16' 2024-08-20T22:08:14.7182189Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:08:14.7203968Z Entering 'third_party/FXdiv' 2024-08-20T22:08:14.7267266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:08:14.7288833Z Entering 'third_party/NNPACK' 2024-08-20T22:08:14.7352608Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:08:14.7375112Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:14.7437359Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:08:14.7459278Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:14.7520931Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:08:14.7558182Z Entering 'third_party/benchmark' 2024-08-20T22:08:14.7619161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:14.7641241Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:14.7702287Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:08:14.7724942Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:14.7788203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:14.7811071Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:14.7874218Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:08:14.7895744Z Entering 'third_party/cutlass' 2024-08-20T22:08:14.7957138Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:14.7985835Z Entering 'third_party/eigen' 2024-08-20T22:08:14.8047904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:08:14.8072003Z Entering 'third_party/fbgemm' 2024-08-20T22:08:14.8135887Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:08:14.8156282Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:14.8217569Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:08:14.8237564Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:14.8298854Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:08:14.8319403Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:14.8379259Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:08:14.8407050Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:14.8467849Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:14.8488453Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:14.8549460Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:08:14.8571502Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:14.8633939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:08:14.8658892Z Entering 'third_party/fmt' 2024-08-20T22:08:14.8721027Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:14.8743045Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:14.8805477Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:08:14.8829052Z Entering 'third_party/gloo' 2024-08-20T22:08:14.8890869Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:08:14.8913775Z Entering 'third_party/googletest' 2024-08-20T22:08:14.8974848Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:14.8997062Z Entering 'third_party/ideep' 2024-08-20T22:08:14.9059170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:08:14.9078749Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:14.9139649Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:08:14.9169204Z Entering 'third_party/ittapi' 2024-08-20T22:08:14.9232385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:08:14.9253907Z Entering 'third_party/kineto' 2024-08-20T22:08:14.9317301Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:08:14.9338047Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:14.9402687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:08:14.9422313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:14.9485721Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:08:14.9507956Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:14.9572392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:08:14.9592999Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:14.9655007Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:08:14.9675398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:14.9739092Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:08:14.9757270Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:14.9820795Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:08:14.9843964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:14.9904934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:08:14.9925144Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:14.9986845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:15.0007340Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:15.0068306Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:08:15.0089951Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:15.0151428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:08:15.0174740Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:15.0237621Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:08:15.0257880Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:15.0319870Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:08:15.0345066Z Entering 'third_party/mimalloc' 2024-08-20T22:08:15.0406550Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:08:15.0428678Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:15.0489731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:08:15.0511203Z Entering 'third_party/nlohmann' 2024-08-20T22:08:15.0572751Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:08:15.0595151Z Entering 'third_party/onnx' 2024-08-20T22:08:15.0657258Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:08:15.0691808Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:15.0757804Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:15.0779190Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:15.0840347Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:15.0866550Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:15.0931714Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:08:15.0952920Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:15.1013515Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:15.1033386Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:15.1093904Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:15.1113512Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:15.1174216Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:08:15.1193880Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:15.1256304Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:08:15.1277846Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:15.1338076Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:08:15.1357805Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:15.1418810Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:08:15.1440416Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:15.1501179Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:08:15.1519017Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:15.1581952Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:08:15.1602554Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:15.1664161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:08:15.1687091Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:15.1747850Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:08:15.1788348Z Entering 'third_party/pocketfft' 2024-08-20T22:08:15.1851267Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:08:15.1872605Z Entering 'third_party/protobuf' 2024-08-20T22:08:15.1935391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:08:15.1958442Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:15.2019165Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:08:15.2043562Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:15.2104211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:15.2128594Z Entering 'third_party/psimd' 2024-08-20T22:08:15.2190383Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:08:15.2211674Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:15.2275570Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:08:15.2297027Z Entering 'third_party/pybind11' 2024-08-20T22:08:15.2359312Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:15.2381167Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:15.2443633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:08:15.2465202Z Entering 'third_party/sleef' 2024-08-20T22:08:15.2527889Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:08:15.2550084Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:15.2613136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:08:15.2633761Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:15.2696841Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:08:15.2716517Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:15.2777517Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:08:15.2797226Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:15.2857934Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:08:15.2882325Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:15.2943435Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:08:15.2961259Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:15.3022600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:08:15.3708630Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:08:15.4075796Z Entering 'android/libs/fbjni' 2024-08-20T22:08:15.4127037Z Entering 'third_party/FP16' 2024-08-20T22:08:15.4179987Z Entering 'third_party/FXdiv' 2024-08-20T22:08:15.4232459Z Entering 'third_party/NNPACK' 2024-08-20T22:08:15.4285172Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:15.4337278Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:15.4406267Z Entering 'third_party/benchmark' 2024-08-20T22:08:15.4459308Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:15.4511970Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:15.4565163Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:15.4617371Z Entering 'third_party/cutlass' 2024-08-20T22:08:15.4675980Z Entering 'third_party/eigen' 2024-08-20T22:08:15.4730380Z Entering 'third_party/fbgemm' 2024-08-20T22:08:15.4781947Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:15.4836454Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:15.4894612Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:15.4952504Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:15.5003187Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:15.5055379Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:15.5109995Z Entering 'third_party/fmt' 2024-08-20T22:08:15.5162870Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:15.5215443Z Entering 'third_party/gloo' 2024-08-20T22:08:15.5268959Z Entering 'third_party/googletest' 2024-08-20T22:08:15.5322174Z Entering 'third_party/ideep' 2024-08-20T22:08:15.5372682Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:15.5431956Z Entering 'third_party/ittapi' 2024-08-20T22:08:15.5484040Z Entering 'third_party/kineto' 2024-08-20T22:08:15.5538764Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:15.5587182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:15.5640595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:15.5693408Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:15.5747617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:15.5796020Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:15.5850758Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:15.5900673Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:15.5953490Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:15.6007622Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:15.6063455Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:15.6113279Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:15.6167713Z Entering 'third_party/mimalloc' 2024-08-20T22:08:15.6220425Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:15.6273377Z Entering 'third_party/nlohmann' 2024-08-20T22:08:15.6327522Z Entering 'third_party/onnx' 2024-08-20T22:08:15.6393379Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:15.6445392Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:15.6501248Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:15.6554057Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:15.6604871Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:15.6657461Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:15.6708485Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:15.6775289Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:15.6825065Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:15.6876083Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:15.6925317Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:15.6977151Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:15.7029356Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:15.7100930Z Entering 'third_party/pocketfft' 2024-08-20T22:08:15.7153136Z Entering 'third_party/protobuf' 2024-08-20T22:08:15.7205137Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:15.7257139Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:15.7311247Z Entering 'third_party/psimd' 2024-08-20T22:08:15.7362730Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:15.7413678Z Entering 'third_party/pybind11' 2024-08-20T22:08:15.7465553Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:15.7516132Z Entering 'third_party/sleef' 2024-08-20T22:08:15.7567468Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:15.7617109Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:15.7672967Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:15.7722993Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:15.7774854Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:15.7823655Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:15.7899718Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:08:15.8264672Z Entering 'android/libs/fbjni' 2024-08-20T22:08:15.8319735Z Entering 'third_party/FP16' 2024-08-20T22:08:15.8372867Z Entering 'third_party/FXdiv' 2024-08-20T22:08:15.8425484Z Entering 'third_party/NNPACK' 2024-08-20T22:08:15.8481345Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:08:15.8533787Z Entering 'third_party/XNNPACK' 2024-08-20T22:08:15.8600818Z Entering 'third_party/benchmark' 2024-08-20T22:08:15.8652329Z Entering 'third_party/cpp-httplib' 2024-08-20T22:08:15.8703662Z Entering 'third_party/cpuinfo' 2024-08-20T22:08:15.8756914Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:08:15.8808352Z Entering 'third_party/cutlass' 2024-08-20T22:08:15.8867351Z Entering 'third_party/eigen' 2024-08-20T22:08:15.8920427Z Entering 'third_party/fbgemm' 2024-08-20T22:08:15.8971821Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:08:15.9022140Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:08:15.9073370Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:08:15.9130374Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:08:15.9180209Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:08:15.9234040Z Entering 'third_party/flatbuffers' 2024-08-20T22:08:15.9288080Z Entering 'third_party/fmt' 2024-08-20T22:08:15.9339782Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:08:15.9392253Z Entering 'third_party/gloo' 2024-08-20T22:08:15.9445878Z Entering 'third_party/googletest' 2024-08-20T22:08:15.9498003Z Entering 'third_party/ideep' 2024-08-20T22:08:15.9550721Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:08:15.9608735Z Entering 'third_party/ittapi' 2024-08-20T22:08:15.9659970Z Entering 'third_party/kineto' 2024-08-20T22:08:15.9711496Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:08:15.9765546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:08:15.9817000Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:08:15.9866923Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:08:15.9916671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:08:15.9965661Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:08:16.0020117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:08:16.0071252Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:08:16.0121925Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:08:16.0174698Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:08:16.0227200Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:08:16.0278763Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:08:16.0331886Z Entering 'third_party/mimalloc' 2024-08-20T22:08:16.0385853Z Entering 'third_party/nccl/nccl' 2024-08-20T22:08:16.0437879Z Entering 'third_party/nlohmann' 2024-08-20T22:08:16.0490526Z Entering 'third_party/onnx' 2024-08-20T22:08:16.0556902Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:08:16.0612892Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:08:16.0676342Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:08:16.0728307Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:08:16.0779709Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:08:16.0835539Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:08:16.0885551Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:08:16.0940653Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:08:16.0991614Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:08:16.1046395Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:08:16.1094812Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:08:16.1146727Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:08:16.1199189Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:08:16.1271068Z Entering 'third_party/pocketfft' 2024-08-20T22:08:16.1322853Z Entering 'third_party/protobuf' 2024-08-20T22:08:16.1376790Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:08:16.1426825Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:08:16.1485381Z Entering 'third_party/psimd' 2024-08-20T22:08:16.1536886Z Entering 'third_party/pthreadpool' 2024-08-20T22:08:16.1588279Z Entering 'third_party/pybind11' 2024-08-20T22:08:16.1640703Z Entering 'third_party/python-peachpy' 2024-08-20T22:08:16.1692666Z Entering 'third_party/sleef' 2024-08-20T22:08:16.1744708Z Entering 'third_party/tensorpipe' 2024-08-20T22:08:16.1796109Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:08:16.1846640Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:08:16.1897377Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:08:16.1954521Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:08:16.2002771Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:08:16.2075137Z ##[endgroup] 2024-08-20T22:08:16.2124354Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:08:16.2161794Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:08:16.2382403Z Prepare all required actions 2024-08-20T22:08:16.2382876Z Getting action download info 2024-08-20T22:08:16.3648885Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:08:16.3649313Z env: 2024-08-20T22:08:16.3649595Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.3649946Z ##[endgroup] 2024-08-20T22:08:16.3708458Z ##[group]Run set -euo pipefail 2024-08-20T22:08:16.3708898Z set -euo pipefail 2024-08-20T22:08:16.3709295Z function get_ec2_metadata() { 2024-08-20T22:08:16.3709840Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:08:16.3710727Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:08:16.3711489Z  category=$1 2024-08-20T22:08:16.3712023Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:08:16.3712667Z  runner_name_str=i-0f9bea9b4b5dbaef9 2024-08-20T22:08:16.3713171Z  if [[ -f /.inarc ]]; then 2024-08-20T22:08:16.3713668Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:08:16.3714234Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:08:16.3714931Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:08:16.3715623Z  else 2024-08-20T22:08:16.3716177Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:08:16.3716756Z  fi 2024-08-20T22:08:16.3717051Z } 2024-08-20T22:08:16.3717410Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:08:16.3718079Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:08:16.3718746Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:08:16.3719326Z echo "system info $(uname -a)" 2024-08-20T22:08:16.3730907Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.3731417Z env: 2024-08-20T22:08:16.3731709Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.3732065Z ##[endgroup] 2024-08-20T22:08:16.3842043Z ami-id: ami-06c68f701d8090592 2024-08-20T22:08:16.3898559Z instance-id: i-0f9bea9b4b5dbaef9 2024-08-20T22:08:16.3953661Z instance-type: g5.4xlarge 2024-08-20T22:08:16.3967897Z system info Linux ip-10-0-78-138.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:08:16.3991461Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:16.3992381Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:08:16.4001671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.4002162Z env: 2024-08-20T22:08:16.4002442Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.4002791Z ##[endgroup] 2024-08-20T22:08:16.4081302Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:08:16.4081923Z if systemctl is-active --quiet docker; then 2024-08-20T22:08:16.4082469Z  echo "Docker daemon is running..."; 2024-08-20T22:08:16.4082930Z else 2024-08-20T22:08:16.4083428Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:08:16.4084165Z fi 2024-08-20T22:08:16.4092995Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:16.4093496Z env: 2024-08-20T22:08:16.4093780Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.4094123Z ##[endgroup] 2024-08-20T22:08:16.4182332Z Docker daemon is running... 2024-08-20T22:08:16.4243708Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:08:16.4244279Z with: 2024-08-20T22:08:16.4244556Z shell: bash 2024-08-20T22:08:16.4244932Z timeout_minutes: 5 2024-08-20T22:08:16.4245265Z max_attempts: 3 2024-08-20T22:08:16.4245689Z retry_wait_seconds: 30 2024-08-20T22:08:16.4248829Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:08:16.4251852Z polling_interval_seconds: 1 2024-08-20T22:08:16.4252229Z warning_on_retry: true 2024-08-20T22:08:16.4252572Z continue_on_error: false 2024-08-20T22:08:16.4252916Z env: 2024-08-20T22:08:16.4253191Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:16.4253543Z AWS_RETRY_MODE: standard 2024-08-20T22:08:16.4253885Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:08:16.4254234Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:08:16.4254613Z ##[endgroup] 2024-08-20T22:08:17.5830784Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:17.5831871Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:17.5832860Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:17.5833578Z 2024-08-20T22:08:17.5833730Z Login Succeeded 2024-08-20T22:08:18.4824728Z Command completed after 1 attempt(s). 2024-08-20T22:08:18.4890258Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:18.4890982Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:18.4891637Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:08:18.4900731Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:18.4901225Z env: 2024-08-20T22:08:18.4901505Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.4901860Z ##[endgroup] 2024-08-20T22:08:18.5001257Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:18.5002023Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:08:18.5002615Z # shellcheck disable=SC2046 2024-08-20T22:08:18.5003073Z docker stop $(docker ps -q) || true 2024-08-20T22:08:18.5003563Z # Prune all of the docker images 2024-08-20T22:08:18.5004014Z docker system prune -af 2024-08-20T22:08:18.5013062Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:18.5013554Z env: 2024-08-20T22:08:18.5013830Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.5014176Z ##[endgroup] 2024-08-20T22:08:18.5323120Z "docker stop" requires at least 1 argument. 2024-08-20T22:08:18.5323863Z See 'docker stop --help'. 2024-08-20T22:08:18.5324111Z 2024-08-20T22:08:18.5324337Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:08:18.5324708Z 2024-08-20T22:08:18.5324865Z Stop one or more running containers 2024-08-20T22:08:18.5509320Z Total reclaimed space: 0B 2024-08-20T22:08:18.5556285Z ##[group]Run set +e 2024-08-20T22:08:18.5556642Z set +e 2024-08-20T22:08:18.5556947Z set -x 2024-08-20T22:08:18.5557241Z  2024-08-20T22:08:18.5557558Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:18.5558665Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:08:18.5559747Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:08:18.5560507Z # one is returned at random 2024-08-20T22:08:18.5561055Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:08:18.5561569Z  2024-08-20T22:08:18.5561900Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:18.5562524Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:08:18.5563467Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:08:18.5564024Z  2024-08-20T22:08:18.5564361Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:08:18.5564927Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:08:18.5565443Z  exit 1 2024-08-20T22:08:18.5565774Z  fi 2024-08-20T22:08:18.5566068Z fi 2024-08-20T22:08:18.5566358Z  2024-08-20T22:08:18.5566733Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:08:18.5567247Z  # Clean up any old records first 2024-08-20T22:08:18.5567766Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:08:18.5568232Z fi 2024-08-20T22:08:18.5568510Z  2024-08-20T22:08:18.5568956Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:08:18.5569542Z cat /etc/hosts 2024-08-20T22:08:18.5579057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:18.5579588Z env: 2024-08-20T22:08:18.5579945Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.5580372Z ##[endgroup] 2024-08-20T22:08:18.5609763Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:08:18.5616657Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:08:18.5617137Z ++ tail -n1 2024-08-20T22:08:18.6034609Z + RESOLVED_IP=18.160.10.36 2024-08-20T22:08:18.6035130Z + '[' -z 18.160.10.36 ']' 2024-08-20T22:08:18.6035572Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:08:18.6055100Z + echo '18.160.10.36 download.pytorch.org' 2024-08-20T22:08:18.6056025Z + sudo tee -a /etc/hosts 2024-08-20T22:08:18.8293535Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:18.8316107Z + cat /etc/hosts 2024-08-20T22:08:18.8327538Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:08:18.8340083Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:08:18.8340754Z 18.160.10.36 download.pytorch.org 2024-08-20T22:08:18.8494364Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:08:18.8494968Z with: 2024-08-20T22:08:18.8496063Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8497287Z docker-build-dir: .ci/docker 2024-08-20T22:08:18.8497682Z working-directory: . 2024-08-20T22:08:18.8498150Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.8498677Z force-push: false 2024-08-20T22:08:18.8498984Z env: 2024-08-20T22:08:18.8499263Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.8499604Z ##[endgroup] 2024-08-20T22:08:18.8518796Z ##[group]Run set -ex 2024-08-20T22:08:18.8519169Z set -ex 2024-08-20T22:08:18.8519472Z  2024-08-20T22:08:18.8520175Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:08:18.8521396Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:08:18.8522288Z # job could then download the pre-built image as usual 2024-08-20T22:08:18.8523048Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:08:18.8523731Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8524542Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8525108Z  2024-08-20T22:08:18.8525630Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:08:18.8526649Z  exit 0 2024-08-20T22:08:18.8526964Z else 2024-08-20T22:08:18.8527336Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8527797Z fi 2024-08-20T22:08:18.8528092Z  2024-08-20T22:08:18.8528573Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:08:18.8529417Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:08:18.8530205Z  # use it as it is, but first let's extract the tag 2024-08-20T22:08:18.8530920Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:08:18.8531650Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8532358Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8532918Z else 2024-08-20T22:08:18.8533360Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:08:18.8534034Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8534932Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8535675Z fi 2024-08-20T22:08:18.8548676Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:18.8549172Z env: 2024-08-20T22:08:18.8549453Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.8549797Z REPO_NAME: pytorch 2024-08-20T22:08:18.8550981Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8552197Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:18.8552697Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.8553199Z ##[endgroup] 2024-08-20T22:08:18.8583662Z + [[ ! -d .ci/docker ]] 2024-08-20T22:08:18.8584227Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:08:18.8584662Z + echo skip=false 2024-08-20T22:08:18.8586976Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:08:18.8591171Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8592435Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:08:18.8619482Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8620277Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8622206Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8655367Z ##[group]Run set +e 2024-08-20T22:08:18.8655854Z set +e 2024-08-20T22:08:18.8656328Z set -x 2024-08-20T22:08:18.8656675Z  2024-08-20T22:08:18.8657049Z login() { 2024-08-20T22:08:18.8657896Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:18.8658716Z } 2024-08-20T22:08:18.8659085Z  2024-08-20T22:08:18.8659576Z retry () { 2024-08-20T22:08:18.8660035Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:18.8660702Z } 2024-08-20T22:08:18.8661082Z  2024-08-20T22:08:18.8661529Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:18.8662215Z  2024-08-20T22:08:18.8662797Z # Check if image already exists, if it does then skip building it 2024-08-20T22:08:18.8663627Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:08:18.8664826Z  exit 0 2024-08-20T22:08:18.8665221Z fi 2024-08-20T22:08:18.8665719Z  2024-08-20T22:08:18.8666288Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:08:18.8667259Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:08:18.8668165Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:08:18.8669011Z  # if we're on the base branch then use the parent commit 2024-08-20T22:08:18.8669791Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:08:18.8670399Z else 2024-08-20T22:08:18.8670998Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:08:18.8671871Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:08:18.8672478Z fi 2024-08-20T22:08:18.8672908Z  2024-08-20T22:08:18.8673309Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:08:18.8673907Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8674542Z  2024-08-20T22:08:18.8675277Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:08:18.8676099Z  exit 0 2024-08-20T22:08:18.8676563Z fi 2024-08-20T22:08:18.8676943Z  2024-08-20T22:08:18.8677488Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:08:18.8678679Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:08:18.8679572Z  exit 1 2024-08-20T22:08:18.8679959Z fi 2024-08-20T22:08:18.8680416Z  2024-08-20T22:08:18.8680960Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:08:18.8682021Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:08:18.8683022Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:08:18.8684081Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:08:18.8685281Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:08:18.8686193Z fi 2024-08-20T22:08:18.8686598Z  2024-08-20T22:08:18.8687039Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:08:18.8695530Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:18.8696121Z env: 2024-08-20T22:08:18.8696548Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:18.8697023Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:08:18.8697551Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:08:18.8698957Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8700332Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:18.8701104Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.8701726Z ##[endgroup] 2024-08-20T22:08:18.8731206Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.8731959Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:18.8733438Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:18.8735491Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:19.4042436Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:19.4044504Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:19.4046980Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:19.4048068Z 2024-08-20T22:08:19.4048496Z Login Succeeded 2024-08-20T22:08:19.4068015Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:19.6119630Z { 2024-08-20T22:08:19.6120361Z "schemaVersion": 2, 2024-08-20T22:08:19.6121109Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:08:19.6121807Z "config": { 2024-08-20T22:08:19.6122418Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:08:19.6123709Z "size": 48711, 2024-08-20T22:08:19.6124607Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:08:19.6125422Z }, 2024-08-20T22:08:19.6126434Z "layers": [ 2024-08-20T22:08:19.6127025Z { 2024-08-20T22:08:19.6127920Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6128689Z "size": 28580681, 2024-08-20T22:08:19.6129606Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:08:19.6130544Z }, 2024-08-20T22:08:19.6131082Z { 2024-08-20T22:08:19.6131760Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6133517Z "size": 7943451, 2024-08-20T22:08:19.6134277Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:08:19.6134951Z }, 2024-08-20T22:08:19.6135359Z { 2024-08-20T22:08:19.6137190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6138216Z "size": 55728572, 2024-08-20T22:08:19.6138878Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:08:19.6139548Z }, 2024-08-20T22:08:19.6140144Z { 2024-08-20T22:08:19.6141186Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6142191Z "size": 186, 2024-08-20T22:08:19.6143135Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:08:19.6144075Z }, 2024-08-20T22:08:19.6144533Z { 2024-08-20T22:08:19.6145100Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6145738Z "size": 6886, 2024-08-20T22:08:19.6146430Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:08:19.6147181Z }, 2024-08-20T22:08:19.6147821Z { 2024-08-20T22:08:19.6148374Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6149068Z "size": 1329001756, 2024-08-20T22:08:19.6149795Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:08:19.6150648Z }, 2024-08-20T22:08:19.6151115Z { 2024-08-20T22:08:19.6151780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6152406Z "size": 62501, 2024-08-20T22:08:19.6153087Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:08:19.6153923Z }, 2024-08-20T22:08:19.6154464Z { 2024-08-20T22:08:19.6155019Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6155713Z "size": 1684, 2024-08-20T22:08:19.6156293Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:08:19.6157007Z }, 2024-08-20T22:08:19.6157389Z { 2024-08-20T22:08:19.6157995Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6158670Z "size": 1523, 2024-08-20T22:08:19.6159370Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:08:19.6160107Z }, 2024-08-20T22:08:19.6160403Z { 2024-08-20T22:08:19.6160965Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6161794Z "size": 2528295403, 2024-08-20T22:08:19.6162441Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:08:19.6163154Z }, 2024-08-20T22:08:19.6163507Z { 2024-08-20T22:08:19.6164028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6164738Z "size": 86016, 2024-08-20T22:08:19.6165347Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:08:19.6166054Z }, 2024-08-20T22:08:19.6166452Z { 2024-08-20T22:08:19.6166941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6167581Z "size": 1822, 2024-08-20T22:08:19.6168282Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:08:19.6168986Z }, 2024-08-20T22:08:19.6169281Z { 2024-08-20T22:08:19.6169867Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6170504Z "size": 246761563, 2024-08-20T22:08:19.6171195Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:08:19.6171918Z }, 2024-08-20T22:08:19.6172249Z { 2024-08-20T22:08:19.6172792Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6173450Z "size": 544, 2024-08-20T22:08:19.6174058Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:08:19.6174764Z }, 2024-08-20T22:08:19.6175150Z { 2024-08-20T22:08:19.6175681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6176365Z "size": 1258, 2024-08-20T22:08:19.6177059Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:08:19.6177788Z }, 2024-08-20T22:08:19.6178094Z { 2024-08-20T22:08:19.6178680Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6179346Z "size": 484, 2024-08-20T22:08:19.6180037Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:08:19.6180722Z }, 2024-08-20T22:08:19.6181108Z { 2024-08-20T22:08:19.6181682Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6182314Z "size": 91711766, 2024-08-20T22:08:19.6182971Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:08:19.6183715Z }, 2024-08-20T22:08:19.6184073Z { 2024-08-20T22:08:19.6184550Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6185241Z "size": 3390, 2024-08-20T22:08:19.6185997Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:08:19.6186690Z }, 2024-08-20T22:08:19.6187039Z { 2024-08-20T22:08:19.6187579Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6188216Z "size": 1909, 2024-08-20T22:08:19.6188865Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:08:19.6189588Z }, 2024-08-20T22:08:19.6190110Z + exit 0 2024-08-20T22:08:19.6190532Z { 2024-08-20T22:08:19.6191017Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6191678Z "size": 701, 2024-08-20T22:08:19.6192348Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:19.6193057Z }, 2024-08-20T22:08:19.6193376Z { 2024-08-20T22:08:19.6193951Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6194595Z "size": 2864254879, 2024-08-20T22:08:19.6195352Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:08:19.6196026Z }, 2024-08-20T22:08:19.6196365Z { 2024-08-20T22:08:19.6196949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6197554Z "size": 380, 2024-08-20T22:08:19.6198278Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:08:19.6199131Z }, 2024-08-20T22:08:19.6199472Z { 2024-08-20T22:08:19.6199978Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6200673Z "size": 12900, 2024-08-20T22:08:19.6201332Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:08:19.6202039Z }, 2024-08-20T22:08:19.6202419Z { 2024-08-20T22:08:19.6202964Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6203602Z "size": 803, 2024-08-20T22:08:19.6204287Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:08:19.6204961Z }, 2024-08-20T22:08:19.6205293Z { 2024-08-20T22:08:19.6205860Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6206485Z "size": 106, 2024-08-20T22:08:19.6207103Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:08:19.6207891Z }, 2024-08-20T22:08:19.6208218Z { 2024-08-20T22:08:19.6208729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6209456Z "size": 503, 2024-08-20T22:08:19.6210081Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:08:19.6210776Z }, 2024-08-20T22:08:19.6211143Z { 2024-08-20T22:08:19.6211664Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6212311Z "size": 121477267, 2024-08-20T22:08:19.6212982Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:08:19.6213675Z }, 2024-08-20T22:08:19.6214041Z { 2024-08-20T22:08:19.6214606Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6215212Z "size": 109, 2024-08-20T22:08:19.6215890Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:08:19.6216650Z }, 2024-08-20T22:08:19.6217001Z { 2024-08-20T22:08:19.6217518Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6218222Z "size": 490, 2024-08-20T22:08:19.6218847Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:08:19.6219608Z }, 2024-08-20T22:08:19.6219914Z { 2024-08-20T22:08:19.6220454Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6221193Z "size": 297, 2024-08-20T22:08:19.6221804Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:08:19.6222655Z }, 2024-08-20T22:08:19.6223066Z { 2024-08-20T22:08:19.6223615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6224219Z "size": 103, 2024-08-20T22:08:19.6224910Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:08:19.6225655Z }, 2024-08-20T22:08:19.6226281Z { 2024-08-20T22:08:19.6226853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6227532Z "size": 1473, 2024-08-20T22:08:19.6228177Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:08:19.6228930Z }, 2024-08-20T22:08:19.6229258Z { 2024-08-20T22:08:19.6229819Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6230527Z "size": 594729516, 2024-08-20T22:08:19.6231194Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:08:19.6231963Z }, 2024-08-20T22:08:19.6232380Z { 2024-08-20T22:08:19.6232904Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6233504Z "size": 164, 2024-08-20T22:08:19.6234210Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:08:19.6234913Z }, 2024-08-20T22:08:19.6235289Z { 2024-08-20T22:08:19.6235825Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6236659Z "size": 564, 2024-08-20T22:08:19.6237304Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:08:19.6238134Z }, 2024-08-20T22:08:19.6238468Z { 2024-08-20T22:08:19.6239018Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6239704Z "size": 35874203, 2024-08-20T22:08:19.6240344Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:08:19.6241106Z }, 2024-08-20T22:08:19.6241491Z { 2024-08-20T22:08:19.6242032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6242635Z "size": 104, 2024-08-20T22:08:19.6243327Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:08:19.6244031Z }, 2024-08-20T22:08:19.6244332Z { 2024-08-20T22:08:19.6244896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6245553Z "size": 425, 2024-08-20T22:08:19.6246239Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:08:19.6246938Z }, 2024-08-20T22:08:19.6247264Z { 2024-08-20T22:08:19.6247818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6248477Z "size": 20262183, 2024-08-20T22:08:19.6249080Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:08:19.6249839Z }, 2024-08-20T22:08:19.6250195Z { 2024-08-20T22:08:19.6250713Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6251410Z "size": 438, 2024-08-20T22:08:19.6252056Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:08:19.6252746Z }, 2024-08-20T22:08:19.6253110Z { 2024-08-20T22:08:19.6253624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6254255Z "size": 701, 2024-08-20T22:08:19.6254960Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:19.6255624Z }, 2024-08-20T22:08:19.6255954Z { 2024-08-20T22:08:19.6256577Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6257217Z "size": 141, 2024-08-20T22:08:19.6257822Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:08:19.6258613Z }, 2024-08-20T22:08:19.6258954Z { 2024-08-20T22:08:19.6259460Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6260328Z "size": 135, 2024-08-20T22:08:19.6260987Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:08:19.6261707Z }, 2024-08-20T22:08:19.6262060Z { 2024-08-20T22:08:19.6262595Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6263261Z "size": 4838068375, 2024-08-20T22:08:19.6263949Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:08:19.6264633Z }, 2024-08-20T22:08:19.6264970Z { 2024-08-20T22:08:19.6265537Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6266192Z "size": 194, 2024-08-20T22:08:19.6266789Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:08:19.6267534Z }, 2024-08-20T22:08:19.6267896Z { 2024-08-20T22:08:19.6268384Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6269084Z "size": 564, 2024-08-20T22:08:19.6269727Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:08:19.6270646Z }, 2024-08-20T22:08:19.6271046Z { 2024-08-20T22:08:19.6271706Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6272509Z "size": 43162921, 2024-08-20T22:08:19.6273177Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:08:19.6273931Z }, 2024-08-20T22:08:19.6274351Z { 2024-08-20T22:08:19.6274874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6275479Z "size": 106, 2024-08-20T22:08:19.6276181Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:08:19.6276890Z }, 2024-08-20T22:08:19.6277272Z { 2024-08-20T22:08:19.6277892Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6278618Z "size": 1380, 2024-08-20T22:08:19.6279253Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:08:19.6280041Z }, 2024-08-20T22:08:19.6280412Z { 2024-08-20T22:08:19.6281095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6281961Z "size": 701, 2024-08-20T22:08:19.6282622Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:19.6283300Z }, 2024-08-20T22:08:19.6283683Z { 2024-08-20T22:08:19.6284272Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6284875Z "size": 139, 2024-08-20T22:08:19.6285549Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:08:19.6286262Z }, 2024-08-20T22:08:19.6286645Z { 2024-08-20T22:08:19.6287131Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6287789Z "size": 118, 2024-08-20T22:08:19.6288470Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:08:19.6289201Z }, 2024-08-20T22:08:19.6289504Z { 2024-08-20T22:08:19.6290079Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6290789Z "size": 4324672728, 2024-08-20T22:08:19.6291467Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:08:19.6292202Z }, 2024-08-20T22:08:19.6292574Z { 2024-08-20T22:08:19.6293102Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6293748Z "size": 174, 2024-08-20T22:08:19.6294384Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:08:19.6295073Z }, 2024-08-20T22:08:19.6295483Z { 2024-08-20T22:08:19.6295967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6296615Z "size": 907, 2024-08-20T22:08:19.6297331Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:08:19.6298195Z }, 2024-08-20T22:08:19.6298504Z { 2024-08-20T22:08:19.6299107Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6299770Z "size": 701, 2024-08-20T22:08:19.6300361Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:08:19.6301204Z }, 2024-08-20T22:08:19.6301546Z { 2024-08-20T22:08:19.6302099Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6302755Z "size": 134, 2024-08-20T22:08:19.6303386Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:08:19.6304108Z }, 2024-08-20T22:08:19.6304528Z { 2024-08-20T22:08:19.6305007Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6305671Z "size": 32, 2024-08-20T22:08:19.6306343Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:19.6307067Z }, 2024-08-20T22:08:19.6307372Z { 2024-08-20T22:08:19.6307930Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6308593Z "size": 158, 2024-08-20T22:08:19.6309198Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:08:19.6309963Z }, 2024-08-20T22:08:19.6310326Z { 2024-08-20T22:08:19.6310887Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6311689Z "size": 1841, 2024-08-20T22:08:19.6312319Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:08:19.6313070Z }, 2024-08-20T22:08:19.6313427Z { 2024-08-20T22:08:19.6313907Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6314598Z "size": 7529776, 2024-08-20T22:08:19.6315265Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:08:19.6315963Z }, 2024-08-20T22:08:19.6316313Z { 2024-08-20T22:08:19.6316865Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6317511Z "size": 163, 2024-08-20T22:08:19.6318293Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:08:19.6318986Z }, 2024-08-20T22:08:19.6319328Z { 2024-08-20T22:08:19.6319920Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6320538Z "size": 7943, 2024-08-20T22:08:19.6321175Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:08:19.6321960Z }, 2024-08-20T22:08:19.6322308Z { 2024-08-20T22:08:19.6322793Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6323516Z "size": 8068, 2024-08-20T22:08:19.6324160Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:08:19.6324837Z }, 2024-08-20T22:08:19.6325275Z { 2024-08-20T22:08:19.6325805Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6326982Z "size": 302, 2024-08-20T22:08:19.6327640Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:08:19.6328339Z }, 2024-08-20T22:08:19.6328691Z { 2024-08-20T22:08:19.6329255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6329858Z "size": 7629638, 2024-08-20T22:08:19.6330524Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:08:19.6331310Z }, 2024-08-20T22:08:19.6331635Z { 2024-08-20T22:08:19.6332204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6332903Z "size": 108, 2024-08-20T22:08:19.6333573Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:08:19.6334243Z }, 2024-08-20T22:08:19.6334642Z { 2024-08-20T22:08:19.6335180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6336029Z "size": 54145661, 2024-08-20T22:08:19.6336680Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:08:19.6337392Z }, 2024-08-20T22:08:19.6337793Z { 2024-08-20T22:08:19.6338333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6338976Z "size": 473, 2024-08-20T22:08:19.6339650Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:08:19.6340389Z }, 2024-08-20T22:08:19.6340696Z { 2024-08-20T22:08:19.6341349Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6341991Z "size": 1374858766, 2024-08-20T22:08:19.6342641Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:08:19.6343371Z }, 2024-08-20T22:08:19.6343707Z { 2024-08-20T22:08:19.6344235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6344941Z "size": 106, 2024-08-20T22:08:19.6345557Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:08:19.6346325Z }, 2024-08-20T22:08:19.6346735Z { 2024-08-20T22:08:19.6347220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6347886Z "size": 559, 2024-08-20T22:08:19.6348566Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:08:19.6349383Z }, 2024-08-20T22:08:19.6349708Z { 2024-08-20T22:08:19.6350299Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6350969Z "size": 46248488, 2024-08-20T22:08:19.6351630Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:08:19.6352348Z }, 2024-08-20T22:08:19.6352767Z { 2024-08-20T22:08:19.6353284Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6353981Z "size": 111, 2024-08-20T22:08:19.6354607Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:08:19.6355304Z }, 2024-08-20T22:08:19.6355694Z { 2024-08-20T22:08:19.6356207Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6356853Z "size": 32, 2024-08-20T22:08:19.6357552Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:19.6358370Z }, 2024-08-20T22:08:19.6358676Z { 2024-08-20T22:08:19.6359301Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6359933Z "size": 32, 2024-08-20T22:08:19.6360627Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:19.6361357Z }, 2024-08-20T22:08:19.6361706Z { 2024-08-20T22:08:19.6362273Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6362902Z "size": 32, 2024-08-20T22:08:19.6363550Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:19.6364325Z }, 2024-08-20T22:08:19.6364679Z { 2024-08-20T22:08:19.6365167Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:08:19.6365881Z "size": 32, 2024-08-20T22:08:19.6366555Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:08:19.6367276Z } 2024-08-20T22:08:19.6367654Z ] 2024-08-20T22:08:19.6367997Z } 2024-08-20T22:08:19.6670132Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:19.6670749Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:08:19.6671444Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:08:19.6681748Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:19.6682235Z env: 2024-08-20T22:08:19.6682510Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:19.6683666Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:19.6684839Z ##[endgroup] 2024-08-20T22:08:19.6720184Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:19.6784936Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:08:19.6785513Z with: 2024-08-20T22:08:19.6786599Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:19.6787909Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:19.6788430Z env: 2024-08-20T22:08:19.6788710Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:19.6789061Z ##[endgroup] 2024-08-20T22:08:19.6806421Z ##[group]Run set -x 2024-08-20T22:08:19.6806760Z set -x 2024-08-20T22:08:19.6807066Z set +e 2024-08-20T22:08:19.6807369Z  2024-08-20T22:08:19.6807640Z login() { 2024-08-20T22:08:19.6808307Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:08:19.6809017Z } 2024-08-20T22:08:19.6809300Z  2024-08-20T22:08:19.6809606Z retry () { 2024-08-20T22:08:19.6810004Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:08:19.6810496Z } 2024-08-20T22:08:19.6810941Z  2024-08-20T22:08:19.6811264Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:08:19.6811676Z  2024-08-20T22:08:19.6811951Z set -e 2024-08-20T22:08:19.6812424Z # ignore output since only exit code is used for conditional 2024-08-20T22:08:19.6813127Z # only pull docker image if it's not available locally 2024-08-20T22:08:19.6813895Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:08:19.6814595Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:08:19.6815039Z fi 2024-08-20T22:08:19.6823820Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:08:19.6824307Z env: 2024-08-20T22:08:19.6824586Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:08:19.6825751Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:19.6827437Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:19.6827959Z ##[endgroup] 2024-08-20T22:08:19.6857833Z + set +e 2024-08-20T22:08:19.6858503Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:19.6859165Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:19.6861504Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:08:19.6863335Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:08:20.2238856Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:08:20.2239664Z Configure a credential helper to remove this warning. See 2024-08-20T22:08:20.2240614Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:08:20.2241241Z 2024-08-20T22:08:20.2241392Z Login Succeeded 2024-08-20T22:08:20.2266644Z + set -e 2024-08-20T22:08:20.2267967Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:20.2432744Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:20.2434894Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:08:20.4631073Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:08:20.4632321Z 7a2c55901189: Pulling fs layer 2024-08-20T22:08:20.4632823Z 224fe954d725: Pulling fs layer 2024-08-20T22:08:20.4633621Z 75722010b82e: Pulling fs layer 2024-08-20T22:08:20.4634027Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:08:20.4634399Z b57676e46aee: Pulling fs layer 2024-08-20T22:08:20.4634795Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:08:20.4635219Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:08:20.4635738Z 0c1227890755: Pulling fs layer 2024-08-20T22:08:20.4636117Z d8d1234baab3: Pulling fs layer 2024-08-20T22:08:20.4636574Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:08:20.4637155Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:08:20.4637658Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:08:20.4638257Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:08:20.4638739Z 4373aa743caf: Pulling fs layer 2024-08-20T22:08:20.4639216Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:08:20.4639730Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:08:20.4640278Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:08:20.4640852Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:08:20.4641368Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:08:20.4641751Z d527cbbb87e3: Waiting 2024-08-20T22:08:20.4642086Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:08:20.4642470Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:08:20.4643050Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:08:20.4643444Z a8c1e85b5e14: Waiting 2024-08-20T22:08:20.4643881Z 33b639226eee: Pulling fs layer 2024-08-20T22:08:20.4644288Z d3547bd43146: Pulling fs layer 2024-08-20T22:08:20.4644777Z 7ed32bc8e469: Waiting 2024-08-20T22:08:20.4645207Z 5645520256fa: Pulling fs layer 2024-08-20T22:08:20.4645700Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:08:20.4646188Z 385d0421d67c: Pulling fs layer 2024-08-20T22:08:20.4646668Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:08:20.4647141Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:08:20.4647590Z d8d1234baab3: Waiting 2024-08-20T22:08:20.4647914Z ca426ffc9d2b: Waiting 2024-08-20T22:08:20.4648229Z 4373aa743caf: Waiting 2024-08-20T22:08:20.4648549Z 2df849f72ba4: Waiting 2024-08-20T22:08:20.4648868Z ef1e3ef1e85f: Waiting 2024-08-20T22:08:20.4649213Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:08:20.4649577Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:08:20.4649939Z d3547bd43146: Waiting 2024-08-20T22:08:20.4650272Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:08:20.4650670Z 7754ee068b33: Pulling fs layer 2024-08-20T22:08:20.4651057Z 0479fbac3361: Pulling fs layer 2024-08-20T22:08:20.4651445Z 385d0421d67c: Waiting 2024-08-20T22:08:20.4651806Z 75b2abb16c68: Waiting 2024-08-20T22:08:20.4652120Z 7cbfce999d76: Waiting 2024-08-20T22:08:20.4652434Z 0bf61114d37b: Waiting 2024-08-20T22:08:20.4652814Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:08:20.4653205Z 5645520256fa: Waiting 2024-08-20T22:08:20.4653525Z afc429e87b59: Pulling fs layer 2024-08-20T22:08:20.4653935Z 34c47d63c5de: Waiting 2024-08-20T22:08:20.4654319Z 6efb96fc87ea: Waiting 2024-08-20T22:08:20.4654655Z 019620471c25: Pulling fs layer 2024-08-20T22:08:20.4655033Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:08:20.4655414Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:08:20.4655798Z 560cd6122a38: Pulling fs layer 2024-08-20T22:08:20.4656170Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:08:20.4656518Z 7c47122f4e09: Waiting 2024-08-20T22:08:20.4656833Z 0479fbac3361: Waiting 2024-08-20T22:08:20.4657171Z da6a838a66c7: Pulling fs layer 2024-08-20T22:08:20.4657559Z cbb79a705353: Pulling fs layer 2024-08-20T22:08:20.4657922Z f1d6874dee9c: Waiting 2024-08-20T22:08:20.4658254Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:08:20.4658611Z afc429e87b59: Waiting 2024-08-20T22:08:20.4658943Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:08:20.4659316Z 4a0412373305: Pulling fs layer 2024-08-20T22:08:20.4659673Z 27aeb1b2cd5e: Waiting 2024-08-20T22:08:20.4659988Z 7754ee068b33: Waiting 2024-08-20T22:08:20.4660314Z 302c8c177944: Pulling fs layer 2024-08-20T22:08:20.4660677Z 1fef4bd19792: Waiting 2024-08-20T22:08:20.4660998Z 4d4fa6802ab3: Waiting 2024-08-20T22:08:20.4661315Z cbb79a705353: Waiting 2024-08-20T22:08:20.4661636Z 02970347dff6: Pulling fs layer 2024-08-20T22:08:20.4661993Z 560cd6122a38: Waiting 2024-08-20T22:08:20.4662487Z 916c5e858016: Pulling fs layer 2024-08-20T22:08:20.4662867Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:08:20.4663245Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:08:20.4663614Z 0f085d4fa279: Waiting 2024-08-20T22:08:20.4663935Z 5a4eb19d7219: Waiting 2024-08-20T22:08:20.4664263Z f957c1624481: Pulling fs layer 2024-08-20T22:08:20.4664642Z 768c31dca016: Pulling fs layer 2024-08-20T22:08:20.4665001Z 4a0412373305: Waiting 2024-08-20T22:08:20.4665319Z da6a838a66c7: Waiting 2024-08-20T22:08:20.4665656Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:08:20.4666044Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:08:20.4666422Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:08:20.4666778Z 916c5e858016: Waiting 2024-08-20T22:08:20.4667117Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:08:20.4667480Z c42b3602a1fb: Waiting 2024-08-20T22:08:20.4667821Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:08:20.4668177Z 829c0e19f4b7: Waiting 2024-08-20T22:08:20.4668501Z 8c32d0ecebbe: Waiting 2024-08-20T22:08:20.4668856Z b10de02255c0: Pulling fs layer 2024-08-20T22:08:20.4669213Z 768c31dca016: Waiting 2024-08-20T22:08:20.4669537Z 78ee77eddef2: Waiting 2024-08-20T22:08:20.4669861Z 4f4fb700ef54: Waiting 2024-08-20T22:08:20.4670422Z e56eeff2db93: Pulling fs layer 2024-08-20T22:08:20.4670915Z 37513e9b43d7: Waiting 2024-08-20T22:08:20.4671329Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:08:20.4671813Z 11fad972ef6c: Waiting 2024-08-20T22:08:20.4672257Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:08:20.4672717Z ee1273153675: Pulling fs layer 2024-08-20T22:08:20.4673124Z a7a5392e2bf6: Waiting 2024-08-20T22:08:20.4673445Z b10de02255c0: Waiting 2024-08-20T22:08:20.4673757Z e56eeff2db93: Waiting 2024-08-20T22:08:20.4674091Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:08:20.4674529Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:08:20.4674910Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:08:20.4675278Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:08:20.4675665Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:08:20.4676048Z d79fd7d52698: Pulling fs layer 2024-08-20T22:08:20.4676437Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:08:20.4676792Z 6c1f0aabce7c: Waiting 2024-08-20T22:08:20.4677117Z 02970347dff6: Waiting 2024-08-20T22:08:20.4677451Z 5d115b29b5d5: Waiting 2024-08-20T22:08:20.4677849Z 7ebe49c3a23e: Waiting 2024-08-20T22:08:20.4678191Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:08:20.4678562Z 9752d6fbbcab: Waiting 2024-08-20T22:08:20.4678887Z 019620471c25: Waiting 2024-08-20T22:08:20.4679200Z ec1e7978c1fe: Waiting 2024-08-20T22:08:20.4679536Z 83808b2eff4d: Waiting 2024-08-20T22:08:20.4679879Z 33b639226eee: Waiting 2024-08-20T22:08:20.4680331Z eb09cbf3e416: Waiting 2024-08-20T22:08:20.5940082Z 224fe954d725: Verifying Checksum 2024-08-20T22:08:20.5940674Z 224fe954d725: Download complete 2024-08-20T22:08:20.6850255Z d527cbbb87e3: Download complete 2024-08-20T22:08:20.7561444Z b57676e46aee: Download complete 2024-08-20T22:08:20.7992972Z 7a2c55901189: Verifying Checksum 2024-08-20T22:08:20.7993518Z 7a2c55901189: Download complete 2024-08-20T22:08:20.8804380Z a41a8d1c11c8: Download complete 2024-08-20T22:08:20.9502010Z 0c1227890755: Download complete 2024-08-20T22:08:21.0117073Z d8d1234baab3: Verifying Checksum 2024-08-20T22:08:21.0117679Z d8d1234baab3: Download complete 2024-08-20T22:08:21.1075771Z 75722010b82e: Verifying Checksum 2024-08-20T22:08:21.1076377Z 75722010b82e: Download complete 2024-08-20T22:08:21.1933111Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:08:21.1933645Z ec1e7978c1fe: Download complete 2024-08-20T22:08:21.2893292Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:08:21.2893904Z ac23c5fcc1ef: Download complete 2024-08-20T22:08:21.9997837Z 7a2c55901189: Pull complete 2024-08-20T22:08:22.3226360Z 224fe954d725: Pull complete 2024-08-20T22:08:23.2489474Z 75722010b82e: Pull complete 2024-08-20T22:08:23.2703457Z d527cbbb87e3: Pull complete 2024-08-20T22:08:23.2904876Z b57676e46aee: Pull complete 2024-08-20T22:08:23.8022112Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:08:23.8022706Z ca426ffc9d2b: Download complete 2024-08-20T22:08:23.8679483Z 4373aa743caf: Download complete 2024-08-20T22:08:23.9444169Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:08:23.9444617Z 75b2abb16c68: Download complete 2024-08-20T22:08:24.0266060Z ef1e3ef1e85f: Download complete 2024-08-20T22:08:24.9998318Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:08:24.9998803Z 0bf61114d37b: Download complete 2024-08-20T22:08:25.0706605Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:08:25.0707059Z 2df849f72ba4: Download complete 2024-08-20T22:08:25.1417672Z 34c47d63c5de: Download complete 2024-08-20T22:08:25.2093248Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:08:25.2093728Z 1fef4bd19792: Download complete 2024-08-20T22:08:34.0987373Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:08:34.0987910Z a8c1e85b5e14: Download complete 2024-08-20T22:08:34.1719604Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:08:34.1720187Z b119c4fc6c58: Download complete 2024-08-20T22:08:34.2420087Z 33b639226eee: Download complete 2024-08-20T22:08:34.3199862Z d3547bd43146: Download complete 2024-08-20T22:08:34.3923314Z 5645520256fa: Verifying Checksum 2024-08-20T22:08:34.3923794Z 5645520256fa: Download complete 2024-08-20T22:08:34.4604695Z 829ed7d91e38: Download complete 2024-08-20T22:08:35.7252519Z 385d0421d67c: Verifying Checksum 2024-08-20T22:08:35.7253155Z 385d0421d67c: Download complete 2024-08-20T22:08:35.7906903Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:08:35.7907502Z 7cbfce999d76: Download complete 2024-08-20T22:08:35.8678350Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:08:35.8678929Z 6658e558ebe4: Download complete 2024-08-20T22:08:35.9416473Z 33444a5a7e01: Download complete 2024-08-20T22:08:36.0352212Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:08:36.0352755Z 7c47122f4e09: Download complete 2024-08-20T22:08:36.1088060Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:08:36.1088519Z 1811ff4d3556: Download complete 2024-08-20T22:08:42.1142038Z 7754ee068b33: Verifying Checksum 2024-08-20T22:08:42.1142638Z 7754ee068b33: Download complete 2024-08-20T22:08:42.2196305Z 0479fbac3361: Verifying Checksum 2024-08-20T22:08:42.2942637Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:08:42.2943195Z f1d6874dee9c: Download complete 2024-08-20T22:08:42.7113883Z afc429e87b59: Verifying Checksum 2024-08-20T22:08:42.7114481Z afc429e87b59: Download complete 2024-08-20T22:08:42.7859975Z 019620471c25: Verifying Checksum 2024-08-20T22:08:42.7860500Z 019620471c25: Download complete 2024-08-20T22:08:42.8658095Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:08:42.8658562Z 27aeb1b2cd5e: Download complete 2024-08-20T22:08:43.1272636Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:08:43.1273113Z 829c0e19f4b7: Download complete 2024-08-20T22:08:43.1900819Z 560cd6122a38: Verifying Checksum 2024-08-20T22:08:43.2531339Z 5a4eb19d7219: Download complete 2024-08-20T22:08:43.3201163Z da6a838a66c7: Download complete 2024-08-20T22:08:46.3441702Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:08:46.3442306Z 7ed32bc8e469: Download complete 2024-08-20T22:08:46.4205969Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:08:46.4206493Z 4d4fa6802ab3: Download complete 2024-08-20T22:08:46.5023552Z 0f085d4fa279: Verifying Checksum 2024-08-20T22:08:46.5024137Z 0f085d4fa279: Download complete 2024-08-20T22:08:47.1063031Z 4a0412373305: Verifying Checksum 2024-08-20T22:08:47.1063510Z 4a0412373305: Download complete 2024-08-20T22:08:47.1860398Z 302c8c177944: Verifying Checksum 2024-08-20T22:08:47.1860845Z 302c8c177944: Download complete 2024-08-20T22:08:47.2542865Z 02970347dff6: Verifying Checksum 2024-08-20T22:08:47.2544210Z 02970347dff6: Download complete 2024-08-20T22:08:47.3287598Z a8c1e85b5e14: Pull complete 2024-08-20T22:08:47.3497355Z 916c5e858016: Verifying Checksum 2024-08-20T22:08:47.3497926Z 916c5e858016: Download complete 2024-08-20T22:08:47.4227628Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:08:47.4228069Z c42b3602a1fb: Download complete 2024-08-20T22:08:47.5606529Z a41a8d1c11c8: Pull complete 2024-08-20T22:08:47.8003197Z 0c1227890755: Pull complete 2024-08-20T22:08:48.0229940Z d8d1234baab3: Pull complete 2024-08-20T22:08:53.8972080Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:08:53.8972743Z 6efb96fc87ea: Download complete 2024-08-20T22:08:54.0061508Z f957c1624481: Verifying Checksum 2024-08-20T22:08:54.0062001Z f957c1624481: Download complete 2024-08-20T22:08:54.0699578Z 768c31dca016: Verifying Checksum 2024-08-20T22:08:54.0700023Z 768c31dca016: Download complete 2024-08-20T22:08:54.1427453Z 8c32d0ecebbe: Download complete 2024-08-20T22:08:54.1531080Z 4f4fb700ef54: Download complete 2024-08-20T22:08:54.2193038Z 78ee77eddef2: Verifying Checksum 2024-08-20T22:08:54.2193505Z 78ee77eddef2: Download complete 2024-08-20T22:08:54.2998023Z 37513e9b43d7: Download complete 2024-08-20T22:08:54.4658770Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:08:54.4659219Z a7a5392e2bf6: Download complete 2024-08-20T22:08:54.5583309Z b10de02255c0: Verifying Checksum 2024-08-20T22:08:54.5583798Z b10de02255c0: Download complete 2024-08-20T22:08:54.6339367Z e56eeff2db93: Download complete 2024-08-20T22:08:54.6930134Z 70ac3e38d0fd: Verifying Checksum 2024-08-20T22:08:54.6930680Z 70ac3e38d0fd: Download complete 2024-08-20T22:08:54.7575595Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:08:54.7576201Z eb09cbf3e416: Download complete 2024-08-20T22:08:54.8987330Z ee1273153675: Verifying Checksum 2024-08-20T22:08:54.8987953Z ee1273153675: Download complete 2024-08-20T22:08:54.9683782Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:08:54.9684347Z e22979f9e0ee: Download complete 2024-08-20T22:08:55.5653509Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:08:55.5653991Z 6c1f0aabce7c: Download complete 2024-08-20T22:08:55.6427777Z 7ebe49c3a23e: Download complete 2024-08-20T22:09:21.2854936Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:09:21.2855546Z 5d115b29b5d5: Download complete 2024-08-20T22:09:21.3520750Z 7f2d7a04fbe6: Verifying Checksum 2024-08-20T22:09:21.3521284Z 7f2d7a04fbe6: Download complete 2024-08-20T22:09:21.4210353Z d79fd7d52698: Verifying Checksum 2024-08-20T22:09:21.4210903Z d79fd7d52698: Download complete 2024-08-20T22:09:22.6779790Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:09:22.6780389Z 9752d6fbbcab: Download complete 2024-08-20T22:09:22.7506170Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:09:22.7506762Z 83808b2eff4d: Download complete 2024-08-20T22:10:01.7279287Z cbb79a705353: Verifying Checksum 2024-08-20T22:10:01.7281573Z cbb79a705353: Download complete 2024-08-20T22:10:02.2904996Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:10:02.2905554Z 11fad972ef6c: Download complete 2024-08-20T22:10:16.8342619Z 7ed32bc8e469: Pull complete 2024-08-20T22:10:17.0674279Z ec1e7978c1fe: Pull complete 2024-08-20T22:10:17.2906144Z ac23c5fcc1ef: Pull complete 2024-08-20T22:10:25.9113540Z ca426ffc9d2b: Pull complete 2024-08-20T22:10:26.1325310Z 4373aa743caf: Pull complete 2024-08-20T22:10:26.3859276Z 75b2abb16c68: Pull complete 2024-08-20T22:10:26.5753373Z ef1e3ef1e85f: Pull complete 2024-08-20T22:10:29.1887162Z 0bf61114d37b: Pull complete 2024-08-20T22:10:29.3055944Z 2df849f72ba4: Pull complete 2024-08-20T22:10:29.3756401Z 34c47d63c5de: Pull complete 2024-08-20T22:10:29.4836838Z 1fef4bd19792: Pull complete 2024-08-20T22:11:28.2684837Z 6efb96fc87ea: Pull complete 2024-08-20T22:11:28.3783130Z b119c4fc6c58: Pull complete 2024-08-20T22:11:28.6140564Z 33b639226eee: Pull complete 2024-08-20T22:11:28.8352744Z d3547bd43146: Pull complete 2024-08-20T22:11:29.0411630Z 5645520256fa: Pull complete 2024-08-20T22:11:29.2684878Z 829ed7d91e38: Pull complete 2024-08-20T22:11:32.9553342Z 385d0421d67c: Pull complete 2024-08-20T22:11:33.0867812Z 7cbfce999d76: Pull complete 2024-08-20T22:11:33.2683557Z 6658e558ebe4: Pull complete 2024-08-20T22:11:33.4536946Z 33444a5a7e01: Pull complete 2024-08-20T22:11:33.5697869Z 7c47122f4e09: Pull complete 2024-08-20T22:11:33.7991669Z 1811ff4d3556: Pull complete 2024-08-20T22:11:43.9754796Z 7754ee068b33: Pull complete 2024-08-20T22:11:44.0935678Z 0479fbac3361: Pull complete 2024-08-20T22:11:44.2481432Z f1d6874dee9c: Pull complete 2024-08-20T22:11:45.1757316Z afc429e87b59: Pull complete 2024-08-20T22:11:45.3661875Z 019620471c25: Pull complete 2024-08-20T22:11:45.5700888Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:11:46.0323151Z 829c0e19f4b7: Pull complete 2024-08-20T22:11:46.1317257Z 560cd6122a38: Pull complete 2024-08-20T22:11:46.4399867Z 5a4eb19d7219: Pull complete 2024-08-20T22:11:46.6646554Z da6a838a66c7: Pull complete 2024-08-20T22:12:25.4312567Z cbb79a705353: Pull complete 2024-08-20T22:12:25.6603844Z 4d4fa6802ab3: Pull complete 2024-08-20T22:12:25.8008503Z 0f085d4fa279: Pull complete 2024-08-20T22:12:28.1032156Z 4a0412373305: Pull complete 2024-08-20T22:12:28.1162873Z 302c8c177944: Pull complete 2024-08-20T22:12:28.1310825Z 02970347dff6: Pull complete 2024-08-20T22:12:28.1571024Z 916c5e858016: Pull complete 2024-08-20T22:12:28.1768067Z c42b3602a1fb: Pull complete 2024-08-20T22:13:49.7985480Z 11fad972ef6c: Pull complete 2024-08-20T22:13:50.0322213Z f957c1624481: Pull complete 2024-08-20T22:13:50.2523889Z 768c31dca016: Pull complete 2024-08-20T22:13:50.6669147Z 8c32d0ecebbe: Pull complete 2024-08-20T22:13:50.8835674Z 4f4fb700ef54: Pull complete 2024-08-20T22:13:51.1167591Z 78ee77eddef2: Pull complete 2024-08-20T22:13:51.3544356Z 37513e9b43d7: Pull complete 2024-08-20T22:13:51.7145703Z a7a5392e2bf6: Pull complete 2024-08-20T22:13:51.9366074Z b10de02255c0: Pull complete 2024-08-20T22:13:52.1684625Z e56eeff2db93: Pull complete 2024-08-20T22:13:52.3950658Z 70ac3e38d0fd: Pull complete 2024-08-20T22:13:52.6172088Z eb09cbf3e416: Pull complete 2024-08-20T22:13:54.0355846Z ee1273153675: Pull complete 2024-08-20T22:13:54.2554539Z e22979f9e0ee: Pull complete 2024-08-20T22:13:56.4415604Z 6c1f0aabce7c: Pull complete 2024-08-20T22:13:56.6669858Z 7ebe49c3a23e: Pull complete 2024-08-20T22:14:10.0234637Z 5d115b29b5d5: Pull complete 2024-08-20T22:14:10.2185584Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:14:10.4246014Z d79fd7d52698: Pull complete 2024-08-20T22:14:11.3140073Z 9752d6fbbcab: Pull complete 2024-08-20T22:14:11.5374251Z 83808b2eff4d: Pull complete 2024-08-20T22:14:12.5160223Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:14:12.5704605Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:12.6003636Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:14:12.6088762Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:12.6089702Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:14:12.6100222Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:14:12.6100712Z env: 2024-08-20T22:14:12.6100991Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:12.6101342Z ##[endgroup] 2024-08-20T22:14:12.6309841Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:14:12.6310406Z with: 2024-08-20T22:14:12.6310691Z driver-version: 550.54.15 2024-08-20T22:14:12.6311023Z env: 2024-08-20T22:14:12.6311298Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:12.6311656Z ##[endgroup] 2024-08-20T22:14:12.6428368Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:14:12.6429074Z with: 2024-08-20T22:14:12.6429426Z timeout_minutes: 10 2024-08-20T22:14:12.6429839Z max_attempts: 3 2024-08-20T22:14:12.6463889Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:14:12.6498698Z retry_wait_seconds: 10 2024-08-20T22:14:12.6499157Z polling_interval_seconds: 1 2024-08-20T22:14:12.6499587Z warning_on_retry: true 2024-08-20T22:14:12.6499940Z continue_on_error: false 2024-08-20T22:14:12.6500277Z env: 2024-08-20T22:14:12.6500551Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:14:12.6501006Z DRIVER_VERSION: 550.54.15 2024-08-20T22:14:12.6501363Z ##[endgroup] 2024-08-20T22:14:12.7330185Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:14:12.7331277Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:14:12.7333389Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:14:13.2134832Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:14:13.2135509Z No packages marked for removal. 2024-08-20T22:14:13.2198402Z Dependencies resolved. 2024-08-20T22:14:13.2209060Z Nothing to do. 2024-08-20T22:14:13.2209574Z Complete! 2024-08-20T22:14:13.3101483Z + install_nvidia_driver_common 2024-08-20T22:14:13.3103920Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:14:13.3104568Z Before installing NVIDIA driver 2024-08-20T22:14:13.3105080Z + lspci 2024-08-20T22:14:13.4000606Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:13.4001621Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:13.4002777Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:13.4003856Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:13.4004782Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:13.4005568Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:13.4006303Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:13.4007109Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:13.4007702Z + lsmod 2024-08-20T22:14:13.4046939Z Module Size Used by 2024-08-20T22:14:13.4047542Z xt_conntrack 16384 1 2024-08-20T22:14:13.4048063Z nft_chain_nat 16384 3 2024-08-20T22:14:13.4048511Z xt_MASQUERADE 20480 1 2024-08-20T22:14:13.4049090Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:13.4049765Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:13.4050509Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:13.4051151Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:13.4051609Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:13.4052660Z xfrm_user 57344 1 2024-08-20T22:14:13.4053134Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:13.4053685Z xt_addrtype 16384 2 2024-08-20T22:14:13.4054217Z nft_compat 20480 4 2024-08-20T22:14:13.4054814Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:13.4055446Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:13.4055999Z br_netfilter 36864 0 2024-08-20T22:14:13.4056411Z bridge 307200 1 br_netfilter 2024-08-20T22:14:13.4056862Z stp 16384 1 bridge 2024-08-20T22:14:13.4057294Z llc 16384 2 bridge,stp 2024-08-20T22:14:13.4057727Z overlay 167936 0 2024-08-20T22:14:13.4058102Z tls 114688 0 2024-08-20T22:14:13.4058479Z nls_ascii 16384 1 2024-08-20T22:14:13.4058859Z nls_cp437 20480 1 2024-08-20T22:14:13.4059231Z vfat 24576 1 2024-08-20T22:14:13.4059604Z fat 86016 1 vfat 2024-08-20T22:14:13.4060006Z sunrpc 692224 1 2024-08-20T22:14:13.4060378Z ena 167936 0 2024-08-20T22:14:13.4060748Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:13.4061141Z ptp 36864 1 ena 2024-08-20T22:14:13.4061547Z pps_core 24576 1 ptp 2024-08-20T22:14:13.4061960Z aesni_intel 393216 0 2024-08-20T22:14:13.4062329Z i8042 45056 0 2024-08-20T22:14:13.4062706Z serio 28672 3 i8042 2024-08-20T22:14:13.4063139Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:13.4063670Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:13.4064378Z button 24576 0 2024-08-20T22:14:13.4064764Z sch_fq_codel 20480 17 2024-08-20T22:14:13.4065148Z dm_mod 188416 0 2024-08-20T22:14:13.4065524Z fuse 163840 1 2024-08-20T22:14:13.4065895Z configfs 57344 1 2024-08-20T22:14:13.4066269Z loop 36864 0 2024-08-20T22:14:13.4066653Z dax 45056 1 dm_mod 2024-08-20T22:14:13.4067054Z dmi_sysfs 20480 0 2024-08-20T22:14:13.4067523Z crc32_pclmul 16384 0 2024-08-20T22:14:13.4067960Z crc32c_intel 24576 0 2024-08-20T22:14:13.4068441Z efivarfs 24576 1 2024-08-20T22:14:13.4068798Z + modinfo nvidia 2024-08-20T22:14:13.4069739Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:13.4070508Z alias: char-major-195-* 2024-08-20T22:14:13.4070891Z version: 550.54.15 2024-08-20T22:14:13.4071240Z supported: external 2024-08-20T22:14:13.4071590Z license: NVIDIA 2024-08-20T22:14:13.4071965Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:13.4072455Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:13.4073017Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:13.4073569Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:13.4074145Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:13.4074635Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:13.4075144Z depends: i2c-core,drm 2024-08-20T22:14:13.4075514Z retpoline: Y 2024-08-20T22:14:13.4075824Z name: nvidia 2024-08-20T22:14:13.4076793Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:13.4077887Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:13.4078780Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:13.4079581Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:13.4080051Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:13.4080485Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:13.4080943Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:13.4081382Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:13.4081814Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:13.4082343Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:13.4082901Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:13.4083379Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:13.4083807Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:13.4084235Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:13.4084693Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:13.4085227Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:13.4085806Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:13.4086366Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:13.4086983Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:13.4087579Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:13.4088189Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:13.4088781Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:13.4089274Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:13.4089807Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:13.4090349Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:13.4090847Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:13.4091316Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:13.4091800Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:13.4092267Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:13.4092734Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:13.4093237Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:13.4093922Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:13.4094397Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:13.4094886Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:13.4095384Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:13.4095870Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:13.4096360Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:13.4096836Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:13.4097250Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:13.4097712Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:13.4098226Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:13.4098699Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:13.4099178Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:13.4099646Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:13.4100117Z parm: rm_firmware_active:charp 2024-08-20T22:14:13.4100545Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:14:13.4100955Z ++ command -v nvidia-smi 2024-08-20T22:14:13.4101358Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:14:13.4101733Z + set +e 2024-08-20T22:14:13.4102244Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:14:15.6514429Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:14:15.6515109Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:15.6515684Z + '[' 0 -ne 0 ']' 2024-08-20T22:14:15.6516058Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:14:15.6516441Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:14:15.6517548Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:14:15.6518391Z + set -e 2024-08-20T22:14:15.6518697Z + '[' 1 -eq 0 ']' 2024-08-20T22:14:15.6519281Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:14:15.6520022Z + post_install_nvidia_driver_common 2024-08-20T22:14:15.6520796Z + sudo modprobe nvidia 2024-08-20T22:14:15.7830896Z + echo 'After installing NVIDIA driver' 2024-08-20T22:14:15.7831551Z + lspci 2024-08-20T22:14:15.7831865Z After installing NVIDIA driver 2024-08-20T22:14:15.7945099Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:14:15.7945990Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:14:15.7946947Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:14:15.7947758Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:14:15.7948589Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:14:15.7949394Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:14:15.7950132Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:14:15.7951148Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:14:15.7951753Z + lsmod 2024-08-20T22:14:15.7976584Z Module Size Used by 2024-08-20T22:14:15.7977014Z nvidia_uvm 4706304 0 2024-08-20T22:14:15.7977423Z nvidia 54071296 1 nvidia_uvm 2024-08-20T22:14:15.7977853Z drm 602112 1 nvidia 2024-08-20T22:14:15.7978369Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:14:15.7978960Z backlight 24576 1 drm 2024-08-20T22:14:15.7979501Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:14:15.7979953Z xt_conntrack 16384 1 2024-08-20T22:14:15.7980336Z nft_chain_nat 16384 3 2024-08-20T22:14:15.7980723Z xt_MASQUERADE 20480 1 2024-08-20T22:14:15.7981361Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:14:15.7981983Z nf_conntrack_netlink 57344 0 2024-08-20T22:14:15.7982625Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:14:15.7983291Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:14:15.7984037Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:14:15.7984466Z xfrm_user 57344 1 2024-08-20T22:14:15.7984858Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:14:15.7985279Z xt_addrtype 16384 2 2024-08-20T22:14:15.7985659Z nft_compat 20480 4 2024-08-20T22:14:15.7986104Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:14:15.7986714Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:14:15.7987592Z br_netfilter 36864 0 2024-08-20T22:14:15.7988004Z bridge 307200 1 br_netfilter 2024-08-20T22:14:15.7988440Z stp 16384 1 bridge 2024-08-20T22:14:15.7988901Z llc 16384 2 bridge,stp 2024-08-20T22:14:15.7989328Z overlay 167936 0 2024-08-20T22:14:15.7989695Z tls 114688 0 2024-08-20T22:14:15.7990054Z nls_ascii 16384 1 2024-08-20T22:14:15.7990431Z nls_cp437 20480 1 2024-08-20T22:14:15.7990793Z vfat 24576 1 2024-08-20T22:14:15.7991151Z fat 86016 1 vfat 2024-08-20T22:14:15.7991538Z sunrpc 692224 1 2024-08-20T22:14:15.7992127Z ena 167936 0 2024-08-20T22:14:15.7992500Z ghash_clmulni_intel 16384 0 2024-08-20T22:14:15.7992936Z ptp 36864 1 ena 2024-08-20T22:14:15.7993332Z pps_core 24576 1 ptp 2024-08-20T22:14:15.7993726Z aesni_intel 393216 0 2024-08-20T22:14:15.7994089Z i8042 45056 0 2024-08-20T22:14:15.7994451Z serio 28672 3 i8042 2024-08-20T22:14:15.7995168Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:14:15.7995875Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:14:15.7996462Z button 24576 0 2024-08-20T22:14:15.7997550Z sch_fq_codel 20480 17 2024-08-20T22:14:15.7998172Z dm_mod 188416 0 2024-08-20T22:14:15.7998660Z fuse 163840 1 2024-08-20T22:14:15.7999109Z configfs 57344 1 2024-08-20T22:14:15.8000030Z loop 36864 0 2024-08-20T22:14:15.8000416Z dax 45056 1 dm_mod 2024-08-20T22:14:15.8000816Z dmi_sysfs 20480 0 2024-08-20T22:14:15.8001190Z crc32_pclmul 16384 0 2024-08-20T22:14:15.8001628Z crc32c_intel 24576 0 2024-08-20T22:14:15.8002104Z efivarfs 24576 1 2024-08-20T22:14:15.8002572Z + modinfo nvidia 2024-08-20T22:14:15.8003341Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:14:15.8004063Z alias: char-major-195-* 2024-08-20T22:14:15.8004446Z version: 550.54.15 2024-08-20T22:14:15.8004793Z supported: external 2024-08-20T22:14:15.8005129Z license: NVIDIA 2024-08-20T22:14:15.8005515Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:14:15.8006518Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:14:15.8007002Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:14:15.8007473Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:14:15.8007972Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:14:15.8008461Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:14:15.8009011Z depends: i2c-core,drm 2024-08-20T22:14:15.8009370Z retpoline: Y 2024-08-20T22:14:15.8009678Z name: nvidia 2024-08-20T22:14:15.8010334Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:14:15.8011035Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:14:15.8011693Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:14:15.8012382Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:14:15.8012827Z parm: NVreg_RmLogonRC:int 2024-08-20T22:14:15.8013261Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:14:15.8013716Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:14:15.8014290Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:14:15.8014731Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:14:15.8015262Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:14:15.8015823Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:14:15.8016305Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:14:15.8016743Z parm: NVreg_EnableMSI:int 2024-08-20T22:14:15.8017156Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:14:15.8017616Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:14:15.8018154Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:14:15.8019116Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:14:15.8019980Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:14:15.8020795Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:15.8021397Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:14:15.8022024Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:14:15.8022617Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:14:15.8023107Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:14:15.8023650Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:14:15.8024192Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:14:15.8024689Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:14:15.8025220Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:14:15.8025708Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:14:15.8026515Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:14:15.8027134Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:14:15.8027646Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:14:15.8028173Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:14:15.8028645Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:14:15.8030381Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:14:15.8031042Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:14:15.8031683Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:14:15.8032335Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:14:15.8032876Z parm: NVreg_RmMsg:charp 2024-08-20T22:14:15.8033284Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:14:15.8033755Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:14:15.8034287Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:14:15.8034911Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:14:15.8035555Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:14:15.8036198Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:14:15.8036664Z parm: rm_firmware_active:charp 2024-08-20T22:14:15.8037395Z + set +e 2024-08-20T22:14:15.8037942Z + nvidia-smi 2024-08-20T22:14:17.3687319Z Tue Aug 20 22:14:17 2024 2024-08-20T22:14:17.3688306Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.3689310Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:14:17.3690105Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.3690913Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:14:17.3691824Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:14:17.3692531Z | | | MIG M. | 2024-08-20T22:14:17.3693098Z |=========================================+========================+======================| 2024-08-20T22:14:17.3787665Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:14:17.3788642Z | 0% 30C P0 58W / 300W | 0MiB / 23028MiB | 5% Default | 2024-08-20T22:14:17.3789759Z | | | N/A | 2024-08-20T22:14:17.3790743Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:14:17.3791356Z 2024-08-20T22:14:17.3792018Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.3792758Z | Processes: | 2024-08-20T22:14:17.3793530Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:14:17.3794242Z | ID ID Usage | 2024-08-20T22:14:17.3794804Z |=========================================================================================| 2024-08-20T22:14:17.3795467Z | No running processes found | 2024-08-20T22:14:17.3796235Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:14:17.9939684Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:14:19.5635226Z NVIDIA A10G 2024-08-20T22:14:20.0247547Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:14:20.0248114Z + '[' 0 -eq 0 ']' 2024-08-20T22:14:20.0248529Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:14:20.0248965Z + set -e 2024-08-20T22:14:20.0249262Z INFO: Ignoring allowed status 0 2024-08-20T22:14:20.0256068Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:14:20.0260292Z + sudo yum install -y yum-utils 2024-08-20T22:14:20.4948915Z Last metadata expiration check: 0:09:35 ago on Tue Aug 20 22:04:45 2024. 2024-08-20T22:14:20.5175878Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:14:20.5488253Z Dependencies resolved. 2024-08-20T22:14:20.5616511Z Nothing to do. 2024-08-20T22:14:20.5616822Z Complete! 2024-08-20T22:14:20.6539253Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:14:20.6540202Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:20.6541575Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:20.9359020Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:14:21.0106807Z + sudo yum install -y nvidia-docker2 2024-08-20T22:14:21.4813854Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:14:21.5036793Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:14:21.5334590Z Dependencies resolved. 2024-08-20T22:14:21.5461011Z Nothing to do. 2024-08-20T22:14:21.5461395Z Complete! 2024-08-20T22:14:21.6369548Z + sudo systemctl restart docker 2024-08-20T22:15:16.1431140Z Tue Aug 20 22:15:16 2024 2024-08-20T22:15:16.1431962Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:16.1432800Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:15:16.1433604Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:16.1437510Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:15:16.1438733Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:15:16.1439610Z | | | MIG M. | 2024-08-20T22:15:16.1440260Z |=========================================+========================+======================| 2024-08-20T22:15:16.1548724Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:15:16.1549964Z | 0% 31C P0 67W / 300W | 0MiB / 23028MiB | 6% Default | 2024-08-20T22:15:16.1550695Z | | | N/A | 2024-08-20T22:15:16.1551536Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:15:16.1552213Z 2024-08-20T22:15:16.1552975Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:16.1553710Z | Processes: | 2024-08-20T22:15:16.1554513Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:15:16.1555293Z | ID ID Usage | 2024-08-20T22:15:16.1555939Z |=========================================================================================| 2024-08-20T22:15:16.1556682Z | No running processes found | 2024-08-20T22:15:16.1557559Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:15:16.7681854Z Command completed after 1 attempt(s). 2024-08-20T22:15:16.7766886Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:16.7767660Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:15:16.7768360Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:15:16.7769006Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:16.7783988Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:16.7784482Z env: 2024-08-20T22:15:16.7784761Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:16.7785203Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:16.7785706Z ##[endgroup] 2024-08-20T22:15:17.0594351Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:17.4786703Z Collecting psutil==5.9.1 2024-08-20T22:15:17.5153655Z Downloading psutil-5.9.1-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (281 kB) 2024-08-20T22:15:17.5659627Z Collecting nvidia-ml-py==11.525.84 2024-08-20T22:15:17.5694561Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2024-08-20T22:15:17.6493850Z Installing collected packages: psutil, nvidia-ml-py 2024-08-20T22:15:17.8056240Z Successfully installed nvidia-ml-py-11.525.84 psutil-5.9.1 2024-08-20T22:15:18.0032448Z Prepare all required actions 2024-08-20T22:15:18.0032968Z Getting action download info 2024-08-20T22:15:18.1137857Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:15:18.3420179Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:15:18.4658662Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:15:18.4659145Z with: 2024-08-20T22:15:18.4659484Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:18.4659933Z s3-bucket: gha-artifacts 2024-08-20T22:15:18.4660263Z env: 2024-08-20T22:15:18.4660536Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:18.4660997Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:18.4661486Z ##[endgroup] 2024-08-20T22:15:18.4696078Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:18.4696546Z with: 2024-08-20T22:15:18.4696958Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:18.4697423Z s3-bucket: gha-artifacts 2024-08-20T22:15:18.4697778Z region: us-east-1 2024-08-20T22:15:18.4698084Z env: 2024-08-20T22:15:18.4698359Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:18.4698815Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:18.4699467Z ##[endgroup] 2024-08-20T22:15:18.9547384Z (node:52500) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:18.9548261Z 2024-08-20T22:15:18.9548531Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:18.9549268Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:18.9550168Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:19.0359295Z Found 1 objects with prefix pytorch/pytorch/10479310721/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:15:19.0360441Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:26.5673233Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:15:26.5680061Z Artifact download has finished successfully 2024-08-20T22:15:26.6050641Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:15:26.6051076Z unzip -o artifacts.zip 2024-08-20T22:15:26.6060121Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:26.6060608Z env: 2024-08-20T22:15:26.6060878Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:26.6061331Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:26.6061809Z ##[endgroup] 2024-08-20T22:15:26.6458447Z Archive: artifacts.zip 2024-08-20T22:15:26.6459946Z creating: dist/ 2024-08-20T22:15:28.8105002Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:28.8105857Z creating: build/custom_test_artifacts/ 2024-08-20T22:15:28.8106546Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:15:28.8107299Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:15:28.8108148Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:28.8114509Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:28.8115610Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:28.8116786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:28.8117968Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:28.8118966Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:28.8120348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:28.8122725Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:28.8123838Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:28.8124849Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:28.8126452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:28.8128320Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:28.8130581Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:28.8132012Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:28.8133513Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:28.8135081Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:28.8136260Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:28.8137286Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:28.8177368Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:28.8218719Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:28.8220586Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:28.8267642Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:28.8269608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:28.8273489Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:28.8277241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:28.8280735Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:28.8282303Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:28.8283901Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:28.8285504Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:28.8287065Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:28.8288464Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:28.8289709Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:28.8290953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:28.8292202Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:28.8293411Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:28.8294637Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:28.8345048Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:28.8406344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:28.8407726Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:28.8408966Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:28.8410058Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:28.8411216Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:28.8412325Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:15:28.8413512Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:28.8414912Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:15:28.8416196Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:15:28.8417398Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:15:28.8418491Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:28.8419841Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:15:28.8420937Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:28.8422028Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:15:28.8423120Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:15:28.8438906Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:15:28.8582942Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:15:28.8584219Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:15:28.8585544Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:15:28.8586989Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:15:28.8588403Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:15:28.8589678Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:15:28.8591024Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:15:28.8592186Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:15:28.8593337Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:15:28.8594483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:15:28.8595608Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:15:28.8612692Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:15:28.8697783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:15:28.8699297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:28.8700625Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:28.8701961Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:15:28.8703092Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:15:28.8704009Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:28.8704919Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:15:28.8705932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:15:28.8707050Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:15:28.8707905Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:15:28.8831408Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:15:28.8894589Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:15:28.8895481Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:15:28.8896195Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:15:28.8897023Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:28.8903643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:28.8904703Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:28.8905936Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:28.8906913Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:28.8907917Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:28.8909214Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:28.8910738Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:28.8911836Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:28.8912860Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:28.8914293Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:28.8915769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:28.8917442Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:28.8918830Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:28.8920291Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:28.8921814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:28.8922983Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:28.8923987Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:28.8963479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:28.9003245Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:28.9004824Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:28.9050695Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:28.9052563Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:28.9054669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:28.9056613Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:28.9058457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:28.9060311Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:28.9062102Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:28.9063891Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:28.9065572Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:28.9066848Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:28.9068089Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:28.9069581Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:28.9071021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:28.9072376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:28.9073752Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:28.9126666Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:28.9187058Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:28.9188638Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:28.9189965Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:28.9191157Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:28.9192376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:28.9193627Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:15:28.9194975Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:15:28.9196555Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:15:28.9198144Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:15:28.9199246Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:15:28.9200350Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:15:28.9201470Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:15:28.9202579Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:15:28.9203686Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:15:28.9204791Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:15:28.9219320Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:15:28.9285806Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:15:28.9288965Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:28.9290360Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:28.9291349Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:15:28.9292362Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:15:28.9293573Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:28.9294760Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:15:28.9295616Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:15:28.9296405Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:15:28.9297227Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:15:28.9345190Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:15:28.9346201Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:15:28.9347009Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:15:28.9348090Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:15:28.9353961Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:15:28.9355531Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:15:28.9356683Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:15:28.9357849Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:15:28.9359174Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:15:28.9360648Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:15:28.9362011Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:15:28.9363098Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:15:28.9364495Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:15:28.9366005Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:15:28.9367438Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:15:28.9368896Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:15:28.9370406Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:15:28.9372000Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:15:28.9373447Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:15:28.9374569Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:15:28.9375643Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:15:28.9413788Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:15:28.9453802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:15:28.9455553Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:15:28.9500654Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:15:28.9502632Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:15:28.9504666Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:15:28.9506730Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:15:28.9508548Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:15:28.9510494Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:15:28.9512441Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:15:28.9514221Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:15:28.9515963Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:15:28.9517308Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:15:28.9518709Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:15:28.9520036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:15:28.9521334Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:15:28.9522586Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:15:28.9523870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:15:28.9576679Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:15:28.9638879Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:15:28.9640575Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:15:28.9641844Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:15:28.9643079Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:15:28.9644414Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:15:28.9645775Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:15:28.9647199Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:28.9648900Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:15:28.9650533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:15:28.9651696Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:15:28.9652870Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:28.9654265Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:15:28.9655810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:28.9657002Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:15:28.9658188Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:15:28.9659489Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:15:28.9785320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:15:28.9786903Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:15:28.9788484Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:15:28.9790217Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:15:28.9791772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:15:28.9793314Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:15:28.9794781Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:15:28.9796016Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:15:28.9797262Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:15:28.9798604Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:15:28.9800161Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:15:28.9815943Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:15:28.9873743Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:15:28.9875486Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:15:28.9876951Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:15:28.9878329Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:15:28.9879810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:15:28.9881018Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:15:28.9882029Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:15:28.9883252Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:15:28.9884383Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:15:28.9885373Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:15:28.9989645Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:15:29.0033033Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:15:29.0033802Z creating: build/lib/ 2024-08-20T22:15:29.0122778Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:15:29.0582093Z inflating: build/lib/libprotobuf.a 2024-08-20T22:15:29.0591350Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:15:29.0599815Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:15:29.0608166Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:15:29.0608820Z inflating: build/lib/libclog.a 2024-08-20T22:15:29.0627650Z inflating: build/lib/libnnpack.a 2024-08-20T22:15:29.0631337Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:15:29.0696350Z inflating: build/lib/libgtest.a 2024-08-20T22:15:29.0772889Z inflating: build/lib/libbenchmark.a 2024-08-20T22:15:29.0837179Z inflating: build/lib/libasmjit.a 2024-08-20T22:15:29.0845109Z inflating: build/lib/libittnotify.a 2024-08-20T22:15:29.0874096Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:15:29.1004820Z inflating: build/lib/libgloo.a 2024-08-20T22:15:29.1025541Z inflating: build/lib/libfmt.a 2024-08-20T22:15:29.1122001Z inflating: build/lib/libc10.so 2024-08-20T22:15:29.1123857Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:15:29.1125129Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:15:29.1634474Z inflating: build/lib/libprotoc.a 2024-08-20T22:15:29.1653521Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:15:29.1671616Z inflating: build/lib/libgmock.a 2024-08-20T22:15:29.1672480Z inflating: build/lib/libgtest_main.a 2024-08-20T22:15:29.1673114Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:15:29.2065571Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:15:30.2399856Z inflating: build/lib/libdnnl.a 2024-08-20T22:15:30.2980513Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:15:30.3039996Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:15:30.3040866Z inflating: build/lib/libgmock_main.a 2024-08-20T22:15:30.4321123Z inflating: build/lib/libfbgemm.a 2024-08-20T22:15:30.4581137Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:15:30.5089685Z inflating: build/lib/libkineto.a 2024-08-20T22:15:30.5275670Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:15:30.5318999Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:15:30.6018257Z inflating: build/lib/libonnx.a 2024-08-20T22:15:33.0799081Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:15:33.0803665Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:15:33.0807929Z inflating: build/lib/libshm.so 2024-08-20T22:15:35.1718972Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:15:35.1720284Z inflating: build/lib/libtorch.so 2024-08-20T22:15:35.1721888Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:15:35.9998398Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:15:36.2014802Z inflating: build/lib/libtorch_python.so 2024-08-20T22:15:36.2089184Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:15:36.2109931Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:15:36.2135822Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:15:36.2162097Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:15:36.2196730Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:15:36.2197359Z creating: build/bin/ 2024-08-20T22:15:36.2248221Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:15:36.2300242Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:15:36.2353516Z inflating: build/bin/c10_Device_test 2024-08-20T22:15:36.2413249Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:15:36.2469627Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:15:36.2519301Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:15:36.2572283Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:15:36.2628900Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:15:36.2685007Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:15:36.2741796Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:15:36.2815064Z inflating: build/bin/c10_cow_test 2024-08-20T22:15:36.2868705Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:15:36.2918070Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:15:36.2968595Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:15:36.3019709Z inflating: build/bin/c10_Half_test 2024-08-20T22:15:36.3078686Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:15:36.3135757Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:15:36.3187839Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:15:36.3244417Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:15:36.3298730Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:15:36.3352013Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:15:36.3403048Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:15:36.3456784Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:15:36.3513839Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:15:36.3565772Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:15:36.3622624Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:15:36.3679699Z inflating: build/bin/c10_complex_test 2024-08-20T22:15:36.3734577Z inflating: build/bin/c10_exception_test 2024-08-20T22:15:36.3786527Z inflating: build/bin/c10_flags_test 2024-08-20T22:15:36.3839266Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:15:36.4008884Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:15:36.4061076Z inflating: build/bin/c10_irange_test 2024-08-20T22:15:36.4115915Z inflating: build/bin/c10_lazy_test 2024-08-20T22:15:36.4173590Z inflating: build/bin/c10_logging_test 2024-08-20T22:15:36.4250680Z inflating: build/bin/c10_optional_test 2024-08-20T22:15:36.4315152Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:15:36.4371494Z inflating: build/bin/c10_registry_test 2024-08-20T22:15:36.4528180Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:15:36.4580120Z inflating: build/bin/c10_ssize_test 2024-08-20T22:15:36.4633658Z inflating: build/bin/c10_string_util_test 2024-08-20T22:15:36.4684447Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:15:36.4743779Z inflating: build/bin/c10_string_view_test 2024-08-20T22:15:36.4792691Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:15:36.4849494Z inflating: build/bin/c10_typeid_test 2024-08-20T22:15:36.5302479Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:15:36.5757028Z inflating: build/bin/protoc 2024-08-20T22:15:36.5810839Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:15:36.5864543Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:15:36.5917651Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:15:36.5970093Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:15:36.6023873Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:15:36.6078304Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:15:36.6128397Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:15:36.6183263Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:15:36.6518425Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:15:36.6869453Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:15:36.7235369Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:15:36.7288561Z inflating: build/bin/BackoffTest 2024-08-20T22:15:36.7342045Z inflating: build/bin/FileStoreTest 2024-08-20T22:15:36.7399248Z inflating: build/bin/TCPStoreTest 2024-08-20T22:15:36.7456993Z inflating: build/bin/HashStoreTest 2024-08-20T22:15:36.7471315Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:15:36.7526702Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:15:36.7531074Z inflating: build/bin/torch_shm_manager 2024-08-20T22:15:36.7534115Z inflating: build/bin/example_allreduce 2024-08-20T22:15:36.7590548Z inflating: build/bin/test_dist_autograd 2024-08-20T22:15:36.7659078Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:15:36.7662485Z inflating: build/bin/parallel_benchmark 2024-08-20T22:15:36.7731259Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:15:36.7740957Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:15:36.8093935Z inflating: build/bin/test_lazy 2024-08-20T22:15:36.9295717Z inflating: build/bin/test_api 2024-08-20T22:15:36.9371746Z inflating: build/bin/Dict_test 2024-08-20T22:15:36.9424743Z inflating: build/bin/Dimname_test 2024-08-20T22:15:36.9489954Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:15:36.9548424Z inflating: build/bin/NamedTensor_test 2024-08-20T22:15:36.9608428Z inflating: build/bin/apply_utils_test 2024-08-20T22:15:36.9669080Z inflating: build/bin/atest 2024-08-20T22:15:36.9733748Z inflating: build/bin/basic 2024-08-20T22:15:36.9791119Z inflating: build/bin/broadcast_test 2024-08-20T22:15:36.9843195Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:15:36.9901829Z inflating: build/bin/cpu_generator_test 2024-08-20T22:15:36.9955786Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:15:37.0050244Z inflating: build/bin/cpu_rng_test 2024-08-20T22:15:37.0100879Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:15:37.0152705Z inflating: build/bin/dlconvertor_test 2024-08-20T22:15:37.0212042Z inflating: build/bin/extension_backend_test 2024-08-20T22:15:37.0269149Z inflating: build/bin/half_test 2024-08-20T22:15:37.0366033Z inflating: build/bin/ivalue_test 2024-08-20T22:15:37.0417114Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:15:37.0472189Z inflating: build/bin/math_kernel_test 2024-08-20T22:15:37.0527925Z inflating: build/bin/memory_format_test 2024-08-20T22:15:37.0583259Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:15:37.0637715Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:15:37.0694623Z inflating: build/bin/native_test 2024-08-20T22:15:37.0746193Z inflating: build/bin/operator_name_test 2024-08-20T22:15:37.0798873Z inflating: build/bin/operators_test 2024-08-20T22:15:37.0850569Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:15:37.0918087Z inflating: build/bin/pow_test 2024-08-20T22:15:37.0975905Z inflating: build/bin/quantized_test 2024-08-20T22:15:37.1026304Z inflating: build/bin/reduce_ops_test 2024-08-20T22:15:37.1077856Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:15:37.1135876Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:15:37.1196491Z inflating: build/bin/scalar_test 2024-08-20T22:15:37.1250729Z inflating: build/bin/StorageUtils_test 2024-08-20T22:15:37.1305596Z inflating: build/bin/stride_properties_test 2024-08-20T22:15:37.1387351Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:15:37.1442602Z inflating: build/bin/test_parallel 2024-08-20T22:15:37.1445675Z inflating: build/bin/thread_init_test 2024-08-20T22:15:37.1502443Z inflating: build/bin/type_ptr_test 2024-08-20T22:15:37.1564952Z inflating: build/bin/type_test 2024-08-20T22:15:37.1618818Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:15:37.1620193Z inflating: build/bin/verify_api_visibility 2024-08-20T22:15:37.1692567Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:15:37.1745685Z inflating: build/bin/weakref_test 2024-08-20T22:15:37.1797308Z inflating: build/bin/wrapdim_test 2024-08-20T22:15:37.1850393Z inflating: build/bin/xla_tensor_test 2024-08-20T22:15:37.1911498Z inflating: build/bin/IListRef_test 2024-08-20T22:15:37.2021058Z inflating: build/bin/List_test 2024-08-20T22:15:37.2090426Z inflating: build/bin/KernelFunction_test 2024-08-20T22:15:37.2214896Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:15:37.2315374Z inflating: build/bin/kernel_function_test 2024-08-20T22:15:37.2448219Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:15:37.2555677Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:15:37.2619038Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:15:37.2719138Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:15:37.2771698Z inflating: build/bin/CppSignature_test 2024-08-20T22:15:37.2821629Z inflating: build/bin/op_allowlist_test 2024-08-20T22:15:37.2879782Z inflating: build/bin/backend_fallback_test 2024-08-20T22:15:37.3188862Z inflating: build/bin/op_registration_test 2024-08-20T22:15:37.3255132Z inflating: build/bin/inline_container_test 2024-08-20T22:15:37.3309291Z inflating: build/bin/cuda_apply_test 2024-08-20T22:15:37.3362813Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:15:37.3422685Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:15:37.3479481Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:15:37.3550597Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:15:37.3610217Z inflating: build/bin/cuda_complex_test 2024-08-20T22:15:37.3660285Z inflating: build/bin/cuda_device_test 2024-08-20T22:15:37.3719949Z inflating: build/bin/cuda_cub_test 2024-08-20T22:15:37.3773181Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:15:37.3840412Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:15:37.3899428Z inflating: build/bin/cuda_generator_test 2024-08-20T22:15:37.3951200Z inflating: build/bin/cuda_half_test 2024-08-20T22:15:37.4004896Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:15:37.4055467Z inflating: build/bin/cuda_optional_test 2024-08-20T22:15:37.4107783Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:15:37.4161197Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:15:37.4212901Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:15:37.4274222Z inflating: build/bin/cuda_stream_test 2024-08-20T22:15:37.4323888Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:15:37.4377412Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:15:37.4392017Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:15:37.4459452Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:15:37.4519135Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:15:37.4584593Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:15:37.4649349Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:15:37.5493308Z inflating: build/bin/test_tensorexpr 2024-08-20T22:15:37.6086041Z inflating: build/bin/test_jit 2024-08-20T22:15:37.6086621Z creating: .additional_ci_files/ 2024-08-20T22:15:37.6149133Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:15:37.6391111Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:15:37.6426228Z ##[group]Run rm artifacts.zip 2024-08-20T22:15:37.6426624Z rm artifacts.zip 2024-08-20T22:15:37.6435506Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:37.6435995Z env: 2024-08-20T22:15:37.6436271Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:37.6436724Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:37.6437202Z ##[endgroup] 2024-08-20T22:15:37.7559691Z ##[group]Run df -H 2024-08-20T22:15:37.7560002Z df -H 2024-08-20T22:15:37.7569226Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:37.7569711Z env: 2024-08-20T22:15:37.7569983Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:37.7570433Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:37.7570917Z ##[endgroup] 2024-08-20T22:15:37.7623382Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:15:37.7624623Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:15:37.7625632Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:15:37.7627691Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:15:37.7628901Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:15:37.7630439Z tmpfs 34G 4.1k 34G 1% /tmp 2024-08-20T22:15:37.7631416Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:15:37.7632142Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:15:37.7672682Z Prepare all required actions 2024-08-20T22:15:37.7673227Z Getting action download info 2024-08-20T22:15:37.9025446Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:15:37.9026179Z with: 2024-08-20T22:15:37.9026438Z env: 2024-08-20T22:15:37.9026709Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:37.9027153Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:37.9027635Z ##[endgroup] 2024-08-20T22:15:37.9087378Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:15:37.9087831Z with: 2024-08-20T22:15:37.9088099Z name: td_results 2024-08-20T22:15:37.9088429Z s3-bucket: gha-artifacts 2024-08-20T22:15:37.9088785Z region: us-east-1 2024-08-20T22:15:37.9089093Z env: 2024-08-20T22:15:37.9089367Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:37.9089823Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:37.9090309Z ##[endgroup] 2024-08-20T22:15:38.3902253Z (node:52520) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:15:38.3903085Z 2024-08-20T22:15:38.3903590Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:15:38.3904309Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:15:38.3905165Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:15:38.4576761Z Found 0 objects with prefix pytorch/pytorch/10479310721/td_results/ 2024-08-20T22:15:38.4584500Z Artifact download has finished successfully 2024-08-20T22:15:38.4908111Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:15:38.4908593Z mkdir -p .additional_ci_files 2024-08-20T22:15:38.4909151Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:15:38.4919138Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:38.4919623Z env: 2024-08-20T22:15:38.4919898Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.4920362Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.4920845Z ##[endgroup] 2024-08-20T22:15:38.4978828Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:15:38.4993026Z ##[error]Process completed with exit code 1. 2024-08-20T22:15:38.5028413Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:15:38.5028891Z .github/scripts/parse_ref.py 2024-08-20T22:15:38.5037023Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:38.5037374Z env: 2024-08-20T22:15:38.5037648Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.5038183Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.5038656Z ##[endgroup] 2024-08-20T22:15:38.5321901Z Prepare all required actions 2024-08-20T22:15:38.5361776Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:15:38.5362237Z with: 2024-08-20T22:15:38.5362806Z github-token: *** 2024-08-20T22:15:38.5363125Z env: 2024-08-20T22:15:38.5363404Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.5363863Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.5364357Z ##[endgroup] 2024-08-20T22:15:38.5382618Z ##[group]Run set -eux 2024-08-20T22:15:38.5383065Z set -eux 2024-08-20T22:15:38.5383790Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:15:38.5392663Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:38.5393146Z env: 2024-08-20T22:15:38.5393419Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:38.5393869Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:38.5394496Z GITHUB_TOKEN: *** 2024-08-20T22:15:38.5394804Z ##[endgroup] 2024-08-20T22:15:38.5424391Z + python3 .github/scripts/get_workflow_job_id.py 10479310721 i-0f9bea9b4b5dbaef9 2024-08-20T22:15:39.2098618Z setting job-id=29026299039 2024-08-20T22:15:39.2099678Z setting job-name=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:39.2322009Z Prepare all required actions 2024-08-20T22:15:39.2322555Z Getting action download info 2024-08-20T22:15:39.3483519Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:15:39.3483993Z with: 2024-08-20T22:15:39.3484455Z github-token: *** 2024-08-20T22:15:39.3490959Z test-matrix: {"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:15:39.3498550Z job-name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:39.3499436Z env: 2024-08-20T22:15:39.3499715Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.3500183Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.3500676Z ##[endgroup] 2024-08-20T22:15:39.3543479Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:15:39.3544041Z with: 2024-08-20T22:15:39.3544316Z shell: bash 2024-08-20T22:15:39.3544621Z timeout_minutes: 10 2024-08-20T22:15:39.3544952Z max_attempts: 5 2024-08-20T22:15:39.3545272Z retry_wait_seconds: 30 2024-08-20T22:15:39.3546386Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:39.3547557Z polling_interval_seconds: 1 2024-08-20T22:15:39.3547935Z warning_on_retry: true 2024-08-20T22:15:39.3548285Z continue_on_error: false 2024-08-20T22:15:39.3548635Z env: 2024-08-20T22:15:39.3548916Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:39.3549364Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:39.3550019Z GITHUB_TOKEN: *** 2024-08-20T22:15:39.3550348Z ##[endgroup] 2024-08-20T22:15:39.4322612Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:15:39.6725056Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:15:39.8192631Z Collecting requests==2.27.1 2024-08-20T22:15:39.8541762Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2024-08-20T22:15:40.0467963Z Collecting pyyaml==6.0.1 2024-08-20T22:15:40.0538189Z Downloading PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (738 kB) 2024-08-20T22:15:40.1271249Z Collecting certifi>=2017.4.17 2024-08-20T22:15:40.1317140Z Downloading certifi-2024.7.4-py3-none-any.whl (162 kB) 2024-08-20T22:15:40.1363969Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:15:40.1371978Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:15:40.4365683Z Collecting charset-normalizer~=2.0.0 2024-08-20T22:15:40.4408114Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2024-08-20T22:15:40.5199862Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2024-08-20T22:15:40.6356154Z Successfully installed certifi-2024.7.4 charset-normalizer-2.0.12 pyyaml-6.0.1 requests-2.27.1 2024-08-20T22:15:41.4113024Z Command completed after 1 attempt(s). 2024-08-20T22:15:41.4173822Z ##[group]Run set -x 2024-08-20T22:15:41.4174151Z set -x 2024-08-20T22:15:41.4174451Z  2024-08-20T22:15:41.4175002Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:41.4175684Z # in runner workspace 2024-08-20T22:15:41.4176223Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:15:41.4185914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.4186413Z env: 2024-08-20T22:15:41.4186880Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.4187328Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.4187819Z ##[endgroup] 2024-08-20T22:15:41.4219171Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:15:41.4466378Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:41.4466930Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:15:41.4467404Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:15:41.4467812Z  2024-08-20T22:15:41.4468368Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:15:41.4469063Z # in runner workspace 2024-08-20T22:15:41.4469667Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:15:41.4470313Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:15:41.4470782Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:15:41.4477534Z  --test-matrix "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:15:41.4484997Z  --selected-test-configs "" \ 2024-08-20T22:15:41.4485465Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:15:41.4486077Z  --tag "${TAG}" \ 2024-08-20T22:15:41.4486480Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:15:41.4486931Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:15:41.4487381Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:15:41.4496312Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.4496816Z env: 2024-08-20T22:15:41.4497096Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.4497558Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.4498246Z GITHUB_TOKEN: *** 2024-08-20T22:15:41.4499080Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:41.4499960Z PR_NUMBER: 2024-08-20T22:15:41.4500286Z TAG: ciflow/inductor/133712 2024-08-20T22:15:41.4500662Z EVENT_NAME: push 2024-08-20T22:15:41.4500979Z SCHEDULE: 2024-08-20T22:15:41.4501272Z HEAD_BRANCH: 2024-08-20T22:15:41.4501584Z ##[endgroup] 2024-08-20T22:15:41.4530979Z Workflow: inductor-periodic 2024-08-20T22:15:41.4532038Z Job name: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:41.6932768Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:15:41.8521636Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:15:41.8522093Z echo "Filtered matrix:" 2024-08-20T22:15:41.8529075Z echo "{"include": [{"config": "dynamo_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamo_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_aot_eager_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:15:41.8536002Z  2024-08-20T22:15:41.8536286Z echo 2024-08-20T22:15:41.8536665Z echo "Is the current job unstable? False" 2024-08-20T22:15:41.8537142Z  2024-08-20T22:15:41.8537408Z echo 2024-08-20T22:15:41.8537772Z echo "Is keep-going label set? False" 2024-08-20T22:15:41.8538219Z  2024-08-20T22:15:41.8538482Z echo 2024-08-20T22:15:41.8538795Z echo "Renabled issues? " 2024-08-20T22:15:41.8547758Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.8548262Z env: 2024-08-20T22:15:41.8548528Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.8548979Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.8549463Z ##[endgroup] 2024-08-20T22:15:41.8579006Z Filtered matrix: 2024-08-20T22:15:41.8587020Z {include: [{config: dynamo_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamo_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_aot_eager_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:15:41.8593869Z 2024-08-20T22:15:41.8594033Z Is the current job unstable? False 2024-08-20T22:15:41.8594310Z 2024-08-20T22:15:41.8594689Z Is keep-going label set? False 2024-08-20T22:15:41.8594951Z 2024-08-20T22:15:41.8595075Z Renabled issues? 2024-08-20T22:15:41.8636643Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:41.8637339Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:15:41.8645614Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:15:41.8646114Z env: 2024-08-20T22:15:41.8646389Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.8646842Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.8647332Z JOB_TIMEOUT: 240 2024-08-20T22:15:41.8647635Z ##[endgroup] 2024-08-20T22:15:41.8726956Z ##[group]Run set -x 2024-08-20T22:15:41.8727367Z set -x 2024-08-20T22:15:41.8727680Z  2024-08-20T22:15:41.8728043Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:15:41.8728592Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:15:41.8729147Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:15:41.8729667Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:15:41.8730095Z else 2024-08-20T22:15:41.8730457Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:41.8730877Z fi 2024-08-20T22:15:41.8731165Z  2024-08-20T22:15:41.8731641Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:15:41.8732404Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:15:41.8733111Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:15:41.8733725Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:15:41.8734196Z container_name=$(docker run \ 2024-08-20T22:15:41.8734619Z  ${GPU_FLAG:-} \ 2024-08-20T22:15:41.8734995Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:15:41.8735403Z  -e PR_NUMBER \ 2024-08-20T22:15:41.8735779Z  -e GITHUB_ACTIONS \ 2024-08-20T22:15:41.8736166Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:15:41.8736572Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:15:41.8736961Z  -e GITHUB_JOB \ 2024-08-20T22:15:41.8737319Z  -e GITHUB_RUN_ID \ 2024-08-20T22:15:41.8737715Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:15:41.8738129Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:15:41.8738530Z  -e JOB_ID \ 2024-08-20T22:15:41.8738867Z  -e JOB_NAME \ 2024-08-20T22:15:41.8739224Z  -e BASE_SHA \ 2024-08-20T22:15:41.8739571Z  -e BRANCH \ 2024-08-20T22:15:41.8739911Z  -e SHA1 \ 2024-08-20T22:15:41.8740250Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:15:41.8740657Z  -e IN_WHEEL_TEST \ 2024-08-20T22:15:41.8741039Z  -e SHARD_NUMBER \ 2024-08-20T22:15:41.8741414Z  -e TEST_CONFIG \ 2024-08-20T22:15:41.8741776Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:15:41.8742175Z  -e REENABLED_ISSUES \ 2024-08-20T22:15:41.8742592Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:15:41.8743066Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:15:41.8743470Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:15:41.8743866Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:15:41.8744247Z  -e NO_TD \ 2024-08-20T22:15:41.8744586Z  -e TD_DISTRIBUTED \ 2024-08-20T22:15:41.8744967Z  -e PR_LABELS \ 2024-08-20T22:15:41.8745379Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:15:41.8745945Z  -e SCCACHE_BUCKET \ 2024-08-20T22:15:41.8746345Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:15:41.8746757Z  -e XLA_CUDA \ 2024-08-20T22:15:41.8747173Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:15:41.8747875Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:15:41.8748392Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:15:41.8748905Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:15:41.8749384Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:15:41.8749846Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:15:41.8750274Z  -e DASHBOARD_TAG \ 2024-08-20T22:15:41.8750760Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:15:41.8751331Z  --security-opt seccomp=unconfined \ 2024-08-20T22:15:41.8751794Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:15:41.8752180Z  --ipc=host \ 2024-08-20T22:15:41.8752546Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:15:41.8753008Z  --tty \ 2024-08-20T22:15:41.8753347Z  --detach \ 2024-08-20T22:15:41.8753837Z  --name="${container_name}" \ 2024-08-20T22:15:41.8754275Z  --user jenkins \ 2024-08-20T22:15:41.8754797Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:15:41.8755382Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:15:41.8755823Z  "${DOCKER_IMAGE}" 2024-08-20T22:15:41.8756180Z ) 2024-08-20T22:15:41.8756590Z # Propagate download.pytorch.org IP to container 2024-08-20T22:15:41.8757510Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:15:41.8758535Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:15:41.8759440Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:15:41.8767947Z shell: /usr/bin/bash -e {0} 2024-08-20T22:15:41.8768302Z env: 2024-08-20T22:15:41.8768577Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:15:41.8769049Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:15:41.8769650Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:15:41.8770156Z PR_NUMBER: 2024-08-20T22:15:41.8770474Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:15:41.8770914Z GITHUB_WORKFLOW: inductor-periodic 2024-08-20T22:15:41.8771331Z GITHUB_JOB: test 2024-08-20T22:15:41.8771646Z GITHUB_RUN_ID: 10479310721 2024-08-20T22:15:41.8772008Z GITHUB_RUN_NUMBER: 54565 2024-08-20T22:15:41.8772364Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:15:41.8772700Z JOB_ID: 29026299039 2024-08-20T22:15:41.8773528Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:15:41.8774397Z BRANCH: 2024-08-20T22:15:41.8774754Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:41.8775287Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:15:41.8775776Z TEST_CONFIG: aot_eager_torchbench 2024-08-20T22:15:41.8776175Z SHARD_NUMBER: 2 2024-08-20T22:15:41.8776489Z NUM_TEST_SHARDS: 2 2024-08-20T22:15:41.8776825Z REENABLED_ISSUES: 2024-08-20T22:15:41.8777164Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:15:41.8777555Z VERBOSE_TEST_LOGS: False 2024-08-20T22:15:41.8777924Z TEST_SHOWLOCALS: False 2024-08-20T22:15:41.8778282Z NO_TEST_TIMEOUT: False 2024-08-20T22:15:41.8778614Z NO_TD: False 2024-08-20T22:15:41.8778921Z TD_DISTRIBUTED: False 2024-08-20T22:15:41.8779348Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:15:41.8779872Z SCCACHE_S3_KEY_PREFIX: inductor-periodic 2024-08-20T22:15:41.8780282Z SHM_SIZE: 2g 2024-08-20T22:15:41.8781363Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:41.8782532Z XLA_CUDA: 2024-08-20T22:15:41.8783066Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:15:41.8783674Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:15:41.8784118Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:15:41.8784629Z DASHBOARD_TAG: 2024-08-20T22:15:41.8785153Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:15:41.8785533Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:15:41.8785913Z ##[endgroup] 2024-08-20T22:15:41.8813472Z + [[ aot_eager_torchbench == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:15:41.8814326Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:15:41.8814846Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:15:41.8822340Z +++ nproc --ignore=2 2024-08-20T22:15:41.8851398Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310721 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:15:53.3789372Z + container_name=9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T22:15:53.3791878Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:15:53.3793601Z + docker exec -i 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:15:53.5395374Z + echo DOCKER_CONTAINER_ID=9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T22:15:53.5400126Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:15:53.5403299Z + docker exec -t 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:15:53.9426977Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:15:54.2711789Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:15:54.2714383Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:15:54.2718030Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:15:54.2721119Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:15:54.2724441Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:15:54.2730715Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:15:54.2750062Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:15:54.2765830Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:15:54.2784292Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:15:54.3179113Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:15:54.9012670Z Installing collected packages: torch 2024-08-20T22:16:04.8766773Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:16:04.8770068Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:16:04.8771152Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:16:04.9585749Z + export TERM=vt100 2024-08-20T22:16:04.9586203Z + TERM=vt100 2024-08-20T22:16:04.9589089Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:16:04.9599502Z + source .ci/pytorch/common.sh 2024-08-20T22:16:04.9602958Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:16:04.9613841Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:16:04.9615865Z +++ declare -f -t trap_add 2024-08-20T22:16:04.9621408Z ++ set -ex 2024-08-20T22:16:04.9621995Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:04.9622648Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:16:04.9623241Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:16:04.9627303Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:16:04.9648736Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:16:04.9649177Z + trap_add cleanup_workspace EXIT 2024-08-20T22:16:04.9649634Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:16:04.9650009Z + shift 2024-08-20T22:16:04.9650293Z + for trap_add_name in "$@" 2024-08-20T22:16:04.9657035Z +++ trap -p EXIT 2024-08-20T22:16:04.9660010Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:16:04.9660515Z +++ extract_trap_cmd 2024-08-20T22:16:04.9661005Z +++ printf '%s\n' '' 2024-08-20T22:16:04.9661425Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:16:04.9664053Z + trap -- ' 2024-08-20T22:16:04.9664505Z cleanup_workspace' EXIT 2024-08-20T22:16:04.9665092Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:16:05.6673622Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:16:05.6696687Z + echo 'Environment variables:' 2024-08-20T22:16:05.6697227Z Environment variables: 2024-08-20T22:16:05.6697649Z + env 2024-08-20T22:16:05.6708900Z INSTALLED_DB=yes 2024-08-20T22:16:05.6710833Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:05.6711410Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:05.6711943Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.6712584Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:05.6713179Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:05.6713767Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:05.6714428Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.6715097Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:05.6715781Z HOSTNAME=9bd17ee7407b 2024-08-20T22:16:05.6716953Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6717968Z GITHUB_ACTION=__self 2024-08-20T22:16:05.6718325Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:05.6723480Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:05.6727992Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:05.6728552Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:05.6728947Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:05.6729387Z TEST_CONFIG=aot_eager_torchbench 2024-08-20T22:16:05.6729885Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:05.6730485Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:05.6731046Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.6731491Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:05.6732061Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:05.6732468Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:05.6732927Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.6733413Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:05.6733874Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:05.6734196Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:05.6734577Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:05.6734963Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.6735411Z XLA_CUDA= 2024-08-20T22:16:05.6735910Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:05.6738197Z *** 2024-08-20T22:16:05.6738493Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:05.6738912Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:05.6739276Z GITHUB_ACTIONS=true 2024-08-20T22:16:05.6739606Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:05.6740117Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:05.6740654Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:05.6741139Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.6741653Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:05.6742131Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.6743061Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.6743819Z UCC_HOME=/usr 2024-08-20T22:16:05.6744158Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:05.6744546Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:05.6744885Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:05.6745348Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:05.6745832Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.6746296Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:05.6746673Z SHARD_NUMBER=2 2024-08-20T22:16:05.6746974Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:05.6747316Z HOME=/var/lib/jenkins 2024-08-20T22:16:05.6747674Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:05.6748114Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:05.6748583Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:05.6749138Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:05.6749601Z CUDA_VERSION=12.1.1 2024-08-20T22:16:05.6750021Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:05.6750450Z NUM_TEST_SHARDS=2 2024-08-20T22:16:05.6750749Z UCX_HOME=/usr 2024-08-20T22:16:05.6751263Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:05.6752339Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6753757Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:05.6755130Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6756253Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:05.6757086Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:05.6757589Z DASHBOARD_TAG= 2024-08-20T22:16:05.6758020Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:05.6758493Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:05.6759031Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:05.6760073Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6760981Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.6761387Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:05.6761761Z PR_NUMBER= 2024-08-20T22:16:05.6762034Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:05.6762365Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:05.6762807Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:05.6763260Z TERM=vt100 2024-08-20T22:16:05.6763585Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:05.6763999Z INSTALLED_VISION=yes 2024-08-20T22:16:05.6764424Z BRANCH= 2024-08-20T22:16:05.6764707Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:05.6765103Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:05.6765526Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:05.6766279Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:05.6767016Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:05.6767499Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:05.6767968Z REENABLED_ISSUES= 2024-08-20T22:16:05.6768260Z SHLVL=1 2024-08-20T22:16:05.6768511Z MAX_JOBS=14 2024-08-20T22:16:05.6768827Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:05.6769180Z NVARCH=x86_64 2024-08-20T22:16:05.6769473Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:05.6769920Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.6770449Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:05.6770944Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:05.6771602Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:05.6772168Z GITHUB_JOB=test 2024-08-20T22:16:05.6772572Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:05.6773125Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:05.6773622Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:05.6773948Z TD_DISTRIBUTED=False 2024-08-20T22:16:05.6774322Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:05.6774752Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:05.6775161Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:05.6775522Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:05.6775865Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:05.6776205Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:05.6777231Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:05.6778281Z GITHUB_BASE_REF= 2024-08-20T22:16:05.6778604Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:05.6778967Z CI=true 2024-08-20T22:16:05.6779310Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.6779749Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:05.6780116Z JOB_ID=29026299039 2024-08-20T22:16:05.6780419Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:05.6780745Z GITHUB_HEAD_REF= 2024-08-20T22:16:05.6781049Z GITHUB_ACTION_REF= 2024-08-20T22:16:05.6781488Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:05.6781942Z TEST_SHOWLOCALS=False 2024-08-20T22:16:05.6782333Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:05.6782739Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:05.6783636Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6784448Z NO_TD=False 2024-08-20T22:16:05.6784748Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:05.6785109Z _=/usr/bin/env 2024-08-20T22:16:05.6785588Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:16:05.6946997Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:16:05.6948496Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:16:05.6949597Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:16:05.6950818Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:16:05.6951523Z + BUILD_DIR=build 2024-08-20T22:16:05.6951856Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:16:05.6952245Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:16:05.6952577Z + SHARD_NUMBER=2 2024-08-20T22:16:05.6952880Z + NUM_TEST_SHARDS=2 2024-08-20T22:16:05.6953199Z + export VALGRIND=ON 2024-08-20T22:16:05.6953516Z + VALGRIND=ON 2024-08-20T22:16:05.6953989Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:16:05.6954478Z + [[ 0 == \1 ]] 2024-08-20T22:16:05.6954777Z + [[ False == \1 ]] 2024-08-20T22:16:05.6955646Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:16:05.6956310Z ++ realpath build/custom_test_artifacts 2024-08-20T22:16:05.6967891Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:16:05.6968804Z + [[ -n '' ]] 2024-08-20T22:16:05.6969173Z + echo 'Environment variables' 2024-08-20T22:16:05.6969552Z Environment variables 2024-08-20T22:16:05.6969876Z + env 2024-08-20T22:16:05.6978088Z INSTALLED_DB=yes 2024-08-20T22:16:05.6978885Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:16:05.6979452Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:16:05.6979998Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.6980818Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:16:05.6981581Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:16:05.6982141Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:16:05.6982753Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.6983341Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:16:05.6983865Z HOSTNAME=9bd17ee7407b 2024-08-20T22:16:05.6984801Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.6985626Z GITHUB_ACTION=__self 2024-08-20T22:16:05.6985976Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:16:05.6990351Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:16:05.6994669Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:16:05.6995181Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:16:05.6995535Z GITHUB_RUN_NUMBER=54565 2024-08-20T22:16:05.6995890Z TEST_CONFIG=aot_eager_torchbench 2024-08-20T22:16:05.6996296Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:16:05.6996770Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:16:05.6997243Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:16:05.6997689Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:16:05.6998215Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:16:05.6998622Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:16:05.6999150Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.6999770Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:16:05.7000352Z GITHUB_REF_TYPE=tag 2024-08-20T22:16:05.7000749Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:16:05.7001160Z NCCL_VERSION=2.17.1-1 2024-08-20T22:16:05.7001569Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.7002206Z XLA_CUDA= 2024-08-20T22:16:05.7002703Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:16:05.7003096Z *** 2024-08-20T22:16:05.7003375Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:16:05.7003773Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:16:05.7004134Z GITHUB_ACTIONS=true 2024-08-20T22:16:05.7004467Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:16:05.7004969Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:16:05.7005515Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:16:05.7005981Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.7006499Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:16:05.7006991Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.7007917Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.7008680Z UCC_HOME=/usr 2024-08-20T22:16:05.7009137Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:16:05.7009592Z VERBOSE_TEST_LOGS=False 2024-08-20T22:16:05.7009935Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:16:05.7010394Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:16:05.7010894Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:16:05.7011357Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:16:05.7011726Z SHARD_NUMBER=2 2024-08-20T22:16:05.7012032Z GITHUB_REF_PROTECTED=false 2024-08-20T22:16:05.7012382Z HOME=/var/lib/jenkins 2024-08-20T22:16:05.7012746Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:16:05.7013176Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:16:05.7013646Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:16:05.7014246Z SCCACHE_S3_KEY_PREFIX=inductor-periodic 2024-08-20T22:16:05.7014665Z CUDA_VERSION=12.1.1 2024-08-20T22:16:05.7015080Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:16:05.7015515Z NUM_TEST_SHARDS=2 2024-08-20T22:16:05.7015815Z UCX_HOME=/usr 2024-08-20T22:16:05.7016336Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:16:05.7017412Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.7018832Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86-periodic-dynamo-benchmarks / test (aot_eager_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:16:05.7020266Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.7021383Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:16:05.7022036Z GITHUB_EVENT_NAME=push 2024-08-20T22:16:05.7022356Z DASHBOARD_TAG= 2024-08-20T22:16:05.7022670Z GITHUB_RUN_ID=10479310721 2024-08-20T22:16:05.7023133Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:16:05.7023665Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:16:05.7024663Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.7025574Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:16:05.7026312Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:16:05.7026693Z PR_NUMBER= 2024-08-20T22:16:05.7026974Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:16:05.7027290Z VALGRIND=ON 2024-08-20T22:16:05.7027587Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:16:05.7028024Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:16:05.7028479Z TERM=vt100 2024-08-20T22:16:05.7028849Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:16:05.7029359Z INSTALLED_VISION=yes 2024-08-20T22:16:05.7029735Z BRANCH= 2024-08-20T22:16:05.7030085Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:16:05.7030578Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:16:05.7031084Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:16:05.7031841Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:16:05.7032592Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:16:05.7033079Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:16:05.7033722Z REENABLED_ISSUES= 2024-08-20T22:16:05.7034018Z SHLVL=1 2024-08-20T22:16:05.7034280Z MAX_JOBS=14 2024-08-20T22:16:05.7034597Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:16:05.7034944Z NVARCH=x86_64 2024-08-20T22:16:05.7035241Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:16:05.7035699Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:16:05.7036232Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:16:05.7036731Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:16:05.7037404Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:16:05.7038066Z GITHUB_JOB=test 2024-08-20T22:16:05.7038463Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:16:05.7039012Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:16:05.7039567Z NO_TEST_TIMEOUT=False 2024-08-20T22:16:05.7039895Z TD_DISTRIBUTED=False 2024-08-20T22:16:05.7040412Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:16:05.7040859Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:16:05.7041281Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:16:05.7041639Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:16:05.7041992Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:16:05.7042344Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:16:05.7043377Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:05.7044426Z GITHUB_BASE_REF= 2024-08-20T22:16:05.7044757Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:16:05.7045128Z CI=true 2024-08-20T22:16:05.7045445Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:16:05.7045856Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:16:05.7046230Z JOB_ID=29026299039 2024-08-20T22:16:05.7046544Z INSTALLED_PROTOBUF=yes 2024-08-20T22:16:05.7046874Z GITHUB_HEAD_REF= 2024-08-20T22:16:05.7047169Z GITHUB_ACTION_REF= 2024-08-20T22:16:05.7047611Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:16:05.7048083Z TEST_SHOWLOCALS=False 2024-08-20T22:16:05.7048467Z GITHUB_WORKFLOW=inductor-periodic 2024-08-20T22:16:05.7048872Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:16:05.7049778Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_eb707060-8ce0-41ba-8f07-fff4e56e1588 2024-08-20T22:16:05.7050641Z NO_TD=False 2024-08-20T22:16:05.7050934Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:16:05.7051299Z _=/usr/bin/env 2024-08-20T22:16:05.7051648Z + echo 'Testing pytorch' 2024-08-20T22:16:05.7051998Z Testing pytorch 2024-08-20T22:16:05.7052319Z + export LANG=C.UTF-8 2024-08-20T22:16:05.7052665Z + LANG=C.UTF-8 2024-08-20T22:16:05.7052961Z + PR_NUMBER= 2024-08-20T22:16:05.7053301Z + [[ aot_eager_torchbench == \d\e\f\a\u\l\t ]] 2024-08-20T22:16:05.7053817Z + [[ aot_eager_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:16:05.7054322Z + [[ aot_eager_torchbench == \s\l\o\w ]] 2024-08-20T22:16:05.7054970Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:16:05.7055664Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:16:05.7056211Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:05.7056677Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:16:05.7057129Z + [[ aot_eager_torchbench == *crossref* ]] 2024-08-20T22:16:05.7057698Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:16:05.7058337Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:16:05.7059055Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:05.7059749Z + pip_install --user ninja==1.10.2 2024-08-20T22:16:05.7060406Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:16:06.8047956Z Collecting ninja==1.10.2 2024-08-20T22:16:06.8236816Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:16:06.9149363Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:16:07.5254854Z Installing collected packages: ninja 2024-08-20T22:16:07.5336621Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:16:07.5337908Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:16:07.5832751Z Successfully installed ninja-1.10.2 2024-08-20T22:16:07.6631440Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.6633571Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:07.6635173Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:07.6636010Z + install_tlparse 2024-08-20T22:16:07.6636412Z + pip_install --user tlparse==0.3.25 2024-08-20T22:16:07.6636990Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:16:08.0958293Z Collecting tlparse==0.3.25 2024-08-20T22:16:08.1135351Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:16:08.1733567Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:16:08.8260135Z Installing collected packages: tlparse 2024-08-20T22:16:08.8596410Z Successfully installed tlparse-0.3.25 2024-08-20T22:16:08.9387168Z ++ python -m site --user-base 2024-08-20T22:16:08.9637150Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:16:08.9639469Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:16:08.9640139Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:16:08.9640822Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:16:08.9641775Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:16:08.9643032Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:16:08.9643738Z + cd test 2024-08-20T22:16:08.9644276Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:16:10.6109534Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:16:10.6110154Z + [[ aot_eager_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:16:10.6114713Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:16:10.6115346Z + [[ aot_eager_torchbench == *pr_time_benchmarks* ]] 2024-08-20T22:16:10.6116035Z + [[ aot_eager_torchbench == *dynamo_eager* ]] 2024-08-20T22:16:10.6116598Z + [[ aot_eager_torchbench == *aot_eager* ]] 2024-08-20T22:16:10.6117295Z + DYNAMO_BENCHMARK_FLAGS+=(--backend aot_eager) 2024-08-20T22:16:10.6117912Z + [[ aot_eager_torchbench == *dynamic* ]] 2024-08-20T22:16:10.6118367Z + [[ aot_eager_torchbench == *cpu* ]] 2024-08-20T22:16:10.6118871Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:16:10.6147910Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:10.6148762Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:16:10.6151117Z + cd test 2024-08-20T22:16:10.6151954Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:16:12.1169519Z PyTorch built with: 2024-08-20T22:16:12.1170553Z - GCC 9.4 2024-08-20T22:16:12.1171084Z - C++ Version: 201703 2024-08-20T22:16:12.1171950Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:12.1173033Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:12.1173730Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:12.1174248Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:16:12.1175093Z - NNPACK is enabled 2024-08-20T22:16:12.1175484Z - CPU capability usage: AVX2 2024-08-20T22:16:12.1175898Z - CUDA Runtime 12.1 2024-08-20T22:16:12.1176423Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:16:12.1177024Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:16:12.1177471Z - Magma 2.6.1 2024-08-20T22:16:12.1184423Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:16:12.1190651Z 2024-08-20T22:16:12.4448618Z + cd test 2024-08-20T22:16:12.4449407Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:16:13.8065270Z ATen/Parallel: 2024-08-20T22:16:13.8065777Z at::get_num_threads() : 8 2024-08-20T22:16:13.8066162Z at::get_num_interop_threads() : 16 2024-08-20T22:16:13.8066609Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:16:13.8067006Z omp_get_max_threads() : 8 2024-08-20T22:16:13.8068076Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:16:13.8068876Z mkl_get_max_threads() : 8 2024-08-20T22:16:13.8069507Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:16:13.8070144Z std::thread::hardware_concurrency() : 16 2024-08-20T22:16:13.8070574Z Environment variables: 2024-08-20T22:16:13.8070926Z OMP_NUM_THREADS : [not set] 2024-08-20T22:16:13.8071301Z MKL_NUM_THREADS : [not set] 2024-08-20T22:16:13.8071678Z ATen parallel backend: OpenMP 2024-08-20T22:16:13.8071934Z 2024-08-20T22:16:14.0878134Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:16:14.0878742Z + [[ aot_eager_torchbench == *backward* ]] 2024-08-20T22:16:14.0879220Z + [[ aot_eager_torchbench == *xla* ]] 2024-08-20T22:16:14.0879665Z + [[ aot_eager_torchbench == *executorch* ]] 2024-08-20T22:16:14.0880178Z + [[ aot_eager_torchbench == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:16:14.0880820Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:16:14.0881437Z + [[ aot_eager_torchbench == distributed ]] 2024-08-20T22:16:14.0882074Z + [[ aot_eager_torchbench == *inductor_distributed* ]] 2024-08-20T22:16:14.0882811Z + [[ aot_eager_torchbench == *inductor-halide* ]] 2024-08-20T22:16:14.0883506Z + [[ aot_eager_torchbench == *inductor-micro-benchmark* ]] 2024-08-20T22:16:14.0884053Z + [[ aot_eager_torchbench == *huggingface* ]] 2024-08-20T22:16:14.0884503Z + [[ aot_eager_torchbench == *timm* ]] 2024-08-20T22:16:14.0884956Z + [[ aot_eager_torchbench == *torchbench* ]] 2024-08-20T22:16:14.0885414Z + [[ aot_eager_torchbench == *cpu* ]] 2024-08-20T22:16:14.0885824Z + install_torchaudio cuda 2024-08-20T22:16:14.0886160Z + local commit 2024-08-20T22:16:14.0886479Z ++ get_pinned_commit audio 2024-08-20T22:16:14.0886853Z ++ cat .github/ci_commit_pins/audio.txt 2024-08-20T22:16:14.0905107Z + commit=b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:14.0905581Z + [[ cuda == \c\u\d\a ]] 2024-08-20T22:16:14.0906019Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:16:14.0906936Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:14.0908383Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:14.4292508Z Collecting git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:14.4296819Z Cloning https://github.com/pytorch/audio.git (to revision b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862) to /tmp/pip-req-build-8c73p8ug 2024-08-20T22:16:14.4325170Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-8c73p8ug 2024-08-20T22:16:15.1741643Z Running command git rev-parse -q --verify 'sha^b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862' 2024-08-20T22:16:15.1770273Z Running command git fetch -q https://github.com/pytorch/audio.git b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:15.5881336Z Resolved https://github.com/pytorch/audio.git to commit b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:16:15.5882468Z Running command git submodule update --init --recursive -q 2024-08-20T22:16:18.0992261Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:16:18.1023214Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.4.0a0+b3f6f51) (2.5.0a0+git40ec5f6) 2024-08-20T22:16:18.1051626Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.13.1) 2024-08-20T22:16:18.1055646Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (4.12.2) 2024-08-20T22:16:18.1057758Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2.8.8) 2024-08-20T22:16:18.1061614Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (3.1.4) 2024-08-20T22:16:18.1064962Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (2024.6.1) 2024-08-20T22:16:18.1070310Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.4.0a0+b3f6f51) (1.13.1) 2024-08-20T22:16:18.1085340Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchaudio==2.4.0a0+b3f6f51) (1.3.0) 2024-08-20T22:16:18.1565631Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.4.0a0+b3f6f51) (2.1.5) 2024-08-20T22:16:18.1628412Z Building wheels for collected packages: torchaudio 2024-08-20T22:18:50.7706554Z Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T22:18:50.7743886Z [?25h Created wheel for torchaudio: filename=torchaudio-2.4.0a0+b3f6f51-cp310-cp310-linux_x86_64.whl size=2310398 sha256=c609acdd9c386408363df0e9d5752de67113c9d234be9fa2df8194d9db9de91c 2024-08-20T22:18:50.7745985Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/b5/02/94/08dddd0295a3ebec4d4a4d9f226adfd865d48fb8ae4af2cc95 2024-08-20T22:18:50.7775694Z Successfully built torchaudio 2024-08-20T22:18:51.2643783Z Installing collected packages: torchaudio 2024-08-20T22:18:51.4773352Z Successfully installed torchaudio-2.4.0a0+b3f6f51 2024-08-20T22:18:51.6617629Z + install_torchtext 2024-08-20T22:18:51.6618091Z + local data_commit 2024-08-20T22:18:51.6618421Z + local text_commit 2024-08-20T22:18:51.6621878Z ++ get_pinned_commit data 2024-08-20T22:18:51.6622293Z ++ cat .github/ci_commit_pins/data.txt 2024-08-20T22:18:51.6645136Z + data_commit=11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:51.6648606Z ++ get_pinned_commit text 2024-08-20T22:18:51.6648999Z ++ cat .github/ci_commit_pins/text.txt 2024-08-20T22:18:51.6664320Z + text_commit=b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:18:51.6665674Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:51.6668036Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.0059179Z Collecting git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.0064291Z Cloning https://github.com/pytorch/data.git (to revision 11bb5b847ea8b9e0d9bb82db3304daf383008d3f) to /tmp/pip-req-build-mtghzpqx 2024-08-20T22:18:52.0101653Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/data.git /tmp/pip-req-build-mtghzpqx 2024-08-20T22:18:52.4447542Z Running command git rev-parse -q --verify 'sha^11bb5b847ea8b9e0d9bb82db3304daf383008d3f' 2024-08-20T22:18:52.4473482Z Running command git fetch -q https://github.com/pytorch/data.git 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.7086231Z Running command git checkout -q 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.8841202Z Resolved https://github.com/pytorch/data.git to commit 11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:18:52.8842451Z Running command git submodule update --init --recursive -q 2024-08-20T22:20:12.6223559Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:20:12.6263065Z [?25hRequirement already satisfied: urllib3>=1.25 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (1.26.19) 2024-08-20T22:20:12.6266125Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.32.3) 2024-08-20T22:20:12.6272005Z Requirement already satisfied: torch>2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchdata==0.7.0a0+11bb5b8) (2.5.0a0+git40ec5f6) 2024-08-20T22:20:12.6303761Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.13.1) 2024-08-20T22:20:12.6307653Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (4.12.2) 2024-08-20T22:20:12.6313340Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.8.8) 2024-08-20T22:20:12.6316521Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (3.1.4) 2024-08-20T22:20:12.6321921Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (2024.6.1) 2024-08-20T22:20:12.6328689Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.13.1) 2024-08-20T22:20:12.6344413Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>2.0->torchdata==0.7.0a0+11bb5b8) (1.3.0) 2024-08-20T22:20:12.6405411Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.3.2) 2024-08-20T22:20:12.6408803Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (3.7) 2024-08-20T22:20:12.6413439Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchdata==0.7.0a0+11bb5b8) (2024.7.4) 2024-08-20T22:20:12.6913571Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>2.0->torchdata==0.7.0a0+11bb5b8) (2.1.5) 2024-08-20T22:20:12.6979616Z Building wheels for collected packages: torchdata 2024-08-20T22:20:14.6651431Z Building wheel for torchdata (setup.py) ... [?25l- \ | done 2024-08-20T22:20:14.6660531Z [?25h Created wheel for torchdata: filename=torchdata-0.7.0a0+11bb5b8-py3-none-any.whl size=182970 sha256=4448e3dc5f15831138b935a862c03940b629ba584b743e8c6ae801dfa1263322 2024-08-20T22:20:14.6662210Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/ca/59/a3/c8250bfc8d3d4d639498d4beb2e0f0e70b9a508ac61fde85ce 2024-08-20T22:20:14.6688554Z Successfully built torchdata 2024-08-20T22:20:15.1515568Z Installing collected packages: torchdata 2024-08-20T22:20:15.2455853Z Successfully installed torchdata-0.7.0a0+11bb5b8 2024-08-20T22:20:20.2894667Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:20.2896161Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:20.6313965Z Collecting git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:20.6319016Z Cloning https://github.com/pytorch/text.git (to revision b0ebddc648d279826089db91775375221777a2db) to /tmp/pip-req-build-cih2lfzj 2024-08-20T22:20:20.6349375Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/text.git /tmp/pip-req-build-cih2lfzj 2024-08-20T22:20:21.4757904Z Running command git rev-parse -q --verify 'sha^b0ebddc648d279826089db91775375221777a2db' 2024-08-20T22:20:21.4787017Z Running command git fetch -q https://github.com/pytorch/text.git b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:21.9760465Z Running command git checkout -q b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:22.1449965Z Resolved https://github.com/pytorch/text.git to commit b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:20:22.1451896Z Running command git submodule update --init --recursive -q 2024-08-20T22:20:29.6836057Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:20:29.6871110Z [?25hRequirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (4.66.5) 2024-08-20T22:20:29.6874803Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.32.3) 2024-08-20T22:20:29.6878930Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (2.5.0a0+git40ec5f6) 2024-08-20T22:20:29.6882844Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (1.22.4) 2024-08-20T22:20:29.6886799Z Requirement already satisfied: torchdata in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchtext==0.16.0a0+b0ebddc) (0.7.0a0+11bb5b8) 2024-08-20T22:20:29.6914319Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.3.2) 2024-08-20T22:20:29.6918974Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (3.7) 2024-08-20T22:20:29.6923956Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (1.26.19) 2024-08-20T22:20:29.6928817Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->torchtext==0.16.0a0+b0ebddc) (2024.7.4) 2024-08-20T22:20:29.6956982Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.13.1) 2024-08-20T22:20:29.6961764Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (4.12.2) 2024-08-20T22:20:29.6965549Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2.8.8) 2024-08-20T22:20:29.6969888Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (3.1.4) 2024-08-20T22:20:29.6973325Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (2024.6.1) 2024-08-20T22:20:29.6979209Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchtext==0.16.0a0+b0ebddc) (1.13.1) 2024-08-20T22:20:29.6994665Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchtext==0.16.0a0+b0ebddc) (1.3.0) 2024-08-20T22:20:29.7613640Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchtext==0.16.0a0+b0ebddc) (2.1.5) 2024-08-20T22:20:29.7683326Z Building wheels for collected packages: torchtext 2024-08-20T22:21:07.6404090Z Building wheel for torchtext (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-08-20T22:21:07.6437056Z [?25h Created wheel for torchtext: filename=torchtext-0.16.0a0+b0ebddc-cp310-cp310-linux_x86_64.whl size=2045465 sha256=7e47cf68a820ea45d0ac710359b380dc075d82230f10b033b5cf14f11c77c23a 2024-08-20T22:21:07.6440689Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/e5/3d/d7/faafad098ec9437ecdf4495c0f5e72b817fffbb06063e32cbf 2024-08-20T22:21:07.6463406Z Successfully built torchtext 2024-08-20T22:21:08.1245444Z Installing collected packages: torchtext 2024-08-20T22:21:08.2384624Z Successfully installed torchtext-0.16.0a0+b0ebddc 2024-08-20T22:21:08.3992650Z + install_torchvision 2024-08-20T22:21:08.3995299Z + local orig_preload 2024-08-20T22:21:08.3995789Z + local commit 2024-08-20T22:21:08.3996198Z ++ get_pinned_commit vision 2024-08-20T22:21:08.3996616Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:08.4014779Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:08.4015313Z + orig_preload= 2024-08-20T22:21:08.4015703Z + '[' -n '' ']' 2024-08-20T22:21:08.4016725Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:08.4019634Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:08.7400310Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:08.7405077Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-xw69arb6 2024-08-20T22:21:08.7435411Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-xw69arb6 2024-08-20T22:21:10.2114881Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:10.2145720Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:11.5646354Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:11.8921405Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:14.4384575Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:14.4417871Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:14.4421787Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:14.4427064Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:14.4505744Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:14.4510298Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:14.4514247Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:14.4516690Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:14.4518555Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:14.4523793Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:14.4538845Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:14.5020470Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:14.5085277Z Building wheels for collected packages: torchvision 2024-08-20T22:22:32.5382514Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done 2024-08-20T22:22:32.5416476Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116123 sha256=8591fd4253130764597c2ec8a5192360eca507d9149f6e2bf035c7fb95da3c26 2024-08-20T22:22:32.5418173Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:32.5454145Z Successfully built torchvision 2024-08-20T22:22:33.0241569Z Installing collected packages: torchvision 2024-08-20T22:22:33.4397562Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:33.5824105Z + '[' -n '' ']' 2024-08-20T22:22:33.5824558Z + TORCH_CUDA_ARCH_LIST='8.0;8.6' 2024-08-20T22:22:33.5828214Z + pip_install git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.5829103Z + pip install --progress-bar off git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.9215276Z Collecting git+https://github.com/pytorch/ao.git 2024-08-20T22:22:33.9220489Z Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-p3yd1cr_ 2024-08-20T22:22:33.9252007Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-p3yd1cr_ 2024-08-20T22:22:34.4697038Z Resolved https://github.com/pytorch/ao.git to commit 2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:22:36.8993390Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:22:36.9028828Z [?25hBuilding wheels for collected packages: torchao 2024-08-20T22:23:55.8508640Z Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - done 2024-08-20T22:23:55.8527999Z [?25h Created wheel for torchao: filename=torchao-0.4.0+git2c8e3f3-cp310-cp310-linux_x86_64.whl size=928019 sha256=d32c9546c32613e69321544bc8ed4d6cdbf9c5bd75c068201d77d3b67822af72 2024-08-20T22:23:55.8529853Z Stored in directory: /tmp/pip-ephem-wheel-cache-1qllz_ju/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e 2024-08-20T22:23:55.8559935Z Successfully built torchao 2024-08-20T22:23:56.3946002Z Installing collected packages: torchao 2024-08-20T22:23:56.6348449Z Successfully installed torchao-0.4.0+git2c8e3f3 2024-08-20T22:23:56.7434762Z + id=1 2024-08-20T22:23:56.7436193Z + pip_install opencv-python==4.8.0.74 2024-08-20T22:23:56.7436811Z + pip install --progress-bar off opencv-python==4.8.0.74 2024-08-20T22:23:57.2122091Z Collecting opencv-python==4.8.0.74 2024-08-20T22:23:57.2309769Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB) 2024-08-20T22:23:57.2378787Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4) 2024-08-20T22:23:57.2427204Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB) 2024-08-20T22:23:58.3192332Z Installing collected packages: opencv-python 2024-08-20T22:23:59.2272642Z Successfully installed opencv-python-4.8.0.74 2024-08-20T22:23:59.3253467Z + [[ aot_eager_torchbench == *inductor_torchbench_smoketest_perf* ]] 2024-08-20T22:23:59.3254202Z + [[ aot_eager_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]] 2024-08-20T22:23:59.3256084Z + [[ aot_eager_torchbench == *torchbench_gcp_smoketest* ]] 2024-08-20T22:23:59.3256750Z + checkout_install_torchbench 2024-08-20T22:23:59.3257257Z + local commit 2024-08-20T22:23:59.3257674Z ++ get_pinned_commit torchbench 2024-08-20T22:23:59.3258137Z ++ cat .github/ci_commit_pins/torchbench.txt 2024-08-20T22:23:59.3276939Z + commit=23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:23:59.3277814Z + git clone https://github.com/pytorch/benchmark torchbench 2024-08-20T22:23:59.3294557Z Cloning into 'torchbench'... 2024-08-20T22:23:59.4803532Z remote: Enumerating objects: 31001, done. 2024-08-20T22:23:59.4804263Z remote: Counting objects: 0% (1/1741) 2024-08-20T22:23:59.4805355Z remote: Counting objects: 1% (18/1741) 2024-08-20T22:23:59.4806001Z remote: Counting objects: 2% (35/1741) 2024-08-20T22:23:59.4806726Z remote: Counting objects: 3% (53/1741) 2024-08-20T22:23:59.4807429Z remote: Counting objects: 4% (70/1741) 2024-08-20T22:23:59.4808102Z remote: Counting objects: 5% (88/1741) 2024-08-20T22:23:59.4808845Z remote: Counting objects: 6% (105/1741) 2024-08-20T22:23:59.4809571Z remote: Counting objects: 7% (122/1741) 2024-08-20T22:23:59.4810135Z remote: Counting objects: 8% (140/1741) 2024-08-20T22:23:59.4813531Z remote: Counting objects: 9% (157/1741) 2024-08-20T22:23:59.4814317Z remote: Counting objects: 10% (175/1741) 2024-08-20T22:23:59.4814892Z remote: Counting objects: 11% (192/1741) 2024-08-20T22:23:59.4815421Z remote: Counting objects: 12% (209/1741) 2024-08-20T22:23:59.4816101Z remote: Counting objects: 13% (227/1741) 2024-08-20T22:23:59.4816816Z remote: Counting objects: 14% (244/1741) 2024-08-20T22:23:59.4817457Z remote: Counting objects: 15% (262/1741) 2024-08-20T22:23:59.4818079Z remote: Counting objects: 16% (279/1741) 2024-08-20T22:23:59.4818669Z remote: Counting objects: 17% (296/1741) 2024-08-20T22:23:59.4819202Z remote: Counting objects: 18% (314/1741) 2024-08-20T22:23:59.4819748Z remote: Counting objects: 19% (331/1741) 2024-08-20T22:23:59.4820461Z remote: Counting objects: 20% (349/1741) 2024-08-20T22:23:59.4821189Z remote: Counting objects: 21% (366/1741) 2024-08-20T22:23:59.4822310Z remote: Counting objects: 22% (384/1741) 2024-08-20T22:23:59.4822978Z remote: Counting objects: 23% (401/1741) 2024-08-20T22:23:59.4823646Z remote: Counting objects: 24% (418/1741) 2024-08-20T22:23:59.4824165Z remote: Counting objects: 25% (436/1741) 2024-08-20T22:23:59.4824685Z remote: Counting objects: 26% (453/1741) 2024-08-20T22:23:59.4825188Z remote: Counting objects: 27% (471/1741) 2024-08-20T22:23:59.4825702Z remote: Counting objects: 28% (488/1741) 2024-08-20T22:23:59.4826562Z remote: Counting objects: 29% (505/1741) 2024-08-20T22:23:59.4827085Z remote: Counting objects: 30% (523/1741) 2024-08-20T22:23:59.4827591Z remote: Counting objects: 31% (540/1741) 2024-08-20T22:23:59.4828104Z remote: Counting objects: 32% (558/1741) 2024-08-20T22:23:59.4828623Z remote: Counting objects: 33% (575/1741) 2024-08-20T22:23:59.4829353Z remote: Counting objects: 34% (592/1741) 2024-08-20T22:23:59.4829869Z remote: Counting objects: 35% (610/1741) 2024-08-20T22:23:59.4830395Z remote: Counting objects: 36% (627/1741) 2024-08-20T22:23:59.4830950Z remote: Counting objects: 37% (645/1741) 2024-08-20T22:23:59.4831502Z remote: Counting objects: 38% (662/1741) 2024-08-20T22:23:59.4832015Z remote: Counting objects: 39% (679/1741) 2024-08-20T22:23:59.4832520Z remote: Counting objects: 40% (697/1741) 2024-08-20T22:23:59.4833037Z remote: Counting objects: 41% (714/1741) 2024-08-20T22:23:59.4833551Z remote: Counting objects: 42% (732/1741) 2024-08-20T22:23:59.4834066Z remote: Counting objects: 43% (749/1741) 2024-08-20T22:23:59.4834567Z remote: Counting objects: 44% (767/1741) 2024-08-20T22:23:59.4835082Z remote: Counting objects: 45% (784/1741) 2024-08-20T22:23:59.4835594Z remote: Counting objects: 46% (801/1741) 2024-08-20T22:23:59.4836098Z remote: Counting objects: 47% (819/1741) 2024-08-20T22:23:59.4836625Z remote: Counting objects: 48% (836/1741) 2024-08-20T22:23:59.4837245Z remote: Counting objects: 49% (854/1741) 2024-08-20T22:23:59.4837778Z remote: Counting objects: 50% (871/1741) 2024-08-20T22:23:59.4838281Z remote: Counting objects: 51% (888/1741) 2024-08-20T22:23:59.4838791Z remote: Counting objects: 52% (906/1741) 2024-08-20T22:23:59.4839304Z remote: Counting objects: 53% (923/1741) 2024-08-20T22:23:59.4839810Z remote: Counting objects: 54% (941/1741) 2024-08-20T22:23:59.4840313Z remote: Counting objects: 55% (958/1741) 2024-08-20T22:23:59.4840826Z remote: Counting objects: 56% (975/1741) 2024-08-20T22:23:59.4841393Z remote: Counting objects: 57% (993/1741) 2024-08-20T22:23:59.4841917Z remote: Counting objects: 58% (1010/1741) 2024-08-20T22:23:59.4842438Z remote: Counting objects: 59% (1028/1741) 2024-08-20T22:23:59.4842963Z remote: Counting objects: 60% (1045/1741) 2024-08-20T22:23:59.4843491Z remote: Counting objects: 61% (1063/1741) 2024-08-20T22:23:59.4844022Z remote: Counting objects: 62% (1080/1741) 2024-08-20T22:23:59.4844545Z remote: Counting objects: 63% (1097/1741) 2024-08-20T22:23:59.4845070Z remote: Counting objects: 64% (1115/1741) 2024-08-20T22:23:59.4845590Z remote: Counting objects: 65% (1132/1741) 2024-08-20T22:23:59.4846103Z remote: Counting objects: 66% (1150/1741) 2024-08-20T22:23:59.4846634Z remote: Counting objects: 67% (1167/1741) 2024-08-20T22:23:59.4847157Z remote: Counting objects: 68% (1184/1741) 2024-08-20T22:23:59.4847680Z remote: Counting objects: 69% (1202/1741) 2024-08-20T22:23:59.4848191Z remote: Counting objects: 70% (1219/1741) 2024-08-20T22:23:59.4848707Z remote: Counting objects: 71% (1237/1741) 2024-08-20T22:23:59.4849232Z remote: Counting objects: 72% (1254/1741) 2024-08-20T22:23:59.4849749Z remote: Counting objects: 73% (1271/1741) 2024-08-20T22:23:59.4850266Z remote: Counting objects: 74% (1289/1741) 2024-08-20T22:23:59.4850818Z remote: Counting objects: 75% (1306/1741) 2024-08-20T22:23:59.4851361Z remote: Counting objects: 76% (1324/1741) 2024-08-20T22:23:59.4852089Z remote: Counting objects: 77% (1341/1741) 2024-08-20T22:23:59.4852605Z remote: Counting objects: 78% (1358/1741) 2024-08-20T22:23:59.4853127Z remote: Counting objects: 79% (1376/1741) 2024-08-20T22:23:59.4853643Z remote: Counting objects: 80% (1393/1741) 2024-08-20T22:23:59.4854163Z remote: Counting objects: 81% (1411/1741) 2024-08-20T22:23:59.4854672Z remote: Counting objects: 82% (1428/1741) 2024-08-20T22:23:59.4855198Z remote: Counting objects: 83% (1446/1741) 2024-08-20T22:23:59.4855729Z remote: Counting objects: 84% (1463/1741) 2024-08-20T22:23:59.4856247Z remote: Counting objects: 85% (1480/1741) 2024-08-20T22:23:59.4856788Z remote: Counting objects: 86% (1498/1741) 2024-08-20T22:23:59.4857330Z remote: Counting objects: 87% (1515/1741) 2024-08-20T22:23:59.4857871Z remote: Counting objects: 88% (1533/1741) 2024-08-20T22:23:59.4858488Z remote: Counting objects: 89% (1550/1741) 2024-08-20T22:23:59.4859028Z remote: Counting objects: 90% (1567/1741) 2024-08-20T22:23:59.4859565Z remote: Counting objects: 91% (1585/1741) 2024-08-20T22:23:59.4860108Z remote: Counting objects: 92% (1602/1741) 2024-08-20T22:23:59.4860641Z remote: Counting objects: 93% (1620/1741) 2024-08-20T22:23:59.4861181Z remote: Counting objects: 94% (1637/1741) 2024-08-20T22:23:59.4861749Z remote: Counting objects: 95% (1654/1741) 2024-08-20T22:23:59.4862285Z remote: Counting objects: 96% (1672/1741) 2024-08-20T22:23:59.4862807Z remote: Counting objects: 97% (1689/1741) 2024-08-20T22:23:59.4863321Z remote: Counting objects: 98% (1707/1741) 2024-08-20T22:23:59.4863849Z remote: Counting objects: 99% (1724/1741) 2024-08-20T22:23:59.4864373Z remote: Counting objects: 100% (1741/1741) 2024-08-20T22:23:59.4864934Z remote: Counting objects: 100% (1741/1741), done. 2024-08-20T22:23:59.4865519Z remote: Compressing objects: 0% (1/900) 2024-08-20T22:23:59.4905903Z remote: Compressing objects: 1% (9/900) 2024-08-20T22:23:59.5019060Z remote: Compressing objects: 2% (18/900) 2024-08-20T22:23:59.5108351Z remote: Compressing objects: 3% (27/900) 2024-08-20T22:23:59.5153970Z remote: Compressing objects: 4% (36/900) 2024-08-20T22:23:59.5201892Z remote: Compressing objects: 5% (45/900) 2024-08-20T22:23:59.5236595Z remote: Compressing objects: 6% (54/900) 2024-08-20T22:23:59.5266467Z remote: Compressing objects: 7% (63/900) 2024-08-20T22:23:59.5298722Z remote: Compressing objects: 8% (72/900) 2024-08-20T22:23:59.5336365Z remote: Compressing objects: 9% (81/900) 2024-08-20T22:23:59.5363225Z remote: Compressing objects: 10% (90/900) 2024-08-20T22:23:59.5378524Z remote: Compressing objects: 11% (99/900) 2024-08-20T22:23:59.5390761Z remote: Compressing objects: 12% (108/900) 2024-08-20T22:23:59.5405420Z remote: Compressing objects: 13% (117/900) 2024-08-20T22:23:59.5424431Z remote: Compressing objects: 14% (126/900) 2024-08-20T22:23:59.5437392Z remote: Compressing objects: 15% (135/900) 2024-08-20T22:23:59.5441316Z remote: Compressing objects: 16% (144/900) 2024-08-20T22:23:59.5445867Z remote: Compressing objects: 17% (153/900) 2024-08-20T22:23:59.5448652Z remote: Compressing objects: 18% (162/900) 2024-08-20T22:23:59.5451600Z remote: Compressing objects: 19% (171/900) 2024-08-20T22:23:59.5454371Z remote: Compressing objects: 20% (180/900) 2024-08-20T22:23:59.5455077Z remote: Compressing objects: 21% (189/900) 2024-08-20T22:23:59.5455645Z remote: Compressing objects: 22% (198/900) 2024-08-20T22:23:59.5458224Z remote: Compressing objects: 23% (207/900) 2024-08-20T22:23:59.5459660Z remote: Compressing objects: 24% (216/900) 2024-08-20T22:23:59.5460669Z remote: Compressing objects: 25% (225/900) 2024-08-20T22:23:59.5464062Z remote: Compressing objects: 26% (234/900) 2024-08-20T22:23:59.5470910Z remote: Compressing objects: 27% (243/900) 2024-08-20T22:23:59.5473946Z remote: Compressing objects: 28% (252/900) 2024-08-20T22:23:59.5477371Z remote: Compressing objects: 29% (261/900) 2024-08-20T22:23:59.5479943Z remote: Compressing objects: 30% (270/900) 2024-08-20T22:23:59.5483114Z remote: Compressing objects: 31% (279/900) 2024-08-20T22:23:59.5486142Z remote: Compressing objects: 32% (288/900) 2024-08-20T22:23:59.5489037Z remote: Compressing objects: 33% (297/900) 2024-08-20T22:23:59.5491871Z remote: Compressing objects: 34% (306/900) 2024-08-20T22:23:59.5494286Z remote: Compressing objects: 35% (315/900) 2024-08-20T22:23:59.5496748Z remote: Compressing objects: 36% (324/900) 2024-08-20T22:23:59.5498900Z remote: Compressing objects: 37% (333/900) 2024-08-20T22:23:59.5501422Z remote: Compressing objects: 38% (342/900) 2024-08-20T22:23:59.5501984Z remote: Compressing objects: 39% (351/900) 2024-08-20T22:23:59.5503864Z remote: Compressing objects: 40% (360/900) 2024-08-20T22:23:59.5505264Z remote: Compressing objects: 41% (369/900) 2024-08-20T22:23:59.5506168Z remote: Compressing objects: 42% (378/900) 2024-08-20T22:23:59.5506883Z remote: Compressing objects: 43% (387/900) 2024-08-20T22:23:59.5507714Z remote: Compressing objects: 44% (396/900) 2024-08-20T22:23:59.5509715Z remote: Compressing objects: 45% (405/900) 2024-08-20T22:23:59.5510486Z remote: Compressing objects: 46% (414/900) 2024-08-20T22:23:59.5511181Z remote: Compressing objects: 47% (423/900) 2024-08-20T22:23:59.5511910Z remote: Compressing objects: 48% (432/900) 2024-08-20T22:23:59.5512645Z remote: Compressing objects: 49% (441/900) 2024-08-20T22:23:59.5513339Z remote: Compressing objects: 50% (450/900) 2024-08-20T22:23:59.5514079Z remote: Compressing objects: 51% (459/900) 2024-08-20T22:23:59.5514782Z remote: Compressing objects: 52% (468/900) 2024-08-20T22:23:59.5515488Z remote: Compressing objects: 53% (477/900) 2024-08-20T22:23:59.5516230Z remote: Compressing objects: 54% (486/900) 2024-08-20T22:23:59.5516971Z remote: Compressing objects: 55% (495/900) 2024-08-20T22:23:59.5517686Z remote: Compressing objects: 56% (504/900) 2024-08-20T22:23:59.5518245Z remote: Compressing objects: 57% (513/900) 2024-08-20T22:23:59.5518800Z remote: Compressing objects: 58% (522/900) 2024-08-20T22:23:59.5519349Z remote: Compressing objects: 59% (531/900) 2024-08-20T22:23:59.5519900Z remote: Compressing objects: 60% (540/900) 2024-08-20T22:23:59.5520439Z remote: Compressing objects: 61% (549/900) 2024-08-20T22:23:59.5520991Z remote: Compressing objects: 62% (558/900) 2024-08-20T22:23:59.5521683Z remote: Compressing objects: 63% (567/900) 2024-08-20T22:23:59.5522429Z remote: Compressing objects: 64% (576/900) 2024-08-20T22:23:59.5523026Z remote: Compressing objects: 65% (585/900) 2024-08-20T22:23:59.5523565Z remote: Compressing objects: 66% (594/900) 2024-08-20T22:23:59.5524108Z remote: Compressing objects: 67% (603/900) 2024-08-20T22:23:59.5524850Z remote: Compressing objects: 68% (612/900) 2024-08-20T22:23:59.5525405Z remote: Compressing objects: 69% (621/900) 2024-08-20T22:23:59.5526228Z remote: Compressing objects: 70% (630/900) 2024-08-20T22:23:59.5526792Z remote: Compressing objects: 71% (639/900) 2024-08-20T22:23:59.5527326Z remote: Compressing objects: 72% (648/900) 2024-08-20T22:23:59.5527861Z remote: Compressing objects: 73% (657/900) 2024-08-20T22:23:59.5528399Z remote: Compressing objects: 74% (666/900) 2024-08-20T22:23:59.5528946Z remote: Compressing objects: 75% (675/900) 2024-08-20T22:23:59.5529806Z remote: Compressing objects: 76% (684/900) 2024-08-20T22:23:59.5537395Z remote: Compressing objects: 77% (693/900) 2024-08-20T22:23:59.5541975Z remote: Compressing objects: 78% (702/900) 2024-08-20T22:23:59.5548441Z remote: Compressing objects: 79% (711/900) 2024-08-20T22:23:59.5554919Z remote: Compressing objects: 80% (720/900) 2024-08-20T22:23:59.5563790Z remote: Compressing objects: 81% (729/900) 2024-08-20T22:23:59.5571637Z remote: Compressing objects: 82% (738/900) 2024-08-20T22:23:59.5574463Z remote: Compressing objects: 83% (747/900) 2024-08-20T22:23:59.5577406Z remote: Compressing objects: 84% (756/900) 2024-08-20T22:23:59.5582250Z remote: Compressing objects: 85% (765/900) 2024-08-20T22:23:59.5586854Z remote: Compressing objects: 86% (774/900) 2024-08-20T22:23:59.5590902Z remote: Compressing objects: 87% (783/900) 2024-08-20T22:23:59.5600440Z remote: Compressing objects: 88% (792/900) 2024-08-20T22:23:59.5601181Z remote: Compressing objects: 89% (801/900) 2024-08-20T22:23:59.5602016Z remote: Compressing objects: 90% (810/900) 2024-08-20T22:23:59.5604721Z remote: Compressing objects: 91% (819/900) 2024-08-20T22:23:59.5608449Z remote: Compressing objects: 92% (828/900) 2024-08-20T22:23:59.5611490Z remote: Compressing objects: 93% (837/900) 2024-08-20T22:23:59.5613550Z remote: Compressing objects: 94% (846/900) 2024-08-20T22:23:59.5616275Z remote: Compressing objects: 95% (855/900) 2024-08-20T22:23:59.5618943Z remote: Compressing objects: 96% (864/900) 2024-08-20T22:23:59.5621925Z remote: Compressing objects: 97% (873/900) 2024-08-20T22:23:59.5623823Z remote: Compressing objects: 98% (882/900) 2024-08-20T22:23:59.5624580Z remote: Compressing objects: 99% (891/900) 2024-08-20T22:23:59.5625327Z remote: Compressing objects: 100% (900/900) 2024-08-20T22:23:59.5626325Z remote: Compressing objects: 100% (900/900), done. 2024-08-20T22:23:59.5705295Z Receiving objects: 0% (1/31001) 2024-08-20T22:23:59.5747451Z Receiving objects: 1% (311/31001) 2024-08-20T22:23:59.5787230Z Receiving objects: 2% (621/31001) 2024-08-20T22:23:59.5823655Z Receiving objects: 3% (931/31001) 2024-08-20T22:23:59.5858516Z Receiving objects: 4% (1241/31001) 2024-08-20T22:23:59.5891911Z Receiving objects: 5% (1551/31001) 2024-08-20T22:23:59.5929778Z Receiving objects: 6% (1861/31001) 2024-08-20T22:23:59.5969953Z Receiving objects: 7% (2171/31001) 2024-08-20T22:23:59.6000474Z Receiving objects: 8% (2481/31001) 2024-08-20T22:23:59.6033955Z Receiving objects: 9% (2791/31001) 2024-08-20T22:23:59.6104308Z Receiving objects: 10% (3101/31001) 2024-08-20T22:23:59.6153756Z Receiving objects: 11% (3411/31001) 2024-08-20T22:23:59.6219933Z Receiving objects: 12% (3721/31001) 2024-08-20T22:23:59.6352262Z Receiving objects: 13% (4031/31001) 2024-08-20T22:23:59.6500514Z Receiving objects: 14% (4341/31001) 2024-08-20T22:23:59.6572401Z Receiving objects: 15% (4651/31001) 2024-08-20T22:23:59.6645945Z Receiving objects: 16% (4961/31001) 2024-08-20T22:23:59.6705270Z Receiving objects: 17% (5271/31001) 2024-08-20T22:23:59.6734037Z Receiving objects: 18% (5581/31001) 2024-08-20T22:23:59.6791917Z Receiving objects: 19% (5891/31001) 2024-08-20T22:23:59.6826591Z Receiving objects: 20% (6201/31001) 2024-08-20T22:23:59.6842401Z Receiving objects: 21% (6511/31001) 2024-08-20T22:23:59.7028987Z Receiving objects: 22% (6821/31001) 2024-08-20T22:24:00.5652037Z Receiving objects: 23% (7131/31001) 2024-08-20T22:24:00.6910048Z Receiving objects: 23% (7375/31001), 75.04 MiB | 75.04 MiB/s 2024-08-20T22:24:01.2458709Z Receiving objects: 24% (7441/31001), 75.04 MiB | 75.04 MiB/s 2024-08-20T22:24:01.3178374Z Receiving objects: 25% (7751/31001), 112.52 MiB | 75.01 MiB/s 2024-08-20T22:24:01.3909582Z Receiving objects: 26% (8061/31001), 112.52 MiB | 75.01 MiB/s 2024-08-20T22:24:01.4625113Z Receiving objects: 27% (8371/31001), 112.52 MiB | 75.01 MiB/s 2024-08-20T22:24:01.5337948Z Receiving objects: 28% (8681/31001), 112.52 MiB | 75.01 MiB/s 2024-08-20T22:24:01.5650945Z Receiving objects: 29% (8991/31001), 112.52 MiB | 75.01 MiB/s 2024-08-20T22:24:01.6060354Z Receiving objects: 29% (9125/31001), 150.50 MiB | 75.29 MiB/s 2024-08-20T22:24:01.6838645Z Receiving objects: 30% (9301/31001), 150.50 MiB | 75.29 MiB/s 2024-08-20T22:24:01.7671784Z Receiving objects: 31% (9611/31001), 150.50 MiB | 75.29 MiB/s 2024-08-20T22:24:02.0029551Z Receiving objects: 32% (9921/31001), 150.50 MiB | 75.29 MiB/s 2024-08-20T22:24:02.0768779Z Receiving objects: 33% (10231/31001), 150.50 MiB | 75.29 MiB/s 2024-08-20T22:24:02.0891995Z Receiving objects: 34% (10541/31001), 183.88 MiB | 73.58 MiB/s 2024-08-20T22:24:02.3507810Z Receiving objects: 35% (10851/31001), 183.88 MiB | 73.58 MiB/s 2024-08-20T22:24:02.5653672Z Receiving objects: 36% (11161/31001), 183.88 MiB | 73.58 MiB/s 2024-08-20T22:24:02.5993806Z Receiving objects: 36% (11297/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6012356Z Receiving objects: 37% (11471/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6034868Z Receiving objects: 38% (11781/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6053270Z Receiving objects: 39% (12091/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6067842Z Receiving objects: 40% (12401/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6079051Z Receiving objects: 41% (12711/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6091780Z Receiving objects: 42% (13021/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6107116Z Receiving objects: 43% (13331/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6123468Z Receiving objects: 44% (13641/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6132054Z Receiving objects: 45% (13951/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6137096Z Receiving objects: 46% (14261/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6151017Z Receiving objects: 47% (14571/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6172333Z Receiving objects: 48% (14881/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6185135Z Receiving objects: 49% (15191/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6384842Z Receiving objects: 50% (15501/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6391197Z Receiving objects: 51% (15811/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6398028Z Receiving objects: 52% (16121/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6404641Z Receiving objects: 53% (16431/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6410464Z Receiving objects: 54% (16741/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6427381Z Receiving objects: 55% (17051/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6438743Z Receiving objects: 56% (17361/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6445107Z Receiving objects: 57% (17671/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6458933Z Receiving objects: 58% (17981/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6471058Z Receiving objects: 59% (18291/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6481387Z Receiving objects: 60% (18601/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6617905Z Receiving objects: 61% (18911/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6624046Z Receiving objects: 62% (19221/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6630943Z Receiving objects: 63% (19531/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6638870Z Receiving objects: 64% (19841/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6646864Z Receiving objects: 65% (20151/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6655967Z Receiving objects: 66% (20461/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.6911989Z Receiving objects: 67% (20771/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7312626Z Receiving objects: 68% (21081/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7320808Z Receiving objects: 69% (21391/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7334897Z Receiving objects: 70% (21701/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7346279Z Receiving objects: 71% (22011/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7361621Z Receiving objects: 72% (22321/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7371399Z Receiving objects: 73% (22631/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7387304Z Receiving objects: 74% (22941/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7398970Z Receiving objects: 75% (23251/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7406453Z Receiving objects: 76% (23561/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7415045Z Receiving objects: 77% (23871/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7634211Z Receiving objects: 78% (24181/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7668388Z Receiving objects: 79% (24491/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7678243Z Receiving objects: 80% (24801/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7691186Z Receiving objects: 81% (25111/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7703948Z Receiving objects: 82% (25421/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7753785Z Receiving objects: 83% (25731/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7894517Z Receiving objects: 84% (26041/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7900054Z Receiving objects: 85% (26351/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7909375Z Receiving objects: 86% (26661/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7914303Z Receiving objects: 87% (26971/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7922911Z Receiving objects: 88% (27281/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.7929475Z Receiving objects: 89% (27591/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8044182Z Receiving objects: 90% (27901/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8058758Z Receiving objects: 91% (28211/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8070861Z Receiving objects: 92% (28521/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8088357Z Receiving objects: 93% (28831/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8122131Z Receiving objects: 94% (29141/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8134685Z Receiving objects: 95% (29451/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8142877Z Receiving objects: 96% (29761/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8148106Z Receiving objects: 97% (30071/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8219180Z Receiving objects: 98% (30381/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8242079Z Receiving objects: 99% (30691/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8243153Z remote: Total 31001 (delta 1012), reused 1473 (delta 822), pack-reused 29260 (from 1) 2024-08-20T22:24:02.8260554Z Receiving objects: 100% (31001/31001), 219.24 MiB | 73.10 MiB/s 2024-08-20T22:24:02.8261236Z Receiving objects: 100% (31001/31001), 232.50 MiB | 71.30 MiB/s, done. 2024-08-20T22:24:02.8293340Z Resolving deltas: 0% (0/16725) 2024-08-20T22:24:02.8308273Z Resolving deltas: 1% (168/16725) 2024-08-20T22:24:02.8318189Z Resolving deltas: 2% (400/16725) 2024-08-20T22:24:02.8342711Z Resolving deltas: 3% (513/16725) 2024-08-20T22:24:02.8358919Z Resolving deltas: 4% (673/16725) 2024-08-20T22:24:02.8368653Z Resolving deltas: 5% (903/16725) 2024-08-20T22:24:02.8378343Z Resolving deltas: 6% (1021/16725) 2024-08-20T22:24:02.8391159Z Resolving deltas: 7% (1172/16725) 2024-08-20T22:24:02.8401048Z Resolving deltas: 8% (1353/16725) 2024-08-20T22:24:02.8413239Z Resolving deltas: 9% (1515/16725) 2024-08-20T22:24:02.8423558Z Resolving deltas: 10% (1677/16725) 2024-08-20T22:24:02.8438588Z Resolving deltas: 11% (1842/16725) 2024-08-20T22:24:02.8451003Z Resolving deltas: 12% (2013/16725) 2024-08-20T22:24:02.8465708Z Resolving deltas: 13% (2176/16725) 2024-08-20T22:24:02.8479578Z Resolving deltas: 14% (2389/16725) 2024-08-20T22:24:02.8495550Z Resolving deltas: 15% (2511/16725) 2024-08-20T22:24:02.8511962Z Resolving deltas: 16% (2680/16725) 2024-08-20T22:24:02.8525776Z Resolving deltas: 17% (2865/16725) 2024-08-20T22:24:02.8575581Z Resolving deltas: 18% (3020/16725) 2024-08-20T22:24:02.8602600Z Resolving deltas: 19% (3197/16725) 2024-08-20T22:24:02.8618310Z Resolving deltas: 21% (3520/16725) 2024-08-20T22:24:02.8634026Z Resolving deltas: 22% (3696/16725) 2024-08-20T22:24:02.8646106Z Resolving deltas: 23% (3897/16725) 2024-08-20T22:24:02.8660769Z Resolving deltas: 24% (4045/16725) 2024-08-20T22:24:02.8699056Z Resolving deltas: 25% (4192/16725) 2024-08-20T22:24:02.8720885Z Resolving deltas: 26% (4359/16725) 2024-08-20T22:24:02.8736539Z Resolving deltas: 27% (4533/16725) 2024-08-20T22:24:02.8755121Z Resolving deltas: 28% (4684/16725) 2024-08-20T22:24:02.8774090Z Resolving deltas: 29% (4851/16725) 2024-08-20T22:24:02.8786878Z Resolving deltas: 30% (5043/16725) 2024-08-20T22:24:02.8797812Z Resolving deltas: 31% (5186/16725) 2024-08-20T22:24:02.8809634Z Resolving deltas: 32% (5353/16725) 2024-08-20T22:24:02.8822657Z Resolving deltas: 33% (5524/16725) 2024-08-20T22:24:02.8845348Z Resolving deltas: 34% (5688/16725) 2024-08-20T22:24:02.8865008Z Resolving deltas: 35% (5858/16725) 2024-08-20T22:24:02.8879983Z Resolving deltas: 36% (6040/16725) 2024-08-20T22:24:02.8889853Z Resolving deltas: 37% (6236/16725) 2024-08-20T22:24:02.8911127Z Resolving deltas: 38% (6357/16725) 2024-08-20T22:24:02.8928270Z Resolving deltas: 39% (6527/16725) 2024-08-20T22:24:02.8945108Z Resolving deltas: 40% (6729/16725) 2024-08-20T22:24:02.8958146Z Resolving deltas: 41% (6888/16725) 2024-08-20T22:24:02.8969002Z Resolving deltas: 42% (7025/16725) 2024-08-20T22:24:02.8978711Z Resolving deltas: 43% (7231/16725) 2024-08-20T22:24:02.8995224Z Resolving deltas: 44% (7367/16725) 2024-08-20T22:24:02.9018186Z Resolving deltas: 45% (7527/16725) 2024-08-20T22:24:02.9027219Z Resolving deltas: 46% (7703/16725) 2024-08-20T22:24:02.9048152Z Resolving deltas: 47% (7864/16725) 2024-08-20T22:24:02.9056397Z Resolving deltas: 48% (8030/16725) 2024-08-20T22:24:02.9065968Z Resolving deltas: 49% (8196/16725) 2024-08-20T22:24:02.9076264Z Resolving deltas: 50% (8364/16725) 2024-08-20T22:24:02.9099558Z Resolving deltas: 51% (8536/16725) 2024-08-20T22:24:02.9099981Z Resolving deltas: 52% (8859/16725) 2024-08-20T22:24:02.9109187Z Resolving deltas: 53% (8866/16725) 2024-08-20T22:24:02.9116074Z Resolving deltas: 54% (9060/16725) 2024-08-20T22:24:02.9131591Z Resolving deltas: 55% (9202/16725) 2024-08-20T22:24:02.9139891Z Resolving deltas: 56% (9366/16725) 2024-08-20T22:24:02.9149607Z Resolving deltas: 57% (9567/16725) 2024-08-20T22:24:02.9152018Z Resolving deltas: 58% (9812/16725) 2024-08-20T22:24:02.9161256Z Resolving deltas: 59% (9868/16725) 2024-08-20T22:24:02.9170996Z Resolving deltas: 60% (10057/16725) 2024-08-20T22:24:02.9179346Z Resolving deltas: 61% (10295/16725) 2024-08-20T22:24:02.9181730Z Resolving deltas: 62% (10493/16725) 2024-08-20T22:24:02.9194353Z Resolving deltas: 63% (10558/16725) 2024-08-20T22:24:02.9213499Z Resolving deltas: 64% (10840/16725) 2024-08-20T22:24:02.9218613Z Resolving deltas: 66% (11125/16725) 2024-08-20T22:24:02.9238971Z Resolving deltas: 67% (11244/16725) 2024-08-20T22:24:02.9248015Z Resolving deltas: 68% (11381/16725) 2024-08-20T22:24:02.9258302Z Resolving deltas: 69% (11575/16725) 2024-08-20T22:24:02.9274096Z Resolving deltas: 70% (11730/16725) 2024-08-20T22:24:02.9287999Z Resolving deltas: 71% (11875/16725) 2024-08-20T22:24:02.9298543Z Resolving deltas: 72% (12066/16725) 2024-08-20T22:24:02.9324604Z Resolving deltas: 73% (12232/16725) 2024-08-20T22:24:02.9330257Z Resolving deltas: 75% (12626/16725) 2024-08-20T22:24:02.9344206Z Resolving deltas: 76% (12712/16725) 2024-08-20T22:24:02.9353025Z Resolving deltas: 77% (12885/16725) 2024-08-20T22:24:02.9370053Z Resolving deltas: 78% (13046/16725) 2024-08-20T22:24:02.9380746Z Resolving deltas: 79% (13222/16725) 2024-08-20T22:24:02.9394439Z Resolving deltas: 80% (13380/16725) 2024-08-20T22:24:02.9404910Z Resolving deltas: 81% (13574/16725) 2024-08-20T22:24:02.9418523Z Resolving deltas: 82% (13715/16725) 2024-08-20T22:24:02.9429570Z Resolving deltas: 83% (13883/16725) 2024-08-20T22:24:02.9437196Z Resolving deltas: 84% (14088/16725) 2024-08-20T22:24:02.9446640Z Resolving deltas: 85% (14224/16725) 2024-08-20T22:24:02.9456727Z Resolving deltas: 86% (14397/16725) 2024-08-20T22:24:02.9466266Z Resolving deltas: 87% (14553/16725) 2024-08-20T22:24:02.9477506Z Resolving deltas: 88% (14722/16725) 2024-08-20T22:24:02.9486605Z Resolving deltas: 89% (14900/16725) 2024-08-20T22:24:02.9496812Z Resolving deltas: 90% (15064/16725) 2024-08-20T22:24:02.9505993Z Resolving deltas: 91% (15220/16725) 2024-08-20T22:24:02.9512244Z Resolving deltas: 92% (15398/16725) 2024-08-20T22:24:02.9520026Z Resolving deltas: 93% (15555/16725) 2024-08-20T22:24:02.9530853Z Resolving deltas: 94% (15725/16725) 2024-08-20T22:24:02.9538709Z Resolving deltas: 95% (15932/16725) 2024-08-20T22:24:02.9547238Z Resolving deltas: 96% (16073/16725) 2024-08-20T22:24:02.9556886Z Resolving deltas: 97% (16262/16725) 2024-08-20T22:24:02.9601823Z Resolving deltas: 98% (16391/16725) 2024-08-20T22:24:02.9678592Z Resolving deltas: 99% (16560/16725) 2024-08-20T22:24:02.9679023Z Resolving deltas: 100% (16725/16725) 2024-08-20T22:24:02.9679456Z Resolving deltas: 100% (16725/16725), done. 2024-08-20T22:24:03.9682096Z + pushd torchbench 2024-08-20T22:24:03.9682514Z ~/workspace/torchbench ~/workspace 2024-08-20T22:24:03.9683353Z + git checkout 23512dbebd44a11eb84afbf53c3c071dd105297e 2024-08-20T22:24:04.0472225Z Note: switching to '23512dbebd44a11eb84afbf53c3c071dd105297e'. 2024-08-20T22:24:04.0472777Z 2024-08-20T22:24:04.0473276Z You are in 'detached HEAD' state. You can look around, make experimental 2024-08-20T22:24:04.0474207Z changes and commit them, and you can discard any commits you make in this 2024-08-20T22:24:04.0474956Z state without impacting any branches by switching back to a branch. 2024-08-20T22:24:04.0475399Z 2024-08-20T22:24:04.0475702Z If you want to create a new branch to retain commits you create, you may 2024-08-20T22:24:04.0476495Z do so (now or later) by using -c with the switch command. Example: 2024-08-20T22:24:04.0476926Z 2024-08-20T22:24:04.0477221Z git switch -c 2024-08-20T22:24:04.0477492Z 2024-08-20T22:24:04.0477628Z Or undo this operation with: 2024-08-20T22:24:04.0477881Z 2024-08-20T22:24:04.0477995Z git switch - 2024-08-20T22:24:04.0478174Z 2024-08-20T22:24:04.0478513Z Turn off this advice by setting config variable advice.detachedHead to false 2024-08-20T22:24:04.0478996Z 2024-08-20T22:24:04.0479173Z HEAD is now at 23512dbe Pin numpy version 2024-08-20T22:24:04.0483049Z + '[' '' ']' 2024-08-20T22:24:04.0483447Z + python install.py --continue_on_fail 2024-08-20T22:24:06.3458616Z checking packages torch, torchvision, torchaudio are installed...OK 2024-08-20T22:24:24.6223355Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK 2024-08-20T22:24:27.7788066Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK 2024-08-20T22:24:29.6792248Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK 2024-08-20T22:24:31.3362079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK 2024-08-20T22:24:31.3601493Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK 2024-08-20T22:24:38.9858498Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK 2024-08-20T22:24:41.7024942Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK 2024-08-20T22:24:44.4227813Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK 2024-08-20T22:24:47.1624634Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK 2024-08-20T22:24:47.1628733Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found 2024-08-20T22:24:48.2233226Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK 2024-08-20T22:24:50.8788967Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK 2024-08-20T22:24:50.9024025Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK 2024-08-20T22:25:48.1513226Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK 2024-08-20T22:25:56.6494767Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK 2024-08-20T22:26:03.7640403Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK 2024-08-20T22:26:10.6200879Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK 2024-08-20T22:26:19.1640658Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK 2024-08-20T22:26:26.1142332Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK 2024-08-20T22:26:33.2032818Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK 2024-08-20T22:26:40.1744208Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK 2024-08-20T22:26:47.2364019Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK 2024-08-20T22:26:55.1000572Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK 2024-08-20T22:27:01.9477506Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK 2024-08-20T22:27:08.9448079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK 2024-08-20T22:27:10.1348452Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK 2024-08-20T22:27:44.5102023Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK 2024-08-20T22:27:49.2820021Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK 2024-08-20T22:27:54.4965569Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK 2024-08-20T22:28:05.4717741Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK 2024-08-20T22:28:06.6564055Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK 2024-08-20T22:28:08.1181678Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK 2024-08-20T22:28:14.6294298Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK 2024-08-20T22:28:21.7693617Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK 2024-08-20T22:28:27.2734611Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK 2024-08-20T22:28:35.1798770Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK 2024-08-20T22:28:41.0865233Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK 2024-08-20T22:28:46.3222552Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK 2024-08-20T22:28:52.5554326Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK 2024-08-20T22:29:09.1609595Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK 2024-08-20T22:29:15.3938514Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK 2024-08-20T22:29:19.3112721Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK 2024-08-20T22:29:24.0922923Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK 2024-08-20T22:29:30.9689422Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK 2024-08-20T22:29:30.9691677Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found 2024-08-20T22:29:44.1720297Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK 2024-08-20T22:29:48.1096477Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK 2024-08-20T22:29:48.1098766Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found 2024-08-20T22:29:55.7474875Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK 2024-08-20T22:29:56.9685284Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK 2024-08-20T22:29:58.2147912Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK 2024-08-20T22:30:43.4566494Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK 2024-08-20T22:32:13.9252279Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK 2024-08-20T22:32:13.9539270Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK 2024-08-20T22:32:15.5017791Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK 2024-08-20T22:32:15.5260220Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK 2024-08-20T22:32:15.5502414Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK 2024-08-20T22:32:15.5740537Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK 2024-08-20T22:32:15.5985431Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK 2024-08-20T22:32:15.6228760Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK 2024-08-20T22:32:36.8098647Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK 2024-08-20T22:32:36.8101175Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found 2024-08-20T22:32:38.0560220Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK 2024-08-20T22:32:40.5065505Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK 2024-08-20T22:32:40.5316488Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK 2024-08-20T22:32:40.5560531Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK 2024-08-20T22:32:40.5806472Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK 2024-08-20T22:32:40.6050393Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK 2024-08-20T22:32:40.6299136Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK 2024-08-20T22:32:47.2867030Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK 2024-08-20T22:32:48.7695442Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK 2024-08-20T22:32:52.2578033Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK 2024-08-20T22:32:52.2822273Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK 2024-08-20T22:32:52.3069938Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK 2024-08-20T22:32:52.3308937Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK 2024-08-20T22:32:52.3554952Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK 2024-08-20T22:32:52.3794170Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK 2024-08-20T22:33:07.4944833Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK 2024-08-20T22:33:25.4272606Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK 2024-08-20T22:33:25.4522289Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK 2024-08-20T22:33:25.4526151Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found 2024-08-20T22:33:25.4528589Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found 2024-08-20T22:33:28.1488256Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK 2024-08-20T22:33:29.8899555Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK 2024-08-20T22:33:29.9146389Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK 2024-08-20T22:33:55.2908725Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK 2024-08-20T22:34:01.9376021Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK 2024-08-20T22:34:05.9403056Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK 2024-08-20T22:34:12.3824583Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK 2024-08-20T22:34:12.4074207Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK 2024-08-20T22:34:12.4321525Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK 2024-08-20T22:34:12.4566708Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK 2024-08-20T22:34:12.4812676Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK 2024-08-20T22:34:12.5054366Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK 2024-08-20T22:34:12.5298750Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK 2024-08-20T22:34:12.5542360Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK 2024-08-20T22:34:19.4976029Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK 2024-08-20T22:34:31.6184011Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK 2024-08-20T22:34:31.6437079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK 2024-08-20T22:34:33.7643079Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK 2024-08-20T22:34:35.7451576Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK 2024-08-20T22:34:36.2327559Z + echo 'Print all dependencies after TorchBench is installed' 2024-08-20T22:34:36.2328291Z Print all dependencies after TorchBench is installed 2024-08-20T22:34:36.2329115Z + python -mpip freeze 2024-08-20T22:34:36.7502793Z absl-py==2.1.0 2024-08-20T22:34:36.7503211Z accelerate==0.33.0 2024-08-20T22:34:36.7503591Z aiohappyeyeballs==2.4.0 2024-08-20T22:34:36.7504054Z aiohttp==3.10.5 2024-08-20T22:34:36.7508745Z aiosignal==1.3.1 2024-08-20T22:34:36.7509451Z annotated-types==0.7.0 2024-08-20T22:34:36.7509922Z antlr4-python3-runtime==4.9.3 2024-08-20T22:34:36.7510357Z anyascii==0.3.2 2024-08-20T22:34:36.7510747Z astroid==3.2.4 2024-08-20T22:34:36.7511131Z astunparse==1.6.3 2024-08-20T22:34:36.7511833Z async-timeout==4.0.3 2024-08-20T22:34:36.7512161Z attrs==23.1.0 2024-08-20T22:34:36.7512444Z audioread==3.0.1 2024-08-20T22:34:36.7517941Z babel==2.16.0 2024-08-20T22:34:36.7518422Z beautifulsoup4==4.12.3 2024-08-20T22:34:36.7520485Z -e git+https://github.com/pytorch/benchmark@23512dbebd44a11eb84afbf53c3c071dd105297e#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch 2024-08-20T22:34:36.7521850Z bibtexparser==2.0.0b7 2024-08-20T22:34:36.7522308Z black==24.8.0 2024-08-20T22:34:36.7522706Z blinker==1.8.2 2024-08-20T22:34:36.7523097Z blis==0.7.11 2024-08-20T22:34:36.7523478Z blobfile==2.1.1 2024-08-20T22:34:36.7523904Z bokeh==3.5.1 2024-08-20T22:34:36.7524333Z boto3==1.19.12 2024-08-20T22:34:36.7524744Z botocore==1.22.12 2024-08-20T22:34:36.7525171Z bs4==0.0.2 2024-08-20T22:34:36.7525557Z cachetools==5.5.0 2024-08-20T22:34:36.7526235Z cardboardlint==1.3.1 2024-08-20T22:34:36.7526702Z catalogue==2.0.10 2024-08-20T22:34:36.7527184Z certifi==2024.7.4 2024-08-20T22:34:36.7527623Z cffi==1.17.0 2024-08-20T22:34:36.7528157Z charset-normalizer==3.3.2 2024-08-20T22:34:36.7528634Z click==8.1.7 2024-08-20T22:34:36.7529025Z clldutils==3.22.2 2024-08-20T22:34:36.7529777Z cloudpathlib==0.18.1 2024-08-20T22:34:36.7530220Z cloudpickle==3.0.0 2024-08-20T22:34:36.7530676Z colorama==0.4.6 2024-08-20T22:34:36.7531089Z colorlog==6.8.2 2024-08-20T22:34:36.7531495Z confection==0.1.5 2024-08-20T22:34:36.7531906Z contourpy==1.2.1 2024-08-20T22:34:36.7532331Z coremltools==5.0b5 2024-08-20T22:34:36.7532759Z cryptography==43.0.0 2024-08-20T22:34:36.7533186Z csvw==3.3.0 2024-08-20T22:34:36.7533558Z cycler==0.12.1 2024-08-20T22:34:36.7533965Z cymem==2.0.8 2024-08-20T22:34:36.7534412Z DALL-E==0.1 2024-08-20T22:34:36.7534785Z datasets==2.21.0 2024-08-20T22:34:36.7535197Z decorator==5.1.1 2024-08-20T22:34:36.7535725Z defusedxml==0.7.1 2024-08-20T22:34:36.7536140Z Deprecated==1.2.14 2024-08-20T22:34:36.7537128Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@1a4df4d954089534daebbd7a0a45f08ece87056e 2024-08-20T22:34:36.7538215Z diffusers==0.20.2 2024-08-20T22:34:36.7538638Z dill==0.3.8 2024-08-20T22:34:36.7539005Z diskcache==5.6.3 2024-08-20T22:34:36.7539420Z distro==1.9.0 2024-08-20T22:34:36.7539791Z dlinfo==1.2.1 2024-08-20T22:34:36.7540275Z docker-pycreds==0.4.0 2024-08-20T22:34:36.7540716Z dominate==2.9.1 2024-08-20T22:34:36.7541835Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881 2024-08-20T22:34:36.7542925Z einops==0.8.0 2024-08-20T22:34:36.7543380Z exceptiongroup==1.2.2 2024-08-20T22:34:36.7543817Z execnet==2.1.1 2024-08-20T22:34:36.7544223Z expecttest==0.1.6 2024-08-20T22:34:36.7544631Z FastNLP==0.6.0 2024-08-20T22:34:36.7545093Z ffmpeg-python==0.2.0 2024-08-20T22:34:36.7545531Z filelock==3.13.1 2024-08-20T22:34:36.7545940Z Flask==3.0.3 2024-08-20T22:34:36.7546326Z flatbuffers==2.0 2024-08-20T22:34:36.7546718Z fonttools==4.53.1 2024-08-20T22:34:36.7547126Z frozenlist==1.4.1 2024-08-20T22:34:36.7547524Z fsspec==2024.6.1 2024-08-20T22:34:36.7547906Z ftfy==6.2.3 2024-08-20T22:34:36.7548759Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74 2024-08-20T22:34:36.7549783Z future==1.0.0 2024-08-20T22:34:36.7550181Z fvcore==0.1.5.post20221221 2024-08-20T22:34:36.7550648Z gdown==5.2.0 2024-08-20T22:34:36.7551029Z geojson==2.5.0 2024-08-20T22:34:36.7551427Z ghstack==0.8.0 2024-08-20T22:34:36.7551800Z gitdb==4.0.11 2024-08-20T22:34:36.7552189Z GitPython==3.1.43 2024-08-20T22:34:36.7552663Z google-auth==2.34.0 2024-08-20T22:34:36.7553171Z google-auth-oauthlib==1.0.0 2024-08-20T22:34:36.7553645Z grpcio==1.65.5 2024-08-20T22:34:36.7554030Z gym==0.26.2 2024-08-20T22:34:36.7554472Z gym-notices==0.0.8 2024-08-20T22:34:36.7554870Z h5py==3.11.0 2024-08-20T22:34:36.7555255Z higher==0.2.1 2024-08-20T22:34:36.7555727Z huggingface-hub==0.24.6 2024-08-20T22:34:36.7556236Z hydra-core==1.3.2 2024-08-20T22:34:36.7556645Z hypothesis==5.35.1 2024-08-20T22:34:36.7557044Z idna==3.7 2024-08-20T22:34:36.7557414Z imageio==2.35.1 2024-08-20T22:34:36.7558019Z importlib_metadata==8.4.0 2024-08-20T22:34:36.7558480Z inflect==7.3.1 2024-08-20T22:34:36.7558872Z iniconfig==2.0.0 2024-08-20T22:34:36.7559280Z iopath==0.1.9 2024-08-20T22:34:36.7559660Z isodate==0.6.1 2024-08-20T22:34:36.7560474Z isort==5.13.2 2024-08-20T22:34:36.7560865Z itsdangerous==2.2.0 2024-08-20T22:34:36.7561285Z Jinja2==3.1.4 2024-08-20T22:34:36.7561650Z jmespath==0.10.0 2024-08-20T22:34:36.7562045Z joblib==1.4.2 2024-08-20T22:34:36.7562431Z jsonpatch==1.33 2024-08-20T22:34:36.7562827Z jsonpointer==3.0.0 2024-08-20T22:34:36.7563241Z jsonschema==4.23.0 2024-08-20T22:34:36.7563794Z jsonschema-specifications==2023.12.1 2024-08-20T22:34:36.7564368Z junitparser==2.1.1 2024-08-20T22:34:36.7564846Z kaldi-io==0.9.8 2024-08-20T22:34:36.7565237Z kiwisolver==1.4.5 2024-08-20T22:34:36.7565636Z kornia==0.7.3 2024-08-20T22:34:36.7566000Z kornia_rs==0.1.5 2024-08-20T22:34:36.7566431Z lameenc==1.7.0 2024-08-20T22:34:36.7566899Z langcodes==3.4.0 2024-08-20T22:34:36.7567398Z langdetect==1.0.9 2024-08-20T22:34:36.7567983Z language-tags==1.2.0 2024-08-20T22:34:36.7568508Z language_data==1.2.0 2024-08-20T22:34:36.7569025Z lark==0.12.0 2024-08-20T22:34:36.7569542Z lazy_loader==0.4 2024-08-20T22:34:36.7569951Z libcst==1.4.0 2024-08-20T22:34:36.7570348Z librosa==0.9.2 2024-08-20T22:34:36.7570734Z lintrunner==0.12.5 2024-08-20T22:34:36.7571125Z llvmlite==0.38.1 2024-08-20T22:34:36.7571514Z lxml==4.9.4 2024-08-20T22:34:36.7571945Z marisa-trie==1.2.0 2024-08-20T22:34:36.7572342Z Markdown==3.7 2024-08-20T22:34:36.7572797Z markdown-it-py==3.0.0 2024-08-20T22:34:36.7573235Z MarkupSafe==2.1.5 2024-08-20T22:34:36.7573646Z matplotlib==3.8.4 2024-08-20T22:34:36.7574034Z mccabe==0.7.0 2024-08-20T22:34:36.7574411Z mdurl==0.1.2 2024-08-20T22:34:36.7574852Z ml-dtypes==0.4.0 2024-08-20T22:34:36.7575258Z MonkeyType==23.3.0 2024-08-20T22:34:36.7575740Z more-itertools==10.4.0 2024-08-20T22:34:36.7576172Z mpmath==1.3.0 2024-08-20T22:34:36.7576618Z msgpack==1.0.8 2024-08-20T22:34:36.7577096Z multidict==6.0.5 2024-08-20T22:34:36.7577609Z multiprocess==0.70.16 2024-08-20T22:34:36.7578158Z murmurhash==1.0.10 2024-08-20T22:34:36.7578671Z musdb==0.4.2 2024-08-20T22:34:36.7579087Z museval==0.4.1 2024-08-20T22:34:36.7579471Z mypy==1.10.0 2024-08-20T22:34:36.7579930Z mypy-extensions==1.0.0 2024-08-20T22:34:36.7580355Z networkx==2.8.8 2024-08-20T22:34:36.7580742Z ninja==1.10.2 2024-08-20T22:34:36.7581123Z nose==1.3.7 2024-08-20T22:34:36.7581483Z numba==0.55.2 2024-08-20T22:34:36.7581853Z numpy==1.21.2 2024-08-20T22:34:36.7582329Z nvidia-cublas-cu12==12.1.3.1 2024-08-20T22:34:36.7582907Z nvidia-cuda-cupti-cu12==12.1.105 2024-08-20T22:34:36.7583508Z nvidia-cuda-nvrtc-cu12==12.1.105 2024-08-20T22:34:36.7584135Z nvidia-cuda-runtime-cu12==12.1.105 2024-08-20T22:34:36.7584745Z nvidia-cudnn-cu12==9.1.0.70 2024-08-20T22:34:36.7585309Z nvidia-cufft-cu12==11.0.2.54 2024-08-20T22:34:36.7585878Z nvidia-curand-cu12==10.3.2.106 2024-08-20T22:34:36.7586475Z nvidia-cusolver-cu12==11.4.5.107 2024-08-20T22:34:36.7587132Z nvidia-cusparse-cu12==12.1.0.106 2024-08-20T22:34:36.7587712Z nvidia-nccl-cu12==2.20.5 2024-08-20T22:34:36.7588255Z nvidia-nvjitlink-cu12==12.6.20 2024-08-20T22:34:36.7588836Z nvidia-nvtx-cu12==12.1.105 2024-08-20T22:34:36.7589308Z oauthlib==3.2.2 2024-08-20T22:34:36.7589699Z omegaconf==2.3.0 2024-08-20T22:34:36.7590105Z onnx==1.16.1 2024-08-20T22:34:36.7590505Z onnxscript==0.1.0.dev20240817 2024-08-20T22:34:36.7590977Z opacus==1.5.2 2024-08-20T22:34:36.7591432Z opencv-python==4.8.0.74 2024-08-20T22:34:36.7591941Z opt-einsum==3.3.0 2024-08-20T22:34:36.7592363Z optree==0.12.1 2024-08-20T22:34:36.7592760Z packaging==24.1 2024-08-20T22:34:36.7593177Z pandas==2.0.3 2024-08-20T22:34:36.7593581Z patch==1.16 2024-08-20T22:34:36.7593968Z pathspec==0.12.1 2024-08-20T22:34:36.7594395Z phonemizer==3.3.0 2024-08-20T22:34:36.7594822Z pillow==10.3.0 2024-08-20T22:34:36.7595237Z platformdirs==4.2.2 2024-08-20T22:34:36.7595670Z pluggy==1.5.0 2024-08-20T22:34:36.7596085Z pooch==1.8.2 2024-08-20T22:34:36.7596558Z portalocker==2.10.1 2024-08-20T22:34:36.7597106Z preshed==3.0.9 2024-08-20T22:34:36.7597608Z prettytable==3.11.0 2024-08-20T22:34:36.7598314Z protobuf==3.20.2 2024-08-20T22:34:36.7598848Z psutil==6.0.0 2024-08-20T22:34:36.7599371Z py-cpuinfo==9.0.0 2024-08-20T22:34:36.7599806Z pyaml==24.7.0 2024-08-20T22:34:36.7600447Z pyarrow==17.0.0 2024-08-20T22:34:36.7600885Z pyasn1==0.6.0 2024-08-20T22:34:36.7601301Z pyasn1_modules==0.4.0 2024-08-20T22:34:36.7601780Z pyclipper==1.3.0.post5 2024-08-20T22:34:36.7602234Z pycocotools==2.0.8 2024-08-20T22:34:36.7602623Z pycparser==2.22 2024-08-20T22:34:36.7603020Z pycryptodomex==3.20.0 2024-08-20T22:34:36.7603432Z pydantic==2.8.2 2024-08-20T22:34:36.7603817Z pydantic_core==2.20.1 2024-08-20T22:34:36.7604214Z pydot==3.0.1 2024-08-20T22:34:36.7604573Z pygame==2.6.0 2024-08-20T22:34:36.7604963Z PyGithub==2.3.0 2024-08-20T22:34:36.7605369Z Pygments==2.15.0 2024-08-20T22:34:36.7605745Z PyJWT==2.9.0 2024-08-20T22:34:36.7606110Z pylatexenc==2.10 2024-08-20T22:34:36.7606515Z pylint==3.2.6 2024-08-20T22:34:36.7606983Z PyNaCl==1.5.0 2024-08-20T22:34:36.7607493Z pynvml==11.5.3 2024-08-20T22:34:36.7608045Z pyparsing==3.1.2 2024-08-20T22:34:36.7608612Z pypdfium2==4.30.0 2024-08-20T22:34:36.7609273Z pysbd==0.3.4 2024-08-20T22:34:36.7609708Z PySocks==1.7.1 2024-08-20T22:34:36.7610157Z pytest==7.3.2 2024-08-20T22:34:36.7610676Z pytest-benchmark==4.0.0 2024-08-20T22:34:36.7611206Z pytest-cpp==2.3.0 2024-08-20T22:34:36.7611721Z pytest-flakefinder==1.1.0 2024-08-20T22:34:36.7612293Z pytest-rerunfailures==14.0 2024-08-20T22:34:36.7612846Z pytest-xdist==3.3.1 2024-08-20T22:34:36.7613375Z python-dateutil==2.9.0.post0 2024-08-20T22:34:36.7614517Z python-doctr @ git+https://github.com/xuzhao9/doctr.git@7f8a3e61bd6480c4e7b7cde47af0aa2153309c56 2024-08-20T22:34:36.7616626Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@de861af9badd03e8b40f5e063d70754a3dc6b4f4 2024-08-20T22:34:36.7618039Z pytz==2024.1 2024-08-20T22:34:36.7618457Z PyWavelets==1.4.1 2024-08-20T22:34:36.7618995Z PyYAML @ file:///croot/pyyaml_1698096049011/work 2024-08-20T22:34:36.7619629Z rapidfuzz==3.9.6 2024-08-20T22:34:36.7620054Z rdflib==7.0.0 2024-08-20T22:34:36.7620462Z redis==5.0.8 2024-08-20T22:34:36.7620858Z referencing==0.35.1 2024-08-20T22:34:36.7621285Z regex==2024.7.24 2024-08-20T22:34:36.7621711Z requests==2.32.3 2024-08-20T22:34:36.7622218Z requests-oauthlib==2.0.0 2024-08-20T22:34:36.7622694Z resampy==0.4.3 2024-08-20T22:34:36.7623103Z rfc3986==1.5.0 2024-08-20T22:34:36.7623502Z rich==13.7.1 2024-08-20T22:34:36.7623899Z rockset==1.0.3 2024-08-20T22:34:36.7624353Z rpds-py==0.20.0 2024-08-20T22:34:36.7624764Z rsa==4.9 2024-08-20T22:34:36.7625146Z s3transfer==0.5.2 2024-08-20T22:34:36.7625566Z safetensors==0.4.4 2024-08-20T22:34:36.7626576Z scikit-image==0.21.0 2024-08-20T22:34:36.7627132Z scikit-learn==1.5.1 2024-08-20T22:34:36.7627567Z scipy==1.10.1 2024-08-20T22:34:36.7627954Z segments==2.2.1 2024-08-20T22:34:36.7628369Z sentencepiece==0.2.0 2024-08-20T22:34:36.7628871Z sentry-sdk==2.13.0 2024-08-20T22:34:36.7629293Z setproctitle==1.3.3 2024-08-20T22:34:36.7629729Z shapely==2.0.6 2024-08-20T22:34:36.7630138Z shellingham==1.5.4 2024-08-20T22:34:36.7630577Z simplejson==3.19.3 2024-08-20T22:34:36.7631097Z six @ file:///tmp/build/80754af9/six_1644875935023/work 2024-08-20T22:34:36.7631804Z smart-open==7.0.4 2024-08-20T22:34:36.7632188Z smmap==5.0.1 2024-08-20T22:34:36.7632593Z sortedcontainers==2.4.0 2024-08-20T22:34:36.7633052Z soundfile==0.12.1 2024-08-20T22:34:36.7633462Z soupsieve==2.6 2024-08-20T22:34:36.7633863Z soxr==0.4.0 2024-08-20T22:34:36.7634231Z spacy==3.7.6 2024-08-20T22:34:36.7634695Z spacy-legacy==3.0.12 2024-08-20T22:34:36.7635209Z spacy-loggers==1.0.5 2024-08-20T22:34:36.7635662Z srsly==2.4.8 2024-08-20T22:34:36.7636058Z stempeg==0.2.3 2024-08-20T22:34:36.7636548Z submitit==1.5.1 2024-08-20T22:34:36.7637067Z sympy==1.13.1 2024-08-20T22:34:36.7637559Z tabulate==0.9.0 2024-08-20T22:34:36.7638289Z tb-nightly==2.13.0a20230426 2024-08-20T22:34:36.7638797Z tensorboard==2.13.0 2024-08-20T22:34:36.7639331Z tensorboard-data-server==0.7.2 2024-08-20T22:34:36.7639857Z tensorboardX==2.6.2.2 2024-08-20T22:34:36.7640304Z termcolor==2.4.0 2024-08-20T22:34:36.7640734Z thinc==8.2.5 2024-08-20T22:34:36.7641138Z threadpoolctl==3.5.0 2024-08-20T22:34:36.7641579Z tifffile==2024.8.10 2024-08-20T22:34:36.7643224Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514 2024-08-20T22:34:36.7644370Z tlparse==0.3.25 2024-08-20T22:34:36.7644787Z tokenizers==0.15.2 2024-08-20T22:34:36.7645214Z tomli==2.0.1 2024-08-20T22:34:36.7645618Z tomlkit==0.13.2 2024-08-20T22:34:36.7647377Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.5.0a0%2Bgit40ec5f6-cp310-cp310-linux_x86_64.whl#sha256=b70a70c0c1fc7cc3f43c6342d758f4ffd3a540a88b786d266604d87618be5127 2024-08-20T22:34:36.7649741Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527 2024-08-20T22:34:36.7651262Z torchao @ git+https://github.com/pytorch/ao.git@2c8e3f368613f896fde14cc24b9268c9e125575e 2024-08-20T22:34:36.7652628Z torchaudio @ git+https://github.com/pytorch/audio.git@b3f6f511f2a1082bd56b13a3f6794e7fc3ba4862 2024-08-20T22:34:36.7654298Z torchdata @ git+https://github.com/pytorch/data.git@11bb5b847ea8b9e0d9bb82db3304daf383008d3f 2024-08-20T22:34:36.7655869Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@e4d288b45b89cee462a21ab264405f3f368adc21 2024-08-20T22:34:36.7657749Z torchtext @ git+https://github.com/pytorch/text.git@b0ebddc648d279826089db91775375221777a2db 2024-08-20T22:34:36.7659424Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:34:36.7660440Z tornado==6.4.1 2024-08-20T22:34:36.7660854Z tqdm==4.66.5 2024-08-20T22:34:36.7661249Z transformers==4.38.1 2024-08-20T22:34:36.7661689Z treetable==0.2.5 2024-08-20T22:34:36.7662846Z -e git+https://github.com/openai/triton@dedb7bdf339a3546896d4820366ca562c586bfa0#egg=triton&subdirectory=python 2024-08-20T22:34:36.7663952Z typeguard==4.3.0 2024-08-20T22:34:36.7664390Z typer==0.12.4 2024-08-20T22:34:36.7664817Z typing_extensions==4.12.2 2024-08-20T22:34:36.7665298Z tzdata==2024.1 2024-08-20T22:34:36.7665708Z Unidecode==1.3.8 2024-08-20T22:34:36.7666251Z unittest-xml-reporting==3.2.0 2024-08-20T22:34:36.7666895Z uritemplate==4.1.1 2024-08-20T22:34:36.7667434Z urllib3==1.26.19 2024-08-20T22:34:36.7667974Z visdom==0.2.4 2024-08-20T22:34:36.7668484Z wandb==0.17.7 2024-08-20T22:34:36.7668904Z wasabi==1.1.3 2024-08-20T22:34:36.7669315Z wcwidth==0.2.13 2024-08-20T22:34:36.7669754Z weasel==0.4.1 2024-08-20T22:34:36.7670261Z websocket-client==1.8.0 2024-08-20T22:34:36.7670684Z Werkzeug==3.0.3 2024-08-20T22:34:36.7671062Z wrapt==1.16.0 2024-08-20T22:34:36.7671423Z xdoctest==1.1.0 2024-08-20T22:34:36.7671792Z xxhash==3.5.0 2024-08-20T22:34:36.7672159Z xyzservices==2024.6.0 2024-08-20T22:34:36.7672561Z yacs==0.1.8 2024-08-20T22:34:36.7672907Z yarl==1.9.4 2024-08-20T22:34:36.7673305Z z3-solver==4.12.2.0 2024-08-20T22:34:36.7673716Z zipp==3.20.0 2024-08-20T22:34:36.7854374Z + popd 2024-08-20T22:34:36.7854708Z ~/workspace 2024-08-20T22:34:36.7855026Z + [[ aot_eager_torchbench != *cpu* ]] 2024-08-20T22:34:36.7855487Z + install_torchrec_and_fbgemm 2024-08-20T22:34:36.7855858Z + local torchrec_commit 2024-08-20T22:34:36.7859008Z ++ get_pinned_commit torchrec 2024-08-20T22:34:36.7859557Z ++ cat .github/ci_commit_pins/torchrec.txt 2024-08-20T22:34:36.7956061Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T22:34:36.7956827Z + local fbgemm_commit 2024-08-20T22:34:36.7959933Z ++ get_pinned_commit fbgemm 2024-08-20T22:34:36.7960531Z ++ cat .github/ci_commit_pins/fbgemm.txt 2024-08-20T22:34:36.7976249Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:36.7977347Z + pip_uninstall torchrec-nightly 2024-08-20T22:34:36.7978058Z + pip uninstall -y torchrec-nightly 2024-08-20T22:34:37.1709386Z WARNING: Skipping torchrec-nightly as it is not installed. 2024-08-20T22:34:37.2063395Z + pip_uninstall fbgemm-gpu-nightly 2024-08-20T22:34:37.2063917Z + pip uninstall -y fbgemm-gpu-nightly 2024-08-20T22:34:37.5763596Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed. 2024-08-20T22:34:37.6113654Z + pip_install setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:34:37.6115028Z + pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions 2024-08-20T22:34:38.1221371Z Collecting setuptools-git-versioning 2024-08-20T22:34:38.1424410Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl.metadata (5.8 kB) 2024-08-20T22:34:38.2576980Z Collecting scikit-build 2024-08-20T22:34:38.2614428Z Downloading scikit_build-0.18.0-py3-none-any.whl.metadata (17 kB) 2024-08-20T22:34:38.3886583Z Collecting pyre-extensions 2024-08-20T22:34:38.3959954Z Downloading pyre_extensions-0.0.30-py3-none-any.whl.metadata (4.0 kB) 2024-08-20T22:34:38.4480852Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.1) 2024-08-20T22:34:38.4483689Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (72.1.0) 2024-08-20T22:34:38.5058116Z Collecting toml>=0.10.2 (from setuptools-git-versioning) 2024-08-20T22:34:38.5096377Z Downloading toml-0.10.2-py2.py3-none-any.whl.metadata (7.1 kB) 2024-08-20T22:34:38.5629559Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0) 2024-08-20T22:34:38.5634590Z Requirement already satisfied: tomli in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (2.0.1) 2024-08-20T22:34:38.5639035Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.43.0) 2024-08-20T22:34:38.6184111Z Collecting typing-inspect (from pyre-extensions) 2024-08-20T22:34:38.6225134Z Downloading typing_inspect-0.9.0-py3-none-any.whl.metadata (1.5 kB) 2024-08-20T22:34:38.6728767Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.12.2) 2024-08-20T22:34:38.6809734Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0) 2024-08-20T22:34:38.6855277Z Downloading setuptools_git_versioning-2.0.0-py3-none-any.whl (10 kB) 2024-08-20T22:34:38.7397048Z Downloading scikit_build-0.18.0-py3-none-any.whl (85 kB) 2024-08-20T22:34:38.7967279Z Downloading pyre_extensions-0.0.30-py3-none-any.whl (12 kB) 2024-08-20T22:34:38.8549044Z Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB) 2024-08-20T22:34:38.9108564Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2024-08-20T22:34:39.7976508Z Installing collected packages: typing-inspect, toml, scikit-build, setuptools-git-versioning, pyre-extensions 2024-08-20T22:34:40.1471966Z Successfully installed pyre-extensions-0.0.30 scikit-build-0.18.0 setuptools-git-versioning-2.0.0 toml-0.10.2 typing-inspect-0.9.0 2024-08-20T22:34:40.2552884Z + CUDA_PATH=/usr/local/cuda-12.1 2024-08-20T22:34:40.2555406Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:34:40.2558597Z + pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu' 2024-08-20T22:34:40.6305439Z Collecting fbgemm-gpu 2024-08-20T22:34:40.6308322Z Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-4851_4pg/fbgemm-gpu_49097f9d32e54dbc9b7dd58e7ce1331f 2024-08-20T22:34:40.6337556Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-4851_4pg/fbgemm-gpu_49097f9d32e54dbc9b7dd58e7ce1331f 2024-08-20T22:34:41.8025493Z Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd' 2024-08-20T22:34:41.8055652Z Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:42.3105648Z Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:42.8605379Z Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd 2024-08-20T22:34:42.8606517Z Running command git submodule update --init --recursive -q 2024-08-20T22:34:52.4623391Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T22:34:52.4643767Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.21.2) 2024-08-20T22:34:52.4650717Z Building wheels for collected packages: fbgemm-gpu 2024-08-20T23:19:39.7343518Z Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done 2024-08-20T23:19:39.9185897Z [?25h Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=252618679 sha256=6c0f5863900fd633455fc1351c6e9c3760ed534bdf785053454afa05aef5d054 2024-08-20T23:19:39.9187566Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39 2024-08-20T23:19:39.9206684Z Successfully built fbgemm-gpu 2024-08-20T23:19:40.7579941Z Installing collected packages: fbgemm-gpu 2024-08-20T23:19:44.1546007Z Successfully installed fbgemm-gpu-0.4.1rc0.post421 2024-08-20T23:19:45.0693096Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:45.0694683Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:45.4384740Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:45.4387562Z Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-e6qr73kf 2024-08-20T23:19:45.4417432Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-e6qr73kf 2024-08-20T23:19:46.1992476Z Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2' 2024-08-20T23:19:46.2019081Z Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:46.5097947Z Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:46.8635389Z Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2 2024-08-20T23:19:47.1431656Z Preparing metadata (setup.py) ... [?25l- done 2024-08-20T23:19:47.1466843Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9) 2024-08-20T23:19:47.1468944Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3) 2024-08-20T23:19:47.1472185Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0) 2024-08-20T23:19:47.1974248Z Collecting torchmetrics (from torchrec==0.3.2) 2024-08-20T23:19:47.2177694Z Downloading torchmetrics-1.4.1-py3-none-any.whl.metadata (20 kB) 2024-08-20T23:19:47.2546490Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.66.5) 2024-08-20T23:19:47.2548846Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421) 2024-08-20T23:19:47.2558871Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.21.2) 2024-08-20T23:19:47.2568319Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (2.10.1) 2024-08-20T23:19:47.2829324Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0) 2024-08-20T23:19:47.2832978Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:19:47.2836346Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2024.1) 2024-08-20T23:19:47.2858552Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.1) 2024-08-20T23:19:47.2863124Z Requirement already satisfied: torch>=1.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.5.0a0+git40ec5f6) 2024-08-20T23:19:47.3006820Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2) 2024-08-20T23:19:47.3042736Z Downloading lightning_utilities-0.11.6-py3-none-any.whl.metadata (5.2 kB) 2024-08-20T23:19:47.3121568Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (72.1.0) 2024-08-20T23:19:47.3125851Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.12.2) 2024-08-20T23:19:47.3149006Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.16.0) 2024-08-20T23:19:47.3184979Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.13.1) 2024-08-20T23:19:47.3188321Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.8.8) 2024-08-20T23:19:47.3193206Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (3.1.4) 2024-08-20T23:19:47.3196776Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2024.6.1) 2024-08-20T23:19:47.3203930Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.13.1) 2024-08-20T23:19:47.3220809Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (1.3.0) 2024-08-20T23:19:47.3637698Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->torchmetrics->torchrec==0.3.2) (2.1.5) 2024-08-20T23:19:47.3778067Z Downloading torchmetrics-1.4.1-py3-none-any.whl (866 kB) 2024-08-20T23:19:47.3963398Z Downloading lightning_utilities-0.11.6-py3-none-any.whl (26 kB) 2024-08-20T23:19:47.4057106Z Building wheels for collected packages: torchrec 2024-08-20T23:19:47.9215698Z Building wheel for torchrec (setup.py) ... [?25l- \ | done 2024-08-20T23:19:47.9224205Z [?25h Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374488 sha256=fd4e21f8d19b4b79f9562bbb9617c50bf809160610191b1d6f66020cfebbb5f1 2024-08-20T23:19:47.9226069Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8 2024-08-20T23:19:47.9248216Z Successfully built torchrec 2024-08-20T23:19:48.6967064Z Installing collected packages: lightning-utilities, torchmetrics, torchrec 2024-08-20T23:19:49.3033805Z Successfully installed lightning-utilities-0.11.6 torchmetrics-1.4.1 torchrec-0.3.2 2024-08-20T23:19:49.4450241Z ++ pwd 2024-08-20T23:19:49.4452175Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench 2024-08-20T23:19:49.4452694Z + test_dynamo_benchmark torchbench 1 2024-08-20T23:19:49.4456877Z ++ pwd 2024-08-20T23:19:49.4459742Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:49.4460289Z + local suite=torchbench 2024-08-20T23:19:49.4460633Z + shift 2024-08-20T23:19:49.4460904Z + local shard_id=1 2024-08-20T23:19:49.4465317Z + shift 2024-08-20T23:19:49.4465725Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:19:49.4466214Z + [[ aot_eager_torchbench == *perf* ]] 2024-08-20T23:19:49.4466638Z + [[ aot_eager_torchbench == *cpu* ]] 2024-08-20T23:19:49.4467211Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:19:49.4468227Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16 2024-08-20T23:19:49.4468811Z ++ pwd 2024-08-20T23:19:49.4469279Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:49.4469947Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:19:49.4513237Z + local name=inference 2024-08-20T23:19:49.4513638Z + shift 2024-08-20T23:19:49.4513978Z + local suite=torchbench 2024-08-20T23:19:49.4514312Z + shift 2024-08-20T23:19:49.4514582Z + local shard_id=1 2024-08-20T23:19:49.4514885Z + shift 2024-08-20T23:19:49.4515156Z + partition_flags=() 2024-08-20T23:19:49.4517232Z + local partition_flags 2024-08-20T23:19:49.4517663Z + [[ -n 2 ]] 2024-08-20T23:19:49.4518089Z + [[ -n 1 ]] 2024-08-20T23:19:49.4518684Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:19:49.4519361Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:19:49.4519849Z + [[ aot_eager_torchbench == *perf* ]] 2024-08-20T23:19:49.4520307Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:19:49.4520778Z + [[ aot_eager_torchbench == *_avx2* ]] 2024-08-20T23:19:49.4521228Z + [[ aot_eager_torchbench == *_avx512* ]] 2024-08-20T23:19:49.4522879Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv 2024-08-20T23:19:55.4056359Z 2024-08-20T23:19:56.1957707Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:19:56.1958310Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:19:56.1991145Z cuda eval llama 2024-08-20T23:20:03.3660284Z pass 2024-08-20T23:20:03.4089676Z TIMING: backend_compile:3.76452 entire_frame_compile:6.53594 total_wall_time:6.53594 2024-08-20T23:20:03.4091222Z STATS: call_* op count: 1058 | FakeTensorMode.__torch_dispatch__:19423 | FakeTensor.__torch_dispatch__:1 | ProxyTorchDispatchMode.__torch_dispatch__:8227 2024-08-20T23:20:03.4092678Z Dynamo produced 1 graphs covering 1058 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:07.2577094Z 2024-08-20T23:20:07.5616811Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:07.5617305Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:07.5617738Z cuda eval llama_v2_7b_16h 2024-08-20T23:20:07.5644525Z Traceback (most recent call last): 2024-08-20T23:20:07.5645336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:20:07.5646181Z ) = runner.load_model( 2024-08-20T23:20:07.5646923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:20:07.5647664Z benchmark = benchmark_cls( 2024-08-20T23:20:07.5648383Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:20:07.5649497Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:20:07.5650419Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 12, in __init__ 2024-08-20T23:20:07.5651630Z super().__init__(name="llama_v2_7b_16h", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:07.5652915Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:20:07.5654091Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:07.5655072Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:20:07.5655812Z self._skip_by_device_name() 2024-08-20T23:20:07.5656623Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:20:07.5657861Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:20:07.5659009Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`. 2024-08-20T23:20:07.5659623Z 2024-08-20T23:20:07.5659750Z model_fail_to_load 2024-08-20T23:20:11.0151568Z 2024-08-20T23:20:11.3198686Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:11.3199316Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:11.3199932Z cuda eval llava 2024-08-20T23:20:11.3207304Z Traceback (most recent call last): 2024-08-20T23:20:11.3208283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:20:11.3209295Z ) = runner.load_model( 2024-08-20T23:20:11.3210267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:20:11.3211361Z benchmark = benchmark_cls( 2024-08-20T23:20:11.3212448Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:20:11.3213644Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:20:11.3214911Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:20:11.3216635Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:11.3218565Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:20:11.3220379Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:20:11.3221905Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:20:11.3223076Z self._skip_by_device_name() 2024-08-20T23:20:11.3224331Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:20:11.3226552Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:20:11.3228560Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:20:11.3229453Z 2024-08-20T23:20:11.3229633Z model_fail_to_load 2024-08-20T23:20:14.7618232Z 2024-08-20T23:20:15.5053634Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:15.5054128Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:15.5060404Z cuda eval maml 2024-08-20T23:20:15.5071805Z pass_due_to_skip 2024-08-20T23:20:15.5080594Z TIMING: 2024-08-20T23:20:15.5080900Z STATS: call_* op count: 0 2024-08-20T23:20:15.5081447Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:19.1434530Z 2024-08-20T23:20:19.7068916Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:19.7069543Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:20:19.7075643Z cuda eval maml_omniglot 2024-08-20T23:20:20.6514175Z pass 2024-08-20T23:20:20.6515196Z TIMING: backend_compile:0.32871 entire_frame_compile:0.72421 total_wall_time:0.72421 2024-08-20T23:20:20.6516230Z STATS: call_* op count: 28 | FakeTensorMode.__torch_dispatch__:1267 | ProxyTorchDispatchMode.__torch_dispatch__:328 2024-08-20T23:20:20.6517185Z Dynamo produced 1 graphs covering 28 ops with 0 graph breaks (0 unique) 2024-08-20T23:20:24.3085872Z 2024-08-20T23:20:24.5247784Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth 2024-08-20T23:20:24.5470963Z 2024-08-20T23:20:24.5471285Z 2024-08-20T23:20:24.6260662Z 0% 0.00/16.9M [00:00 will be ignored 2024-08-20T23:21:28.0053549Z [rank0]:W0820 23:21:28.004000 32792 torch/_dynamo/backends/distributed.py:89] [2/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:21:28.0300635Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:21:28.0305654Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:21:28.0307173Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:21:28.0308642Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:21:28.0310060Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] │ 0 │ 470 │ 44910720 │ 2024-08-20T23:21:28.0311493Z [rank0]:W0820 23:21:28.029000 32792 torch/_dynamo/backends/distributed.py:106] [2/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:21:39.6120765Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:21:39.6122739Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:21:39.6124349Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] or: 2024-08-20T23:21:39.6125570Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:21:39.6127026Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] to include these operations in the captured graph. 2024-08-20T23:21:39.6128112Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:39.6129053Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] Graph break: from user code at: 2024-08-20T23:21:39.6130792Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:21:39.6132490Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] self._dequeue_and_enqueue(k) 2024-08-20T23:21:39.6134616Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:21:39.6136153Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] return func(*args, **kwargs) 2024-08-20T23:21:39.6138126Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:21:39.6139724Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] ptr = int(self.queue_ptr) 2024-08-20T23:21:39.6140803Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:39.6141587Z [rank0]:W0820 23:21:39.611000 32792 torch/_dynamo/variables/tensor.py:776] [5/0] 2024-08-20T23:21:46.8970204Z pass 2024-08-20T23:21:46.8999490Z TIMING: entire_frame_compile:29.96243 backend_compile:9.86162 total_wall_time:29.96243 2024-08-20T23:21:46.9001866Z STATS: call_* op count: 879 | FakeTensorMode.__torch_dispatch__:32694 | FakeTensor.__torch_dispatch__:69 | ProxyTorchDispatchMode.__torch_dispatch__:8866 2024-08-20T23:21:46.9004058Z Dynamo produced 7 graphs covering 879 ops with 5 graph breaks (3 unique) 2024-08-20T23:21:47.5128084Z [rank0]:[W820 23:21:47.829001487 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:21:51.5894116Z 2024-08-20T23:21:51.8932484Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:21:51.8932973Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:21:51.8933416Z cuda eval moondream 2024-08-20T23:21:51.8940522Z Traceback (most recent call last): 2024-08-20T23:21:51.8941334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:21:51.8942169Z ) = runner.load_model( 2024-08-20T23:21:51.8942839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 309, in load_model 2024-08-20T23:21:51.8943561Z benchmark = benchmark_cls( 2024-08-20T23:21:51.8944310Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:21:51.8945086Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:21:51.8945983Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 11, in __init__ 2024-08-20T23:21:51.8947133Z super().__init__(name="moondream", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:21:51.8948405Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:21:51.8949573Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:21:51.8950551Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:21:51.8951297Z self._skip_by_device_name() 2024-08-20T23:21:51.8952104Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:21:51.8953331Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:21:51.8954430Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`. 2024-08-20T23:21:51.8955019Z 2024-08-20T23:21:51.8955460Z model_fail_to_load 2024-08-20T23:21:55.3448985Z 2024-08-20T23:21:57.3274237Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:21:57.5836114Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:21:57.5837044Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:21:57.5838336Z using fused AdamW: True 2024-08-20T23:21:57.9198589Z 2024-08-20T23:21:57.9198913Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:21:57.9237656Z cuda eval nanogpt 2024-08-20T23:22:03.9403694Z pass 2024-08-20T23:22:03.9410741Z TIMING: backend_compile:2.74777 entire_frame_compile:5.43306 total_wall_time:5.43306 2024-08-20T23:22:03.9412025Z STATS: call_* op count: 786 | FakeTensorMode.__torch_dispatch__:15263 | FakeTensor.__torch_dispatch__:216 | ProxyTorchDispatchMode.__torch_dispatch__:6106 2024-08-20T23:22:03.9413698Z Dynamo produced 1 graphs covering 786 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:07.6414138Z 2024-08-20T23:22:10.0692396Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:10.0693004Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:22:10.0695347Z cuda eval nvidia_deeprecommender 2024-08-20T23:22:10.9052501Z pass 2024-08-20T23:22:10.9087677Z TIMING: backend_compile:0.19792 entire_frame_compile:0.52711 total_wall_time:0.52711 2024-08-20T23:22:10.9088911Z STATS: call_* op count: 26 | FakeTensorMode.__torch_dispatch__:731 | ProxyTorchDispatchMode.__torch_dispatch__:242 2024-08-20T23:22:10.9089871Z Dynamo produced 1 graphs covering 26 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:14.5447370Z 2024-08-20T23:22:15.4920791Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:15.4921306Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:15.4938326Z cuda eval opacus_cifar10 2024-08-20T23:22:18.9794331Z pass 2024-08-20T23:22:18.9795100Z TIMING: backend_compile:0.8682 entire_frame_compile:3.13321 total_wall_time:3.13321 2024-08-20T23:22:18.9796120Z STATS: call_* op count: 138 | FakeTensorMode.__torch_dispatch__:2797 | ProxyTorchDispatchMode.__torch_dispatch__:1188 2024-08-20T23:22:18.9797118Z Dynamo produced 1 graphs covering 138 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:22.6887380Z 2024-08-20T23:22:23.3445250Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:23.3446106Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:23.3509215Z cuda eval phlippe_densenet 2024-08-20T23:22:33.2911964Z pass 2024-08-20T23:22:33.2914266Z TIMING: backend_compile:4.26175 entire_frame_compile:9.55196 total_wall_time:9.55196 2024-08-20T23:22:33.2915308Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:18197 | ProxyTorchDispatchMode.__torch_dispatch__:4433 2024-08-20T23:22:33.2916288Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:37.0940528Z 2024-08-20T23:22:37.6619183Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:37.6619732Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:37.6644778Z cuda eval phlippe_resnet 2024-08-20T23:22:41.2298834Z pass 2024-08-20T23:22:41.2299710Z TIMING: backend_compile:1.49409 entire_frame_compile:3.28777 total_wall_time:3.28777 2024-08-20T23:22:41.2301235Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:6793 | ProxyTorchDispatchMode.__torch_dispatch__:1681 2024-08-20T23:22:41.2302679Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:44.9217465Z 2024-08-20T23:22:45.2774718Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:45.2775249Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:45.2775785Z cuda eval pyhpc_equation_of_state 2024-08-20T23:22:47.2240253Z pass 2024-08-20T23:22:47.2241049Z TIMING: backend_compile:1.19153 entire_frame_compile:1.74668 total_wall_time:1.74668 2024-08-20T23:22:47.2242172Z STATS: call_* op count: 732 | FakeTensorMode.__torch_dispatch__:8227 | ProxyTorchDispatchMode.__torch_dispatch__:3732 2024-08-20T23:22:47.2243147Z Dynamo produced 1 graphs covering 732 ops with 0 graph breaks (0 unique) 2024-08-20T23:22:50.8550981Z 2024-08-20T23:22:51.2037165Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:51.2037910Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:51.2040284Z cuda eval pyhpc_isoneutral_mixing 2024-08-20T23:23:00.2297742Z pass 2024-08-20T23:23:00.2298524Z TIMING: backend_compile:6.72676 entire_frame_compile:8.72952 total_wall_time:8.72952 2024-08-20T23:23:00.2299777Z STATS: call_* op count: 1488 | FakeTensorMode.__torch_dispatch__:32445 | FakeTensor.__torch_dispatch__:18 | ProxyTorchDispatchMode.__torch_dispatch__:13834 2024-08-20T23:23:00.2300893Z Dynamo produced 1 graphs covering 1488 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:04.0253781Z 2024-08-20T23:23:04.7937130Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:04.9099410Z 2024-08-20T23:23:04.9099839Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:04.9102537Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:04.9103390Z cuda eval pyhpc_turbulent_kinetic_energy 2024-08-20T23:23:04.9551021Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead 2024-08-20T23:23:15.1773053Z pass 2024-08-20T23:23:15.1876606Z TIMING: backend_compile:7.73127 entire_frame_compile:9.91901 total_wall_time:9.91901 2024-08-20T23:23:15.1877884Z STATS: call_* op count: 1992 | FakeTensorMode.__torch_dispatch__:34456 | FakeTensor.__torch_dispatch__:71 | ProxyTorchDispatchMode.__torch_dispatch__:14474 2024-08-20T23:23:15.1879009Z Dynamo produced 1 graphs covering 1992 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:19.0140886Z 2024-08-20T23:23:19.8089045Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:19.8089542Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:19.8101251Z cuda eval pytorch_CycleGAN_and_pix2pix 2024-08-20T23:23:28.5795763Z pass 2024-08-20T23:23:28.5802404Z TIMING: backend_compile:1.38898 entire_frame_compile:3.09111 total_wall_time:3.09111 2024-08-20T23:23:28.5803900Z STATS: call_* op count: 182 | FakeTensorMode.__torch_dispatch__:6389 | ProxyTorchDispatchMode.__torch_dispatch__:2092 2024-08-20T23:23:28.5805285Z Dynamo produced 1 graphs covering 182 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:32.2656132Z 2024-08-20T23:23:33.4464386Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:33.4464968Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:23:33.4483847Z cuda eval pytorch_stargan 2024-08-20T23:23:52.1372891Z pass 2024-08-20T23:23:52.1419456Z TIMING: backend_compile:2.30349 entire_frame_compile:3.98907 total_wall_time:3.98907 2024-08-20T23:23:52.1421867Z STATS: call_* op count: 112 | FakeTensorMode.__torch_dispatch__:10845 | FakeTensor.__torch_dispatch__:9 | ProxyTorchDispatchMode.__torch_dispatch__:3626 2024-08-20T23:23:52.1423714Z Dynamo produced 1 graphs covering 112 ops with 0 graph breaks (0 unique) 2024-08-20T23:23:55.8635797Z 2024-08-20T23:23:56.5992671Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:56.5993138Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:56.6013695Z cuda eval pytorch_unet 2024-08-20T23:24:08.0080569Z pass 2024-08-20T23:24:08.0274431Z TIMING: backend_compile:1.86712 entire_frame_compile:3.89816 total_wall_time:3.89816 2024-08-20T23:24:08.0275943Z STATS: call_* op count: 142 | FakeTensorMode.__torch_dispatch__:9567 | ProxyTorchDispatchMode.__torch_dispatch__:2964 2024-08-20T23:24:08.0277280Z Dynamo produced 1 graphs covering 142 ops with 0 graph breaks (0 unique) 2024-08-20T23:24:11.7431545Z 2024-08-20T23:24:12.5629782Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth 2024-08-20T23:24:12.5812367Z 2024-08-20T23:24:12.5812497Z 2024-08-20T23:24:12.6815149Z 0% 0.00/230M [00:00 2024-08-20T23:38:02.6410542Z W0820 23:38:02.637000 38866 torch/_dynamo/variables/tensor.py:776] [13/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:38:02.6412497Z W0820 23:38:02.637000 38866 torch/_dynamo/variables/tensor.py:776] [13/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:38:02.6414215Z W0820 23:38:02.637000 38866 torch/_dynamo/variables/tensor.py:776] [13/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:38:02.6415577Z W0820 23:38:02.637000 38866 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:38:02.6416412Z W0820 23:38:02.637000 38866 torch/_dynamo/variables/tensor.py:776] [13/0] 2024-08-20T23:38:30.6205538Z pass 2024-08-20T23:38:30.6301679Z TIMING: entire_frame_compile:46.44681 backend_compile:10.45108 total_wall_time:46.44681 2024-08-20T23:38:30.6303352Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:51797 | FakeTensor.__torch_dispatch__:42 | ProxyTorchDispatchMode.__torch_dispatch__:17420 | attempt fast:88 | fast is_contiguous:52 | slow both tensors nontrivially broadcast:36 2024-08-20T23:38:30.6304875Z Dynamo produced 25 graphs covering 1358 ops with 16 graph breaks (2 unique) 2024-08-20T23:38:34.9043379Z 2024-08-20T23:38:37.0883700Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:38:37.0884151Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:38:37.0973492Z cuda eval yolov3 2024-08-20T23:38:57.7429694Z pass 2024-08-20T23:38:57.7546219Z TIMING: backend_compile:6.56296 entire_frame_compile:13.99685 total_wall_time:13.99685 2024-08-20T23:38:57.7547765Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:26684 | FakeTensor.__torch_dispatch__:35 | ProxyTorchDispatchMode.__torch_dispatch__:6718 2024-08-20T23:38:57.7550371Z Dynamo produced 1 graphs covering 649 ops with 0 graph breaks (0 unique) 2024-08-20T23:38:58.9463631Z accuracy pass_rate=82.61% 2024-08-20T23:38:58.9467986Z calls_captured gmean=0.00x mean=662.391x 2024-08-20T23:38:58.9471376Z unique_graphs gmean=0.00x mean=1.674x 2024-08-20T23:38:58.9474888Z graph_breaks gmean=0.00x mean=0.717x 2024-08-20T23:38:58.9478564Z unique_graph_breaks gmean=0.00x mean=0.174x 2024-08-20T23:38:58.9482639Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:38:58.9486148Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:38:58.9489731Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:38:59.7368719Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_inference.csv 2024-08-20T23:39:00.0770138Z llama PASS 2024-08-20T23:39:00.0773822Z llama_v2_7b_16h XFAIL 2024-08-20T23:39:00.0778576Z llava XFAIL 2024-08-20T23:39:00.0783577Z maml XFAIL 2024-08-20T23:39:00.0788126Z maml_omniglot PASS 2024-08-20T23:39:00.0793143Z mnasnet1_0 PASS 2024-08-20T23:39:00.0797592Z mobilenet_v2 PASS 2024-08-20T23:39:00.0802654Z mobilenet_v2_quantized_qat XFAIL 2024-08-20T23:39:00.0807260Z mobilenet_v3_large PASS 2024-08-20T23:39:00.0811877Z moco PASS 2024-08-20T23:39:00.0816326Z moondream XFAIL 2024-08-20T23:39:00.0821275Z nanogpt PASS 2024-08-20T23:39:00.0826427Z nvidia_deeprecommender PASS 2024-08-20T23:39:00.0832230Z opacus_cifar10 PASS 2024-08-20T23:39:00.0836696Z phlippe_densenet PASS 2024-08-20T23:39:00.0841974Z phlippe_resnet PASS 2024-08-20T23:39:00.0846546Z pyhpc_equation_of_state PASS 2024-08-20T23:39:00.0851372Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:39:00.0856038Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:39:00.0860748Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:39:00.0865574Z pytorch_stargan PASS 2024-08-20T23:39:00.0870272Z pytorch_unet PASS 2024-08-20T23:39:00.0874987Z resnet152 PASS 2024-08-20T23:39:00.0879926Z resnet18 PASS 2024-08-20T23:39:00.0884713Z resnet50 PASS 2024-08-20T23:39:00.0889428Z resnet50_quantized_qat XFAIL 2024-08-20T23:39:00.0894090Z resnext50_32x4d PASS 2024-08-20T23:39:00.0899109Z sam PASS 2024-08-20T23:39:00.0903467Z sam_fast PASS 2024-08-20T23:39:00.0908167Z shufflenet_v2_x1_0 PASS 2024-08-20T23:39:00.0912806Z soft_actor_critic PASS 2024-08-20T23:39:00.0917648Z speech_transformer PASS 2024-08-20T23:39:00.0922237Z squeezenet1_1 PASS 2024-08-20T23:39:00.0927136Z stable_diffusion_text_encoder PASS 2024-08-20T23:39:00.0931759Z stable_diffusion_unet XFAIL 2024-08-20T23:39:00.0936475Z timm_efficientnet PASS 2024-08-20T23:39:00.0940966Z timm_regnet PASS 2024-08-20T23:39:00.0945694Z timm_resnest PASS 2024-08-20T23:39:00.0950216Z timm_vision_transformer PASS 2024-08-20T23:39:00.0954877Z timm_vision_transformer_large XFAIL 2024-08-20T23:39:00.0959544Z timm_vovnet PASS 2024-08-20T23:39:00.0964289Z torch_multimodal_clip PASS 2024-08-20T23:39:00.0968720Z tts_angular PASS 2024-08-20T23:39:00.0973531Z vgg16 PASS 2024-08-20T23:39:00.0977949Z vision_maskrcnn PASS 2024-08-20T23:39:00.0982790Z yolov3 PASS 2024-08-20T23:39:00.1505155Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_inference.csv 2024-08-20T23:39:00.4861725Z llama PASS 2024-08-20T23:39:00.4865713Z llama_v2_7b_16h PASS 2024-08-20T23:39:00.4870820Z llava PASS 2024-08-20T23:39:00.4875609Z maml PASS 2024-08-20T23:39:00.4880610Z maml_omniglot PASS 2024-08-20T23:39:00.4885537Z mnasnet1_0 PASS 2024-08-20T23:39:00.4890421Z mobilenet_v2 PASS 2024-08-20T23:39:00.4895042Z mobilenet_v2_quantized_qat PASS 2024-08-20T23:39:00.4899725Z mobilenet_v3_large PASS 2024-08-20T23:39:00.4904551Z moco PASS 2024-08-20T23:39:00.4909306Z moondream PASS 2024-08-20T23:39:00.4914066Z nanogpt PASS 2024-08-20T23:39:00.4918938Z nvidia_deeprecommender PASS 2024-08-20T23:39:00.4923682Z opacus_cifar10 PASS 2024-08-20T23:39:00.4929043Z phlippe_densenet PASS 2024-08-20T23:39:00.4933919Z phlippe_resnet PASS 2024-08-20T23:39:00.4938595Z pyhpc_equation_of_state PASS 2024-08-20T23:39:00.4943271Z pyhpc_isoneutral_mixing PASS 2024-08-20T23:39:00.4948124Z pyhpc_turbulent_kinetic_energy PASS 2024-08-20T23:39:00.4952790Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:39:00.4957732Z pytorch_stargan PASS 2024-08-20T23:39:00.4962581Z pytorch_unet PASS 2024-08-20T23:39:00.4967351Z resnet152 PASS 2024-08-20T23:39:00.4971998Z resnet18 PASS 2024-08-20T23:39:00.4976794Z resnet50 PASS 2024-08-20T23:39:00.4981543Z resnet50_quantized_qat PASS 2024-08-20T23:39:00.4986304Z resnext50_32x4d PASS 2024-08-20T23:39:00.4991007Z sam PASS 2024-08-20T23:39:00.4995637Z sam_fast PASS 2024-08-20T23:39:00.5000570Z shufflenet_v2_x1_0 PASS 2024-08-20T23:39:00.5005454Z soft_actor_critic PASS 2024-08-20T23:39:00.5010369Z speech_transformer PASS 2024-08-20T23:39:00.5015174Z squeezenet1_1 PASS 2024-08-20T23:39:00.5020071Z stable_diffusion_text_encoder PASS 2024-08-20T23:39:00.5024909Z stable_diffusion_unet PASS 2024-08-20T23:39:00.5030776Z timm_efficientnet PASS 2024-08-20T23:39:00.5035737Z timm_regnet PASS 2024-08-20T23:39:00.5040843Z timm_resnest PASS 2024-08-20T23:39:00.5045663Z timm_vision_transformer PASS 2024-08-20T23:39:00.5050439Z timm_vision_transformer_large PASS 2024-08-20T23:39:00.5055251Z timm_vovnet PASS 2024-08-20T23:39:00.5060253Z torch_multimodal_clip PASS 2024-08-20T23:39:00.5064994Z tts_angular PASS 2024-08-20T23:39:00.5069967Z vgg16 PASS 2024-08-20T23:39:00.5074751Z vision_maskrcnn PASS 2024-08-20T23:39:00.5079859Z yolov3 PASS 2024-08-20T23:39:00.5598718Z + test_single_dynamo_benchmark training torchbench 1 --training --amp 2024-08-20T23:39:00.5601794Z ++ pwd 2024-08-20T23:39:00.5605276Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:39:00.5605968Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T23:39:00.5630107Z + local name=training 2024-08-20T23:39:00.5630549Z + shift 2024-08-20T23:39:00.5630923Z + local suite=torchbench 2024-08-20T23:39:00.5631329Z + shift 2024-08-20T23:39:00.5631685Z + local shard_id=1 2024-08-20T23:39:00.5631994Z + shift 2024-08-20T23:39:00.5632258Z + partition_flags=() 2024-08-20T23:39:00.5632599Z + local partition_flags 2024-08-20T23:39:00.5633128Z + [[ -n 2 ]] 2024-08-20T23:39:00.5633518Z + [[ -n 1 ]] 2024-08-20T23:39:00.5636925Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T23:39:00.5637934Z + [[ aot_eager_torchbench == *perf_compare* ]] 2024-08-20T23:39:00.5638412Z + [[ aot_eager_torchbench == *perf* ]] 2024-08-20T23:39:00.5638859Z + [[ aot_eager_torchbench == *aot_inductor* ]] 2024-08-20T23:39:00.5639317Z + [[ aot_eager_torchbench == *_avx2* ]] 2024-08-20T23:39:00.5639755Z + [[ aot_eager_torchbench == *_avx512* ]] 2024-08-20T23:39:00.5641371Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --backend aot_eager --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv 2024-08-20T23:39:06.3194039Z 2024-08-20T23:39:06.6221155Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:06.6222059Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:06.6222938Z cuda train llava 2024-08-20T23:39:06.6227399Z Traceback (most recent call last): 2024-08-20T23:39:06.6228218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:39:06.6229075Z ) = runner.load_model( 2024-08-20T23:39:06.6229791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 302, in load_model 2024-08-20T23:39:06.6230575Z benchmark = benchmark_cls( 2024-08-20T23:39:06.6231371Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 24, in __call__ 2024-08-20T23:39:06.6232161Z obj = type.__call__(cls, *args, **kwargs) 2024-08-20T23:39:06.6233035Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 11, in __init__ 2024-08-20T23:39:06.6234472Z super().__init__(name="llava", test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:39:06.6235722Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 81, in __init__ 2024-08-20T23:39:06.6236901Z super().__init__(test=test, device=device, batch_size=batch_size, extra_args=extra_args) 2024-08-20T23:39:06.6237962Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 92, in __init__ 2024-08-20T23:39:06.6238718Z self._skip_by_device_name() 2024-08-20T23:39:06.6239521Z File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 165, in _skip_by_device_name 2024-08-20T23:39:06.6240753Z raise NotImplementedError(f"The current device {current_device_name} is skipped by its `{self.name}/metadata.yaml`.") 2024-08-20T23:39:06.6241980Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`. 2024-08-20T23:39:06.6242554Z 2024-08-20T23:39:06.6242673Z model_fail_to_load 2024-08-20T23:39:10.0820903Z 2024-08-20T23:39:10.7188502Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:10.7188974Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:10.7189409Z cuda train maml_omniglot 2024-08-20T23:39:12.8335375Z W0820 23:39:12.832000 39165 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:39:14.2149593Z pass 2024-08-20T23:39:14.2150434Z TIMING: entire_frame_compile:3.16719 backend_compile:1.90262 total_wall_time:3.16719 2024-08-20T23:39:14.2151615Z STATS: call_* op count: 77 | FakeTensorMode.__torch_dispatch__:3060 | ProxyTorchDispatchMode.__torch_dispatch__:1158 | FakeTensor.__torch_dispatch__:296 2024-08-20T23:39:14.2152774Z Dynamo produced 3 graphs covering 77 ops with 7 graph breaks (5 unique) 2024-08-20T23:39:18.0280691Z 2024-08-20T23:39:20.5824077Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:20.5824976Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:39:20.5825831Z cuda train mnasnet1_0 2024-08-20T23:39:34.7644363Z W0820 23:39:34.763000 39261 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:39:50.9596109Z pass 2024-08-20T23:39:50.9640759Z TIMING: entire_frame_compile:28.38883 backend_compile:14.14335 total_wall_time:28.38883 2024-08-20T23:39:50.9642062Z STATS: call_* op count: 647 | FakeTensorMode.__torch_dispatch__:35096 | ProxyTorchDispatchMode.__torch_dispatch__:12573 | FakeTensor.__torch_dispatch__:3072 2024-08-20T23:39:50.9643341Z Dynamo produced 3 graphs covering 647 ops with 7 graph breaks (5 unique) 2024-08-20T23:39:55.3537370Z 2024-08-20T23:39:57.3907640Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:39:57.3908099Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:39:57.3910949Z cuda train mobilenet_v2 2024-08-20T23:40:11.8532111Z pass 2024-08-20T23:40:11.8580175Z TIMING: entire_frame_compile:12.61727 backend_compile:5.74785 total_wall_time:12.61727 2024-08-20T23:40:11.8581375Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:19940 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:5206 2024-08-20T23:40:11.8582556Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique) 2024-08-20T23:40:15.9456383Z 2024-08-20T23:40:17.9240416Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:40:17.9240986Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:40:17.9241437Z cuda train mobilenet_v2_quantized_qat 2024-08-20T23:40:17.9241912Z Traceback (most recent call last): 2024-08-20T23:40:17.9242629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:40:17.9245109Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:40:17.9246210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:40:17.9248353Z pred = mod(*cloned_inputs) 2024-08-20T23:40:17.9249423Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:40:17.9250292Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:40:17.9251242Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:40:17.9252045Z raise e 2024-08-20T23:40:17.9253083Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:40:17.9254361Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:40:17.9255667Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:40:17.9256801Z return self._call_impl(*args, **kwargs) 2024-08-20T23:40:17.9257750Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:40:17.9258596Z return forward_call(*args, **kwargs) 2024-08-20T23:40:17.9259057Z File ".3", line 207, in forward 2024-08-20T23:40:17.9259795Z activation_post_process_101 = self.activation_post_process_101(classifier_1); classifier_1 = None 2024-08-20T23:40:17.9261063Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:40:17.9261953Z return self._call_impl(*args, **kwargs) 2024-08-20T23:40:17.9262880Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:40:17.9263715Z return forward_call(*args, **kwargs) 2024-08-20T23:40:17.9264704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:40:17.9265611Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:40:17.9266172Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:40:17.9266548Z 2024-08-20T23:40:17.9266840Z The above exception was the direct cause of the following exception: 2024-08-20T23:40:17.9267290Z 2024-08-20T23:40:17.9267433Z Traceback (most recent call last): 2024-08-20T23:40:17.9268075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:40:17.9268723Z ) = runner.load_model( 2024-08-20T23:40:17.9269371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:40:17.9270134Z self.validate_model(model, example_inputs) 2024-08-20T23:40:17.9270884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:40:17.9271644Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:40:17.9272115Z RuntimeError: Eager run failed 2024-08-20T23:40:17.9272377Z 2024-08-20T23:40:17.9272494Z eager_fail_to_run 2024-08-20T23:40:21.5527527Z 2024-08-20T23:40:23.9953574Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:40:23.9954057Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:40:23.9954504Z cuda train mobilenet_v3_large 2024-08-20T23:40:39.4110823Z W0820 23:40:39.410000 39489 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:40:57.7436156Z pass 2024-08-20T23:40:57.7487238Z TIMING: entire_frame_compile:31.59759 backend_compile:15.43586 total_wall_time:31.59759 2024-08-20T23:40:57.7488478Z STATS: call_* op count: 730 | FakeTensorMode.__torch_dispatch__:37587 | FakeTensor.__torch_dispatch__:3428 | ProxyTorchDispatchMode.__torch_dispatch__:13925 2024-08-20T23:40:57.7489607Z Dynamo produced 3 graphs covering 730 ops with 7 graph breaks (5 unique) 2024-08-20T23:41:02.1445586Z 2024-08-20T23:41:04.4706160Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:41:04.4706758Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:04.4708705Z cuda train moco 2024-08-20T23:41:09.1158987Z [rank0]:W0820 23:41:09.115000 39586 torch/_logging/_internal.py:1058] [3/0] Profiler function will be ignored 2024-08-20T23:41:17.3267676Z [rank0]:W0820 23:41:17.325000 39586 torch/_dynamo/backends/distributed.py:89] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance. 2024-08-20T23:41:17.3503283Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes: 2024-08-20T23:41:17.3506787Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐ 2024-08-20T23:41:17.3508291Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ Index │ Extra Ops │ Extra Param Size (b) │ 2024-08-20T23:41:17.3509751Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] ├─────────┼─────────────┼────────────────────────┤ 2024-08-20T23:41:17.3511149Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] │ 0 │ 480 │ 94032128 │ 2024-08-20T23:41:17.3512572Z [rank0]:W0820 23:41:17.349000 39586 torch/_dynamo/backends/distributed.py:106] [4/0_1] └─────────┴─────────────┴────────────────────────┘ 2024-08-20T23:41:31.7122895Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:41:31.7125215Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:41:31.7126755Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] or: 2024-08-20T23:41:31.7127789Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:41:31.7129040Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] to include these operations in the captured graph. 2024-08-20T23:41:31.7130070Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:41:31.7131014Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] Graph break: from user code at: 2024-08-20T23:41:31.7132741Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 158, in torch_dynamo_resume_in_forward_at_133 2024-08-20T23:41:31.7134447Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] self._dequeue_and_enqueue(k) 2024-08-20T23:41:31.7136315Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context 2024-08-20T23:41:31.7137841Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] return func(*args, **kwargs) 2024-08-20T23:41:31.7139450Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _dequeue_and_enqueue 2024-08-20T23:41:31.7141042Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] ptr = int(self.queue_ptr) 2024-08-20T23:41:31.7141976Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:41:31.7142752Z [rank0]:W0820 23:41:31.711000 39586 torch/_dynamo/variables/tensor.py:776] [7/0] 2024-08-20T23:41:39.9756607Z pass 2024-08-20T23:41:40.0124635Z TIMING: entire_frame_compile:30.46861 backend_compile:13.2412 total_wall_time:30.46861 2024-08-20T23:41:40.0126480Z STATS: call_* op count: 889 | FakeTensorMode.__torch_dispatch__:46139 | FakeTensor.__torch_dispatch__:114 | ProxyTorchDispatchMode.__torch_dispatch__:13177 2024-08-20T23:41:40.0127717Z Dynamo produced 10 graphs covering 889 ops with 11 graph breaks (8 unique) 2024-08-20T23:41:40.8655946Z [rank0]:[W820 23:41:40.182264788 ProcessGroupNCCL.cpp:1223] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator()) 2024-08-20T23:41:45.1696817Z 2024-08-20T23:41:47.1627275Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M 2024-08-20T23:41:47.4168755Z num decayed parameter tensors: 50, with 124,354,560 parameters 2024-08-20T23:41:47.4170053Z num non-decayed parameter tensors: 98, with 121,344 parameters 2024-08-20T23:41:47.4171098Z using fused AdamW: True 2024-08-20T23:41:47.8512800Z 2024-08-20T23:41:47.8513351Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:41:47.8513846Z cuda train nanogpt 2024-08-20T23:41:57.7395177Z W0820 23:41:57.738000 39692 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:42:10.7612902Z pass 2024-08-20T23:42:10.8723455Z TIMING: entire_frame_compile:21.03686 backend_compile:11.78077 total_wall_time:21.03686 2024-08-20T23:42:10.8724862Z STATS: call_* op count: 858 | FakeTensorMode.__torch_dispatch__:33992 | FakeTensor.__torch_dispatch__:2868 | ProxyTorchDispatchMode.__torch_dispatch__:15889 2024-08-20T23:42:10.8726475Z Dynamo produced 3 graphs covering 858 ops with 7 graph breaks (5 unique) 2024-08-20T23:42:15.0288322Z 2024-08-20T23:42:18.5889269Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:18.5889871Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:42:18.5890372Z cuda train nvidia_deeprecommender 2024-08-20T23:42:20.8405336Z W0820 23:42:20.839000 39789 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:42:22.2197142Z pass 2024-08-20T23:42:22.2497222Z TIMING: entire_frame_compile:2.65147 backend_compile:1.69637 total_wall_time:2.65147 2024-08-20T23:42:22.2498625Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:2462 | ProxyTorchDispatchMode.__torch_dispatch__:1070 | FakeTensor.__torch_dispatch__:272 2024-08-20T23:42:22.2499765Z Dynamo produced 3 graphs covering 70 ops with 7 graph breaks (5 unique) 2024-08-20T23:42:25.9933346Z 2024-08-20T23:42:26.9504772Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:26.9505320Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:26.9505769Z cuda train opacus_cifar10 2024-08-20T23:42:26.9520084Z Traceback (most recent call last): 2024-08-20T23:42:26.9520907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:42:26.9521683Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:42:26.9523565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:42:26.9524452Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:42:26.9525514Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:42:26.9526628Z torch.autograd.backward( 2024-08-20T23:42:26.9527648Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:42:26.9528706Z _engine_run_backward( 2024-08-20T23:42:26.9529598Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:42:26.9530771Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:42:26.9531957Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 98, in __call__ 2024-08-20T23:42:26.9532780Z return self.hook(module, *args, **kwargs) 2024-08-20T23:42:26.9533868Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 328, in capture_backprops_hook 2024-08-20T23:42:26.9534894Z activations, backprops = self.rearrange_grad_samples( 2024-08-20T23:42:26.9536179Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 384, in rearrange_grad_samples 2024-08-20T23:42:26.9537102Z raise ValueError( 2024-08-20T23:42:26.9537953Z ValueError: No activations detected for , run forward after add_hooks(model) 2024-08-20T23:42:26.9538637Z 2024-08-20T23:42:26.9538950Z The above exception was the direct cause of the following exception: 2024-08-20T23:42:26.9539388Z 2024-08-20T23:42:26.9539536Z Traceback (most recent call last): 2024-08-20T23:42:26.9540169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:42:26.9540819Z ) = runner.load_model( 2024-08-20T23:42:26.9541474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:42:26.9542224Z self.validate_model(model, example_inputs) 2024-08-20T23:42:26.9542966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:42:26.9543725Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:42:26.9544204Z RuntimeError: Eager run failed 2024-08-20T23:42:26.9544456Z 2024-08-20T23:42:26.9544580Z eager_fail_to_run 2024-08-20T23:42:30.5221303Z 2024-08-20T23:42:31.3513336Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:31.3513820Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:31.3514263Z cuda train phlippe_densenet 2024-08-20T23:42:45.6836054Z pass 2024-08-20T23:42:45.6839244Z TIMING: entire_frame_compile:13.61074 backend_compile:6.26324 total_wall_time:13.61074 2024-08-20T23:42:45.6840779Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:21652 | ProxyTorchDispatchMode.__torch_dispatch__:6202 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:42:45.6841920Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique) 2024-08-20T23:42:49.7118522Z 2024-08-20T23:42:50.4444056Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:50.4444535Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:42:50.4445002Z cuda train phlippe_resnet 2024-08-20T23:42:55.7321013Z pass 2024-08-20T23:42:55.7321706Z TIMING: entire_frame_compile:4.85973 backend_compile:2.48788 total_wall_time:4.85973 2024-08-20T23:42:55.7322914Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:8091 | ProxyTorchDispatchMode.__torch_dispatch__:2343 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:42:55.7324042Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique) 2024-08-20T23:42:59.6257287Z 2024-08-20T23:43:00.9228090Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:00.9228692Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:43:00.9229266Z cuda train pytorch_CycleGAN_and_pix2pix 2024-08-20T23:43:14.7043105Z pass 2024-08-20T23:43:14.7132610Z TIMING: entire_frame_compile:4.85967 backend_compile:2.77839 total_wall_time:4.85967 2024-08-20T23:43:14.7134019Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:8246 | ProxyTorchDispatchMode.__torch_dispatch__:3021 | FakeTensor.__torch_dispatch__:25 2024-08-20T23:43:14.7135164Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique) 2024-08-20T23:43:18.6182105Z 2024-08-20T23:43:20.1526773Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:20.1527730Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:43:20.1528391Z cuda train pytorch_stargan 2024-08-20T23:43:51.6897875Z pass 2024-08-20T23:43:51.7086645Z TIMING: entire_frame_compile:5.30753 backend_compile:3.19273 total_wall_time:5.30753 2024-08-20T23:43:51.7089005Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:10625 | FakeTensor.__torch_dispatch__:37 | ProxyTorchDispatchMode.__torch_dispatch__:3747 2024-08-20T23:43:51.7091200Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique) 2024-08-20T23:43:55.6189712Z 2024-08-20T23:43:56.5351334Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:56.5351818Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:43:56.5352267Z cuda train pytorch_unet 2024-08-20T23:44:22.1146763Z W0820 23:44:22.113000 40310 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:44:29.6033887Z pass_due_to_skip 2024-08-20T23:44:29.7790264Z TIMING: entire_frame_compile:12.89735 backend_compile:6.82988 total_wall_time:12.89735 2024-08-20T23:44:29.7791515Z STATS: call_* op count: 314 | FakeTensorMode.__torch_dispatch__:17695 | ProxyTorchDispatchMode.__torch_dispatch__:6972 | FakeTensor.__torch_dispatch__:1428 2024-08-20T23:44:29.7792673Z Dynamo produced 3 graphs covering 314 ops with 7 graph breaks (5 unique) 2024-08-20T23:44:33.7920730Z 2024-08-20T23:44:36.2015325Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:44:36.2015908Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:44:36.2018086Z cuda train resnet152 2024-08-20T23:45:16.2476392Z W0820 23:45:16.246000 40399 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:46:04.5485211Z pass 2024-08-20T23:46:04.6171508Z TIMING: entire_frame_compile:81.60233 backend_compile:40.16321 total_wall_time:81.60233 2024-08-20T23:46:04.6172808Z STATS: call_* op count: 1937 | FakeTensorMode.__torch_dispatch__:104022 | ProxyTorchDispatchMode.__torch_dispatch__:38010 | FakeTensor.__torch_dispatch__:8542 2024-08-20T23:46:04.6173975Z Dynamo produced 3 graphs covering 1937 ops with 7 graph breaks (5 unique) 2024-08-20T23:46:10.2320041Z 2024-08-20T23:46:11.5288043Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:11.5288640Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:11.5292005Z cuda train resnet18 2024-08-20T23:46:17.9866831Z pass 2024-08-20T23:46:17.9946668Z TIMING: entire_frame_compile:4.84557 backend_compile:2.53201 total_wall_time:4.84557 2024-08-20T23:46:17.9947883Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:8083 | ProxyTorchDispatchMode.__torch_dispatch__:2258 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:46:17.9949018Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique) 2024-08-20T23:46:21.9166681Z 2024-08-20T23:46:23.7565492Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:23.7566040Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:23.7566598Z cuda train resnet50 2024-08-20T23:46:40.6564732Z pass 2024-08-20T23:46:40.6756448Z TIMING: entire_frame_compile:12.51319 backend_compile:5.99572 total_wall_time:12.51319 2024-08-20T23:46:40.6757660Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:20858 | ProxyTorchDispatchMode.__torch_dispatch__:5826 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:46:40.6758866Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique) 2024-08-20T23:46:44.7763323Z 2024-08-20T23:46:46.8380090Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:46.8380565Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:46:46.8385010Z cuda train resnet50_quantized_qat 2024-08-20T23:46:46.8385648Z Traceback (most recent call last): 2024-08-20T23:46:46.8386498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:46:46.8387701Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:46:46.8388842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 443, in forward_and_backward_pass 2024-08-20T23:46:46.8389828Z pred = mod(*cloned_inputs) 2024-08-20T23:46:46.8391294Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 785, in call_wrapped 2024-08-20T23:46:46.8392394Z return self._wrapped_call(self, *args, **kwargs) 2024-08-20T23:46:46.8393587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 362, in __call__ 2024-08-20T23:46:46.8394564Z raise e 2024-08-20T23:46:46.8395579Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 349, in __call__ 2024-08-20T23:46:46.8397083Z return super(self.cls, obj).__call__(*args, **kwargs) # type: ignore[misc] 2024-08-20T23:46:46.8398776Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:46:46.8399936Z return self._call_impl(*args, **kwargs) 2024-08-20T23:46:46.8401172Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:46:46.8402155Z return forward_call(*args, **kwargs) 2024-08-20T23:46:46.8402619Z File ".3", line 167, in forward 2024-08-20T23:46:46.8403248Z activation_post_process_73 = self.activation_post_process_73(fc); fc = None 2024-08-20T23:46:46.8404400Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T23:46:46.8405274Z return self._call_impl(*args, **kwargs) 2024-08-20T23:46:46.8406203Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T23:46:46.8407051Z return forward_call(*args, **kwargs) 2024-08-20T23:46:46.8408028Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward 2024-08-20T23:46:46.8408933Z return torch.fused_moving_avg_obs_fake_quant( 2024-08-20T23:46:46.8409482Z RuntimeError: expected scalar type Float but found Half 2024-08-20T23:46:46.8409859Z 2024-08-20T23:46:46.8410148Z The above exception was the direct cause of the following exception: 2024-08-20T23:46:46.8410600Z 2024-08-20T23:46:46.8410749Z Traceback (most recent call last): 2024-08-20T23:46:46.8411384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:46:46.8412023Z ) = runner.load_model( 2024-08-20T23:46:46.8412674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:46:46.8413421Z self.validate_model(model, example_inputs) 2024-08-20T23:46:46.8414169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:46:46.8414917Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:46:46.8415381Z RuntimeError: Eager run failed 2024-08-20T23:46:46.8415630Z 2024-08-20T23:46:46.8415750Z eager_fail_to_run 2024-08-20T23:46:50.4510044Z 2024-08-20T23:46:52.4393132Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:46:52.4393734Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:46:52.4394291Z cuda train resnext50_32x4d 2024-08-20T23:47:10.7903415Z W0820 23:47:10.789000 40723 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:47:27.1559130Z pass 2024-08-20T23:47:27.1765699Z TIMING: entire_frame_compile:28.52855 backend_compile:14.35978 total_wall_time:28.52855 2024-08-20T23:47:27.1768381Z STATS: call_* op count: 679 | FakeTensorMode.__torch_dispatch__:36019 | ProxyTorchDispatchMode.__torch_dispatch__:13088 | FakeTensor.__torch_dispatch__:3026 2024-08-20T23:47:27.1769571Z Dynamo produced 3 graphs covering 679 ops with 7 graph breaks (5 unique) 2024-08-20T23:47:31.4988185Z 2024-08-20T23:47:39.0810776Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:39.0811331Z loading model: 0it [00:07, ?it/s] 2024-08-20T23:47:39.0811752Z cuda train sam 2024-08-20T23:47:39.0821200Z Traceback (most recent call last): 2024-08-20T23:47:39.0822076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T23:47:39.0822846Z self.model_iter_fn(model, example_inputs) 2024-08-20T23:47:39.0823697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 445, in forward_and_backward_pass 2024-08-20T23:47:39.0824652Z self.grad_scaler.scale(loss).backward() 2024-08-20T23:47:39.0826235Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward 2024-08-20T23:47:39.0827260Z torch.autograd.backward( 2024-08-20T23:47:39.0828683Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 347, in backward 2024-08-20T23:47:39.0829658Z _engine_run_backward( 2024-08-20T23:47:39.0830717Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 817, in _engine_run_backward 2024-08-20T23:47:39.0832289Z return Variable._execution_engine.run_backward( # Calls into the C++ engine to run the backward pass 2024-08-20T23:47:39.0833522Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn 2024-08-20T23:47:39.0834192Z 2024-08-20T23:47:39.0834578Z The above exception was the direct cause of the following exception: 2024-08-20T23:47:39.0835044Z 2024-08-20T23:47:39.0835190Z Traceback (most recent call last): 2024-08-20T23:47:39.0835880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T23:47:39.0836613Z ) = runner.load_model( 2024-08-20T23:47:39.0837270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 362, in load_model 2024-08-20T23:47:39.0838135Z self.validate_model(model, example_inputs) 2024-08-20T23:47:39.0838897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T23:47:39.0839663Z raise RuntimeError("Eager run failed") from e 2024-08-20T23:47:39.0840136Z RuntimeError: Eager run failed 2024-08-20T23:47:39.0840395Z 2024-08-20T23:47:39.0840513Z eager_fail_to_run 2024-08-20T23:47:42.6942656Z 2024-08-20T23:47:44.0184224Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:47:44.0184822Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:47:44.0185385Z cuda train shufflenet_v2_x1_0 2024-08-20T23:47:58.6862800Z pass 2024-08-20T23:47:58.6872894Z TIMING: entire_frame_compile:13.63205 backend_compile:6.45011 total_wall_time:13.63205 2024-08-20T23:47:58.6874317Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:23523 | ProxyTorchDispatchMode.__torch_dispatch__:6673 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:47:58.6875427Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique) 2024-08-20T23:48:02.8147366Z 2024-08-20T23:48:03.6663222Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:03.6663774Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:03.6664215Z cuda train soft_actor_critic 2024-08-20T23:48:05.2684741Z W0820 23:48:05.267000 40974 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:48:05.8524613Z pass 2024-08-20T23:48:05.8525378Z TIMING: entire_frame_compile:1.92911 backend_compile:1.30621 total_wall_time:1.92911 2024-08-20T23:48:05.8527086Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:1651 | ProxyTorchDispatchMode.__torch_dispatch__:686 | FakeTensor.__torch_dispatch__:156 2024-08-20T23:48:05.8528566Z Dynamo produced 4 graphs covering 56 ops with 6 graph breaks (5 unique) 2024-08-20T23:48:09.6496237Z 2024-08-20T23:48:10.9360832Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:10.9361740Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:48:10.9363062Z cuda train speech_transformer 2024-08-20T23:48:27.2044827Z pass 2024-08-20T23:48:27.2503130Z TIMING: entire_frame_compile:13.54983 backend_compile:8.87536 total_wall_time:13.54983 2024-08-20T23:48:27.2505518Z STATS: call_* op count: 771 | FakeTensorMode.__torch_dispatch__:32505 | FakeTensor.__torch_dispatch__:80 | ProxyTorchDispatchMode.__torch_dispatch__:17627 2024-08-20T23:48:27.2507620Z Dynamo produced 10 graphs covering 771 ops with 16 graph breaks (7 unique) 2024-08-20T23:48:31.3160682Z 2024-08-20T23:48:32.4469222Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:32.4469695Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:48:32.4470133Z cuda train squeezenet1_1 2024-08-20T23:48:37.0190460Z pass 2024-08-20T23:48:37.0204805Z TIMING: entire_frame_compile:3.40917 backend_compile:1.98452 total_wall_time:3.40917 2024-08-20T23:48:37.0206539Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:4326 | ProxyTorchDispatchMode.__torch_dispatch__:1964 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:48:37.0207674Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique) 2024-08-20T23:48:40.9175704Z 2024-08-20T23:48:42.1941358Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:48:42.1941699Z 2024-08-20T23:48:43.1811065Z Loading pipeline components...: 0% 0/6 [00:00 will be ignored 2024-08-20T23:50:05.1724857Z pass 2024-08-20T23:50:05.1776853Z TIMING: entire_frame_compile:38.38272 backend_compile:19.16904 total_wall_time:38.38272 2024-08-20T23:50:05.1779319Z STATS: call_* op count: 899 | FakeTensorMode.__torch_dispatch__:45952 | FakeTensor.__torch_dispatch__:4172 | ProxyTorchDispatchMode.__torch_dispatch__:18598 2024-08-20T23:50:05.1780765Z Dynamo produced 3 graphs covering 899 ops with 7 graph breaks (5 unique) 2024-08-20T23:50:09.8527233Z 2024-08-20T23:50:12.6683147Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:12.6683699Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:50:12.6684126Z cuda train timm_regnet 2024-08-20T23:50:42.4756631Z pass 2024-08-20T23:50:42.5188073Z TIMING: entire_frame_compile:19.74267 backend_compile:8.95494 total_wall_time:19.74267 2024-08-20T23:50:42.5189910Z STATS: call_* op count: 323 | FakeTensorMode.__torch_dispatch__:31084 | ProxyTorchDispatchMode.__torch_dispatch__:10183 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:50:42.5191526Z Dynamo produced 2 graphs covering 323 ops with 6 graph breaks (5 unique) 2024-08-20T23:50:46.8697692Z 2024-08-20T23:50:48.8966685Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:50:48.8967252Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:50:48.8968692Z cuda train timm_resnest 2024-08-20T23:50:59.5955325Z W0820 23:50:59.594000 41608 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:51:08.1143180Z pass 2024-08-20T23:51:08.1237832Z TIMING: entire_frame_compile:14.93045 backend_compile:7.65656 total_wall_time:14.93045 2024-08-20T23:51:08.1239290Z STATS: call_* op count: 401 | FakeTensorMode.__torch_dispatch__:19338 | ProxyTorchDispatchMode.__torch_dispatch__:7268 | FakeTensor.__torch_dispatch__:1638 2024-08-20T23:51:08.1240411Z Dynamo produced 3 graphs covering 401 ops with 7 graph breaks (5 unique) 2024-08-20T23:51:12.3695984Z 2024-08-20T23:51:14.0259496Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:14.0260528Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:51:14.0261446Z cuda train timm_vision_transformer 2024-08-20T23:51:23.4436689Z pass 2024-08-20T23:51:23.4599269Z TIMING: entire_frame_compile:8.07553 backend_compile:4.46472 total_wall_time:8.07553 2024-08-20T23:51:23.4602127Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:16504 | FakeTensor.__torch_dispatch__:166 | ProxyTorchDispatchMode.__torch_dispatch__:7504 2024-08-20T23:51:23.4603265Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique) 2024-08-20T23:51:27.5634414Z 2024-08-20T23:51:40.1550632Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:40.1551212Z loading model: 0it [00:12, ?it/s] 2024-08-20T23:51:40.1551666Z cuda train timm_vision_transformer_large 2024-08-20T23:51:40.1553867Z pass_due_to_skip 2024-08-20T23:51:40.4239844Z TIMING: 2024-08-20T23:51:40.4240178Z STATS: call_* op count: 0 2024-08-20T23:51:40.4240806Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2024-08-20T23:51:44.0967322Z 2024-08-20T23:51:46.1473952Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:51:46.1474584Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:51:46.1475114Z cuda train timm_vovnet 2024-08-20T23:52:02.6810145Z pass 2024-08-20T23:52:02.6986227Z TIMING: entire_frame_compile:10.28473 backend_compile:4.76597 total_wall_time:10.28473 2024-08-20T23:52:02.6987477Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:15935 | ProxyTorchDispatchMode.__torch_dispatch__:4649 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:52:02.6988593Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique) 2024-08-20T23:52:06.8697310Z 2024-08-20T23:52:08.8892772Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:52:08.8893244Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:52:08.8893696Z cuda train torch_multimodal_clip 2024-08-20T23:52:31.7864792Z W0820 23:52:31.785000 41953 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:52:58.3004057Z pass 2024-08-20T23:52:58.4476174Z TIMING: entire_frame_compile:44.17319 backend_compile:24.31425 total_wall_time:44.17319 2024-08-20T23:52:58.4478124Z STATS: call_* op count: 1404 | FakeTensorMode.__torch_dispatch__:69734 | FakeTensor.__torch_dispatch__:5900 | ProxyTorchDispatchMode.__torch_dispatch__:32713 2024-08-20T23:52:58.4479569Z Dynamo produced 3 graphs covering 1404 ops with 7 graph breaks (5 unique) 2024-08-20T23:53:03.1516999Z 2024-08-20T23:53:03.8183844Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:03.8184452Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:03.8184922Z cuda train tts_angular 2024-08-20T23:53:05.3394860Z W0820 23:53:05.338000 42050 torch/_logging/_internal.py:1058] [10/0] Profiler function will be ignored 2024-08-20T23:53:06.8009632Z pass 2024-08-20T23:53:06.8113455Z TIMING: entire_frame_compile:2.49365 backend_compile:1.65595 total_wall_time:2.49365 2024-08-20T23:53:06.8115263Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:1768 | FakeTensor.__torch_dispatch__:334 | ProxyTorchDispatchMode.__torch_dispatch__:825 2024-08-20T23:53:06.8116387Z Dynamo produced 5 graphs covering 70 ops with 9 graph breaks (6 unique) 2024-08-20T23:53:10.5799338Z 2024-08-20T23:53:13.2044178Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:13.2044793Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:53:13.2045252Z cuda train vgg16 2024-08-20T23:53:33.1144979Z pass 2024-08-20T23:53:33.1398462Z TIMING: entire_frame_compile:2.50839 backend_compile:1.6078 total_wall_time:2.50839 2024-08-20T23:53:33.1399667Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:2816 | ProxyTorchDispatchMode.__torch_dispatch__:1238 | FakeTensor.__torch_dispatch__:22 2024-08-20T23:53:33.1400779Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique) 2024-08-20T23:53:36.9835054Z 2024-08-20T23:53:40.3586254Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:53:40.3586856Z loading model: 0it [00:03, ?it/s] 2024-08-20T23:53:40.3587419Z cuda train vision_maskrcnn 2024-08-20T23:54:06.8581186Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break from `Tensor.item()`, consider setting: 2024-08-20T23:54:06.8582660Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] torch._dynamo.config.capture_scalar_outputs = True 2024-08-20T23:54:06.8585016Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] or: 2024-08-20T23:54:06.8586043Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1 2024-08-20T23:54:06.8587246Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] to include these operations in the captured graph. 2024-08-20T23:54:06.8588381Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:54:06.8589289Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] Graph break: from user code at: 2024-08-20T23:54:06.8591814Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104 2024-08-20T23:54:06.8594072Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets) 2024-08-20T23:54:06.8596180Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward 2024-08-20T23:54:06.8597973Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] box_features = self.box_roi_pool(features, proposals, image_shapes) 2024-08-20T23:54:06.8599718Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward 2024-08-20T23:54:06.8601513Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] self.scales, self.map_levels = _setup_scales( 2024-08-20T23:54:06.8603184Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales 2024-08-20T23:54:06.8604813Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:54:06.8606574Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in 2024-08-20T23:54:06.8608191Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] scales = [_infer_scale(feat, original_input_shape) for feat in features] 2024-08-20T23:54:06.8610117Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale 2024-08-20T23:54:06.8611709Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] scale = 2 ** float(torch.tensor(approx_scale).log2().round()) 2024-08-20T23:54:06.8612716Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:54:06.8613445Z W0820 23:54:06.857000 42244 torch/_dynamo/variables/tensor.py:776] [18/0] 2024-08-20T23:55:02.3477325Z W0820 23:55:02.347000 42244 torch/_dynamo/convert_frame.py:831] [30/8] torch._dynamo hit config.cache_size_limit (8) 2024-08-20T23:55:02.3479677Z W0820 23:55:02.347000 42244 torch/_dynamo/convert_frame.py:831] [30/8] function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186) 2024-08-20T23:55:02.3481510Z W0820 23:55:02.347000 42244 torch/_dynamo/convert_frame.py:831] [30/8] last reason: 30/0: tensor 'L['boxes']' size mismatch at index 0. expected 648, actual 619 2024-08-20T23:55:02.3482941Z W0820 23:55:02.347000 42244 torch/_dynamo/convert_frame.py:831] [30/8] To log all recompilation reasons, use TORCH_LOGS="recompiles". 2024-08-20T23:55:02.3484478Z W0820 23:55:02.347000 42244 torch/_dynamo/convert_frame.py:831] [30/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html. 2024-08-20T23:55:32.8663368Z pass 2024-08-20T23:55:32.9309831Z TIMING: entire_frame_compile:103.92073 backend_compile:56.94914 total_wall_time:103.92073 2024-08-20T23:55:32.9311585Z STATS: call_* op count: 2436 | FakeTensorMode.__torch_dispatch__:115903 | FakeTensor.__torch_dispatch__:680 | ProxyTorchDispatchMode.__torch_dispatch__:41391 | attempt fast:1737 | slow no contiguity match:182 | fast is_contiguous:1219 | slow both tensors nontrivially broadcast:336 2024-08-20T23:55:32.9313291Z Dynamo produced 51 graphs covering 2436 ops with 33 graph breaks (8 unique) 2024-08-20T23:55:39.6588208Z 2024-08-20T23:55:42.5321156Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:55:42.5321619Z loading model: 0it [00:02, ?it/s] 2024-08-20T23:55:42.5322045Z cuda train yolov3 2024-08-20T23:56:20.3878651Z W0820 23:56:20.387000 42342 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:57:01.9559129Z pass 2024-08-20T23:57:02.0180280Z TIMING: entire_frame_compile:60.5784 backend_compile:28.66799 total_wall_time:60.5784 2024-08-20T23:57:02.0181662Z STATS: call_* op count: 1377 | FakeTensorMode.__torch_dispatch__:81978 | FakeTensor.__torch_dispatch__:4200 | ProxyTorchDispatchMode.__torch_dispatch__:28065 2024-08-20T23:57:02.0182786Z Dynamo produced 4 graphs covering 1377 ops with 8 graph breaks (5 unique) 2024-08-20T23:57:04.4066148Z accuracy pass_rate=78.95% 2024-08-20T23:57:04.4069857Z calls_captured gmean=0.00x mean=431.184x 2024-08-20T23:57:04.4073696Z unique_graphs gmean=0.00x mean=3.816x 2024-08-20T23:57:04.4077606Z graph_breaks gmean=0.00x mean=6.395x 2024-08-20T23:57:04.4081365Z unique_graph_breaks gmean=0.00x mean=4.316x 2024-08-20T23:57:04.4085068Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:57:04.4088850Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:57:04.4092647Z cudagraph_skips gmean=0.00x mean=0.000x 2024-08-20T23:57:05.0012116Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_training.csv 2024-08-20T23:57:05.3361389Z llava XFAIL 2024-08-20T23:57:05.3365387Z maml_omniglot PASS 2024-08-20T23:57:05.3369429Z mnasnet1_0 PASS 2024-08-20T23:57:05.3373996Z mobilenet_v2 PASS 2024-08-20T23:57:05.3378728Z mobilenet_v2_quantized_qat XFAIL 2024-08-20T23:57:05.3382934Z mobilenet_v3_large PASS 2024-08-20T23:57:05.3387234Z moco PASS 2024-08-20T23:57:05.3391791Z nanogpt PASS 2024-08-20T23:57:05.3396175Z nvidia_deeprecommender PASS 2024-08-20T23:57:05.3400623Z opacus_cifar10 XFAIL 2024-08-20T23:57:05.3404815Z phlippe_densenet PASS 2024-08-20T23:57:05.3409082Z phlippe_resnet PASS 2024-08-20T23:57:05.3413481Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:57:05.3417719Z pytorch_stargan PASS 2024-08-20T23:57:05.3421947Z pytorch_unet XFAIL 2024-08-20T23:57:05.3426476Z resnet152 PASS 2024-08-20T23:57:05.3431457Z resnet18 PASS 2024-08-20T23:57:05.3435782Z resnet50 PASS 2024-08-20T23:57:05.3440226Z resnet50_quantized_qat XFAIL 2024-08-20T23:57:05.3444585Z resnext50_32x4d PASS 2024-08-20T23:57:05.3448690Z sam XFAIL 2024-08-20T23:57:05.3453111Z shufflenet_v2_x1_0 PASS 2024-08-20T23:57:05.3457272Z soft_actor_critic PASS 2024-08-20T23:57:05.3461711Z speech_transformer PASS 2024-08-20T23:57:05.3465951Z squeezenet1_1 PASS 2024-08-20T23:57:05.3470207Z stable_diffusion_text_encoder PASS 2024-08-20T23:57:05.3474515Z stable_diffusion_unet XFAIL 2024-08-20T23:57:05.3478886Z timm_efficientnet PASS 2024-08-20T23:57:05.3483248Z timm_regnet PASS 2024-08-20T23:57:05.3487509Z timm_resnest PASS 2024-08-20T23:57:05.3491744Z timm_vision_transformer PASS 2024-08-20T23:57:05.3496118Z timm_vision_transformer_large XFAIL 2024-08-20T23:57:05.3500313Z timm_vovnet PASS 2024-08-20T23:57:05.3504725Z torch_multimodal_clip PASS 2024-08-20T23:57:05.3508900Z tts_angular PASS 2024-08-20T23:57:05.3513326Z vgg16 PASS 2024-08-20T23:57:05.3517569Z vision_maskrcnn PASS 2024-08-20T23:57:05.3522217Z yolov3 PASS_BUT_FLAKY: accuracy=pass, expected=fail_accuracy 2024-08-20T23:57:05.4036855Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/aot_eager_torchbench_training.csv 2024-08-20T23:57:05.7391807Z llava PASS 2024-08-20T23:57:05.7394473Z maml_omniglot PASS 2024-08-20T23:57:05.7399040Z mnasnet1_0 PASS 2024-08-20T23:57:05.7403452Z mobilenet_v2 PASS 2024-08-20T23:57:05.7407750Z mobilenet_v2_quantized_qat PASS 2024-08-20T23:57:05.7412352Z mobilenet_v3_large PASS 2024-08-20T23:57:05.7416748Z moco PASS 2024-08-20T23:57:05.7421008Z nanogpt PASS 2024-08-20T23:57:05.7425504Z nvidia_deeprecommender PASS 2024-08-20T23:57:05.7430400Z opacus_cifar10 PASS 2024-08-20T23:57:05.7434685Z phlippe_densenet PASS 2024-08-20T23:57:05.7439312Z phlippe_resnet PASS 2024-08-20T23:57:05.7443864Z pytorch_CycleGAN_and_pix2pix PASS 2024-08-20T23:57:05.7448218Z pytorch_stargan PASS 2024-08-20T23:57:05.7452477Z pytorch_unet PASS 2024-08-20T23:57:05.7457041Z resnet152 PASS 2024-08-20T23:57:05.7461320Z resnet18 PASS 2024-08-20T23:57:05.7465684Z resnet50 PASS 2024-08-20T23:57:05.7470142Z resnet50_quantized_qat PASS 2024-08-20T23:57:05.7474548Z resnext50_32x4d PASS 2024-08-20T23:57:05.7479149Z sam PASS 2024-08-20T23:57:05.7483810Z shufflenet_v2_x1_0 PASS 2024-08-20T23:57:05.7487897Z soft_actor_critic PASS 2024-08-20T23:57:05.7492399Z speech_transformer PASS 2024-08-20T23:57:05.7496702Z squeezenet1_1 PASS 2024-08-20T23:57:05.7501184Z stable_diffusion_text_encoder PASS 2024-08-20T23:57:05.7505606Z stable_diffusion_unet PASS 2024-08-20T23:57:05.7510025Z timm_efficientnet PASS 2024-08-20T23:57:05.7514415Z timm_regnet PASS 2024-08-20T23:57:05.7518953Z timm_resnest PASS 2024-08-20T23:57:05.7523433Z timm_vision_transformer PASS 2024-08-20T23:57:05.7528173Z timm_vision_transformer_large PASS 2024-08-20T23:57:05.7532465Z timm_vovnet PASS 2024-08-20T23:57:05.7536827Z torch_multimodal_clip PASS 2024-08-20T23:57:05.7541251Z tts_angular PASS 2024-08-20T23:57:05.7545684Z vgg16 PASS 2024-08-20T23:57:05.7550099Z vision_maskrcnn PASS 2024-08-20T23:57:05.7554563Z yolov3 PASS 2024-08-20T23:57:05.8069402Z + cleanup_workspace 2024-08-20T23:57:05.8070673Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:57:05.8072124Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:57:05.8073411Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:57:05.8074212Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:57:05.8075036Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:57:05.8075926Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:57:05.8076635Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:57:06.5790326Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:57:06.5790852Z cat test/**/*_toprint.log || true 2024-08-20T23:57:06.5804794Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:06.5805310Z env: 2024-08-20T23:57:06.5805598Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:06.5806065Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:06.5806820Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:06.5807481Z ##[endgroup] 2024-08-20T23:57:06.5901100Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:57:06.5934591Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:57:06.5935055Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:57:06.5943667Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:06.5944160Z env: 2024-08-20T23:57:06.5944430Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:06.5944872Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:06.5945596Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:06.5946402Z MONITOR_SCRIPT_PID: 52483 2024-08-20T23:57:06.5946750Z ##[endgroup] 2024-08-20T23:57:06.6147583Z Prepare all required actions 2024-08-20T23:57:06.6148035Z Getting action download info 2024-08-20T23:57:06.8100845Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:57:06.9769986Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:57:06.9770475Z with: 2024-08-20T23:57:06.9771020Z file-suffix: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039 2024-08-20T23:57:06.9771691Z s3-bucket: gha-artifacts 2024-08-20T23:57:06.9772029Z env: 2024-08-20T23:57:06.9772312Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:06.9772771Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:06.9773643Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:06.9774288Z ##[endgroup] 2024-08-20T23:57:06.9802681Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:57:06.9803362Z # Remove any previous test jsons if they exist 2024-08-20T23:57:06.9803946Z rm -f test-jsons-*.zip 2024-08-20T23:57:06.9804465Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:57:06.9813737Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:06.9814227Z env: 2024-08-20T23:57:06.9814503Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:06.9814943Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:06.9815767Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:06.9816675Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039 2024-08-20T23:57:06.9817309Z ##[endgroup] 2024-08-20T23:57:07.0162978Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:57:07.0201141Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:57:07.0204801Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:57:07.0217012Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:57:07.0227492Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:57:07.0260073Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:57:07.0260718Z # Remove any previous test reports if they exist 2024-08-20T23:57:07.0261245Z rm -f test-reports-*.zip 2024-08-20T23:57:07.0261842Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:57:07.0270893Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:07.0271388Z env: 2024-08-20T23:57:07.0271670Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:07.0272123Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:07.0272861Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:07.0273751Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039 2024-08-20T23:57:07.0274413Z ##[endgroup] 2024-08-20T23:57:07.0542062Z adding: test/test-reports/inference_torchbench.csv (deflated 69%) 2024-08-20T23:57:07.0544325Z adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%) 2024-08-20T23:57:07.0545530Z adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-20T23:57:07.0546415Z adding: test/test-reports/training_torchbench.csv (deflated 67%) 2024-08-20T23:57:07.0553592Z adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 96%) 2024-08-20T23:57:07.0554799Z adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 76%) 2024-08-20T23:57:07.0588289Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:57:07.0588911Z # Remove any previous usage logs if they exist 2024-08-20T23:57:07.0589416Z rm -f logs-*.zip 2024-08-20T23:57:07.0590081Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:57:07.0591193Z # so check to see if the file exists first 2024-08-20T23:57:07.0591690Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:57:07.0592215Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:57:07.0592698Z fi 2024-08-20T23:57:07.0593060Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:57:07.0593626Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:57:07.0594123Z fi 2024-08-20T23:57:07.0602904Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:07.0603398Z env: 2024-08-20T23:57:07.0603664Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:07.0604109Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:07.0604837Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:07.0605720Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039 2024-08-20T23:57:07.0606371Z ##[endgroup] 2024-08-20T23:57:07.0709870Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:57:07.0800925Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:57:07.0801646Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:57:07.0802204Z rm -f debug-*.zip 2024-08-20T23:57:07.0802590Z if [ -d 'test/debug' ]; then 2024-08-20T23:57:07.0803089Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:57:07.0803564Z fi 2024-08-20T23:57:07.0811687Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:07.0812178Z env: 2024-08-20T23:57:07.0812452Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:07.0812905Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:07.0813633Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:07.0814543Z FILE_SUFFIX: test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039 2024-08-20T23:57:07.0815198Z ##[endgroup] 2024-08-20T23:57:07.0925782Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:57:07.0926692Z with: 2024-08-20T23:57:07.0927119Z s3-bucket: gha-artifacts 2024-08-20T23:57:07.0927776Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:07.0928500Z retention-days: 14 2024-08-20T23:57:07.0929034Z if-no-files-found: warn 2024-08-20T23:57:07.0929545Z path: test-jsons-*.zip 2024-08-20T23:57:07.0930032Z name: artifact 2024-08-20T23:57:07.0930450Z region: us-east-1 2024-08-20T23:57:07.0930916Z env: 2024-08-20T23:57:07.0931355Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:07.0932034Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:07.0933160Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:07.0934268Z ##[endgroup] 2024-08-20T23:57:07.4647315Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:57:07.4648056Z With the provided path, there will be 1 file uploaded 2024-08-20T23:57:07.4648677Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:07.5648136Z Starting upload of test-jsons-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:07.6990025Z Finished upload of test-jsons-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:07.7271328Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:57:07.7271779Z with: 2024-08-20T23:57:07.7272073Z s3-bucket: gha-artifacts 2024-08-20T23:57:07.7272513Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:07.7273000Z retention-days: 14 2024-08-20T23:57:07.7273349Z if-no-files-found: error 2024-08-20T23:57:07.7273725Z path: test-reports-*.zip 2024-08-20T23:57:07.7274073Z name: artifact 2024-08-20T23:57:07.7274385Z region: us-east-1 2024-08-20T23:57:07.7274687Z env: 2024-08-20T23:57:07.7275152Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:07.7275792Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:07.7276542Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:07.7277191Z ##[endgroup] 2024-08-20T23:57:08.0888531Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:57:08.0889335Z With the provided path, there will be 1 file uploaded 2024-08-20T23:57:08.0890099Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:08.0947866Z Starting upload of test-reports-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:08.2081020Z Finished upload of test-reports-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:08.2364534Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:57:08.2364988Z with: 2024-08-20T23:57:08.2365272Z s3-bucket: gha-artifacts 2024-08-20T23:57:08.2365724Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:08.2366226Z retention-days: 14 2024-08-20T23:57:08.2383512Z if-no-files-found: ignore 2024-08-20T23:57:08.2383889Z path: logs-*.zip 2024-08-20T23:57:08.2384205Z name: artifact 2024-08-20T23:57:08.2384506Z region: us-east-1 2024-08-20T23:57:08.2384804Z env: 2024-08-20T23:57:08.2385076Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:08.2385531Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:08.2386276Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:08.2386925Z ##[endgroup] 2024-08-20T23:57:08.5930537Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:57:08.5931149Z With the provided path, there will be 1 file uploaded 2024-08-20T23:57:08.5931765Z Uploading to s3 prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:08.5987505Z Starting upload of logs-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:08.7638412Z Finished upload of logs-test-aot_eager_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_29026299039.zip 2024-08-20T23:57:08.7922406Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:57:08.7922850Z with: 2024-08-20T23:57:08.7923130Z s3-bucket: gha-artifacts 2024-08-20T23:57:08.7923569Z s3-prefix: pytorch/pytorch/10479310721/1/artifact 2024-08-20T23:57:08.7924041Z retention-days: 14 2024-08-20T23:57:08.7924368Z if-no-files-found: ignore 2024-08-20T23:57:08.7924737Z path: debug-*.zip 2024-08-20T23:57:08.7925049Z name: artifact 2024-08-20T23:57:08.7925361Z region: us-east-1 2024-08-20T23:57:08.7925648Z env: 2024-08-20T23:57:08.7926203Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:08.7926663Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:08.7927403Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:08.7928041Z ##[endgroup] 2024-08-20T23:57:09.1432587Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:57:09.1711557Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:57:09.1712049Z # shellcheck disable=SC2156 2024-08-20T23:57:09.1712921Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:57:09.1725701Z shell: /usr/bin/bash -e {0} 2024-08-20T23:57:09.1726280Z env: 2024-08-20T23:57:09.1726560Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:09.1727011Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:09.1727750Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:09.1728391Z ##[endgroup] 2024-08-20T23:57:09.4406202Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:57:09.4406766Z with: 2024-08-20T23:57:09.4407027Z env: 2024-08-20T23:57:09.4407303Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:09.4407745Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:09.4408660Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:09.4409290Z ##[endgroup] 2024-08-20T23:57:09.4433807Z ##[group]Run set -eou pipefail 2024-08-20T23:57:09.4434210Z set -eou pipefail 2024-08-20T23:57:09.4434561Z  2024-08-20T23:57:09.4435063Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:57:09.4435690Z for _ in $(seq 1440); do 2024-08-20T23:57:09.4436161Z  # Break if no ssh session exists anymore 2024-08-20T23:57:09.4436655Z  if [ "$(who)" = "" ]; then 2024-08-20T23:57:09.4437055Z  break 2024-08-20T23:57:09.4437451Z  fi 2024-08-20T23:57:09.4437746Z  echo "." 2024-08-20T23:57:09.4438080Z  sleep 5 2024-08-20T23:57:09.4438395Z done 2024-08-20T23:57:09.4447170Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:09.4447664Z env: 2024-08-20T23:57:09.4447951Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:09.4448406Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:09.4449137Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:09.4449773Z ##[endgroup] 2024-08-20T23:57:09.4477718Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:57:09.4559033Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:57:09.4559811Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:57:09.4560405Z # shellcheck disable=SC2046 2024-08-20T23:57:09.4560890Z docker stop $(docker ps -q) || true 2024-08-20T23:57:09.4561363Z # Prune all of the docker images 2024-08-20T23:57:09.4561816Z docker system prune -af 2024-08-20T23:57:09.4570684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:09.4571188Z env: 2024-08-20T23:57:09.4571460Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:57:09.4571926Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:57:09.4572673Z DOCKER_CONTAINER_ID: 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:09.4573325Z ##[endgroup] 2024-08-20T23:57:10.1717081Z 9bd17ee7407b 2024-08-20T23:57:14.2272251Z Deleted Containers: 2024-08-20T23:57:14.2272877Z 9bd17ee7407b06adca528f85e22ae929f5168c6da99cd498e799b39dd6d7399e 2024-08-20T23:57:14.2273451Z 2024-08-20T23:57:23.1572110Z Deleted Images: 2024-08-20T23:57:23.1574050Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:57:23.1577061Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:57:23.1579090Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:57:23.1580223Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:57:23.1581406Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:57:23.1582501Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:57:23.1583604Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:57:23.1584741Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:57:23.1585845Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:57:23.1587329Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:57:23.1588452Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:57:23.1589580Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:57:23.1590960Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:57:23.1592065Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:57:23.1593183Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:57:23.1594281Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:57:23.1595391Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:57:23.1596509Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:57:23.1597780Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:57:23.1598901Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:57:23.1600010Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:57:23.1601133Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:57:23.1602257Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:57:23.1603380Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:57:23.1604496Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:57:23.1605604Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:57:23.1606713Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:57:23.1607838Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:57:23.1608949Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:57:23.1610084Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:57:23.1611210Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:57:23.1612342Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:57:23.1613450Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:57:23.1614553Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:57:23.1615365Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:57:23.1616181Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:57:23.1617004Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:57:23.1617819Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:57:23.1618639Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:57:23.1619460Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:57:23.1620289Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:57:23.1621104Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:57:23.1621928Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:57:23.1622770Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:57:23.1623587Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:57:23.1624671Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:57:23.1625643Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:57:23.1626961Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:57:23.1628009Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:57:23.1629210Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:57:23.1630259Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:57:23.1631329Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:57:23.1632403Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:57:23.1633360Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:57:23.1634183Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:57:23.1635028Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:57:23.1635871Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:57:23.1636697Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:57:23.1637608Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:57:23.1638425Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:57:23.1639231Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:57:23.1640039Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:57:23.1640861Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:57:23.1641705Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:57:23.1642537Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:57:23.1643357Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:57:23.1644218Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:57:23.1645042Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:57:23.1645874Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:57:23.1646705Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:57:23.1647538Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:57:23.1648361Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:57:23.1649176Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:57:23.1649992Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:57:23.1650829Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:57:23.1651643Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:57:23.1652483Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:57:23.1653330Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:57:23.1654171Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:57:23.1655002Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:57:23.1655829Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:57:23.1656332Z 2024-08-20T23:57:23.1656484Z Total reclaimed space: 49.95GB 2024-08-20T23:57:23.1723683Z Post job cleanup. 2024-08-20T23:57:23.1773904Z Post job cleanup. 2024-08-20T23:57:23.2701033Z [command]/usr/bin/git version 2024-08-20T23:57:23.2764664Z git version 2.40.1 2024-08-20T23:57:23.2807146Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/d6285d1b-c973-419a-9c36-43514ef5ccb2' before making global git config changes 2024-08-20T23:57:23.2809002Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:57:23.2813762Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:57:23.2864169Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:57:23.2910671Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:57:23.3295136Z Entering 'android/libs/fbjni' 2024-08-20T23:57:23.3362603Z Entering 'third_party/FP16' 2024-08-20T23:57:23.3430445Z Entering 'third_party/FXdiv' 2024-08-20T23:57:23.3497519Z Entering 'third_party/NNPACK' 2024-08-20T23:57:23.3565069Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:57:23.3635296Z Entering 'third_party/XNNPACK' 2024-08-20T23:57:23.3718935Z Entering 'third_party/benchmark' 2024-08-20T23:57:23.3789784Z Entering 'third_party/cpp-httplib' 2024-08-20T23:57:23.3857805Z Entering 'third_party/cpuinfo' 2024-08-20T23:57:23.3927998Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:57:23.4000079Z Entering 'third_party/cutlass' 2024-08-20T23:57:23.4075564Z Entering 'third_party/eigen' 2024-08-20T23:57:23.4144935Z Entering 'third_party/fbgemm' 2024-08-20T23:57:23.4211733Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:57:23.4278639Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:57:23.4345852Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:57:23.4418639Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:57:23.4487031Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:57:23.4556774Z Entering 'third_party/flatbuffers' 2024-08-20T23:57:23.4632151Z Entering 'third_party/fmt' 2024-08-20T23:57:23.4701457Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:57:23.4771136Z Entering 'third_party/gloo' 2024-08-20T23:57:23.4841344Z Entering 'third_party/googletest' 2024-08-20T23:57:23.4911729Z Entering 'third_party/ideep' 2024-08-20T23:57:23.4978777Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:57:23.5056089Z Entering 'third_party/ittapi' 2024-08-20T23:57:23.5125194Z Entering 'third_party/kineto' 2024-08-20T23:57:23.5193250Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:57:23.5260182Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:57:23.5330761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:57:23.5399612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:57:23.5468352Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:57:23.5534143Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:57:23.5604611Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:57:23.5672949Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:57:23.5742236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:57:23.5812226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:57:23.5883859Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:57:23.5951361Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:57:23.6021660Z Entering 'third_party/mimalloc' 2024-08-20T23:57:23.6093962Z Entering 'third_party/nccl/nccl' 2024-08-20T23:57:23.6165003Z Entering 'third_party/nlohmann' 2024-08-20T23:57:23.6232944Z Entering 'third_party/onnx' 2024-08-20T23:57:23.6314663Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:57:23.6381930Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:57:23.6452833Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:57:23.6521179Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:57:23.6588411Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:57:23.6655542Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:57:23.6721593Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:57:23.6791677Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:57:23.6860288Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:57:23.6926630Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:57:23.6990909Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:57:23.7059119Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:57:23.7129802Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:57:23.7217284Z Entering 'third_party/pocketfft' 2024-08-20T23:57:23.7284731Z Entering 'third_party/protobuf' 2024-08-20T23:57:23.7353514Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:57:23.7421078Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:57:23.7492067Z Entering 'third_party/psimd' 2024-08-20T23:57:23.7563470Z Entering 'third_party/pthreadpool' 2024-08-20T23:57:23.7631295Z Entering 'third_party/pybind11' 2024-08-20T23:57:23.7699944Z Entering 'third_party/python-peachpy' 2024-08-20T23:57:23.7768882Z Entering 'third_party/sleef' 2024-08-20T23:57:23.7839289Z Entering 'third_party/tensorpipe' 2024-08-20T23:57:23.7906088Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:57:23.7971547Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:57:23.8036511Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:57:23.8101814Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:57:23.8165644Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:57:23.8259912Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:57:23.8296304Z http.https://github.com/.extraheader 2024-08-20T23:57:23.8307053Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:57:23.8355882Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:57:23.8722313Z Entering 'android/libs/fbjni' 2024-08-20T23:57:23.8766818Z http.https://github.com/.extraheader 2024-08-20T23:57:23.8812001Z Entering 'third_party/FP16' 2024-08-20T23:57:23.8857645Z http.https://github.com/.extraheader 2024-08-20T23:57:23.8902126Z Entering 'third_party/FXdiv' 2024-08-20T23:57:23.8947741Z http.https://github.com/.extraheader 2024-08-20T23:57:23.8991309Z Entering 'third_party/NNPACK' 2024-08-20T23:57:23.9035965Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9080178Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:57:23.9124377Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9169356Z Entering 'third_party/XNNPACK' 2024-08-20T23:57:23.9214502Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9277179Z Entering 'third_party/benchmark' 2024-08-20T23:57:23.9322231Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9367991Z Entering 'third_party/cpp-httplib' 2024-08-20T23:57:23.9412207Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9457394Z Entering 'third_party/cpuinfo' 2024-08-20T23:57:23.9500595Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9545150Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:57:23.9587982Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9631067Z Entering 'third_party/cutlass' 2024-08-20T23:57:23.9674065Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9723732Z Entering 'third_party/eigen' 2024-08-20T23:57:23.9767068Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9811914Z Entering 'third_party/fbgemm' 2024-08-20T23:57:23.9857935Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9899803Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:57:23.9941616Z http.https://github.com/.extraheader 2024-08-20T23:57:23.9983302Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:57:24.0023946Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0066924Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:57:24.0106405Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0154432Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:57:24.0195470Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0237478Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:57:24.0279265Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0322685Z Entering 'third_party/flatbuffers' 2024-08-20T23:57:24.0366686Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0413090Z Entering 'third_party/fmt' 2024-08-20T23:57:24.0457442Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0499952Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:57:24.0544091Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0587408Z Entering 'third_party/gloo' 2024-08-20T23:57:24.0631532Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0675160Z Entering 'third_party/googletest' 2024-08-20T23:57:24.0717917Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0762568Z Entering 'third_party/ideep' 2024-08-20T23:57:24.0806234Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0849939Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:57:24.0895526Z http.https://github.com/.extraheader 2024-08-20T23:57:24.0950504Z Entering 'third_party/ittapi' 2024-08-20T23:57:24.0996812Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1042159Z Entering 'third_party/kineto' 2024-08-20T23:57:24.1090242Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1132620Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:57:24.1175428Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1216123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:57:24.1258544Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1302565Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:57:24.1344802Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1388347Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:57:24.1434014Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1479662Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:57:24.1522553Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1565530Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:57:24.1609420Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1657639Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:57:24.1700715Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1745567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:57:24.1788897Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1833718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:57:24.1876610Z http.https://github.com/.extraheader 2024-08-20T23:57:24.1921765Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:57:24.1970204Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2018784Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:57:24.2063401Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2108070Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:57:24.2154261Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2201997Z Entering 'third_party/mimalloc' 2024-08-20T23:57:24.2246476Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2290028Z Entering 'third_party/nccl/nccl' 2024-08-20T23:57:24.2333298Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2378055Z Entering 'third_party/nlohmann' 2024-08-20T23:57:24.2421904Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2467706Z Entering 'third_party/onnx' 2024-08-20T23:57:24.2509537Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2568861Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:57:24.2612031Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2656091Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:57:24.2698928Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2747457Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:57:24.2792197Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2837041Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:57:24.2879629Z http.https://github.com/.extraheader 2024-08-20T23:57:24.2922791Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:57:24.2965839Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3009281Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:57:24.3051890Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3094308Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:57:24.3142463Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3185342Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:57:24.3232451Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3274875Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:57:24.3322283Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3365351Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:57:24.3408277Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3451090Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:57:24.3494084Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3540028Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:57:24.3581721Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3629489Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:57:24.3674161Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3738430Z Entering 'third_party/pocketfft' 2024-08-20T23:57:24.3781862Z http.https://github.com/.extraheader 2024-08-20T23:57:24.3825224Z Entering 'third_party/protobuf' 2024-08-20T23:57:24.3868609Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4452120Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:57:24.4494798Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4539275Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:57:24.4582121Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4629254Z Entering 'third_party/psimd' 2024-08-20T23:57:24.4673548Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4717573Z Entering 'third_party/pthreadpool' 2024-08-20T23:57:24.4763496Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4806895Z Entering 'third_party/pybind11' 2024-08-20T23:57:24.4850692Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4894181Z Entering 'third_party/python-peachpy' 2024-08-20T23:57:24.4939691Z http.https://github.com/.extraheader 2024-08-20T23:57:24.4983930Z Entering 'third_party/sleef' 2024-08-20T23:57:24.5028113Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5071932Z Entering 'third_party/tensorpipe' 2024-08-20T23:57:24.5115843Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5159865Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:57:24.5202268Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5249958Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:57:24.5292010Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5336280Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:57:24.5379419Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5422208Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:57:24.5469972Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5511838Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:57:24.5555614Z http.https://github.com/.extraheader 2024-08-20T23:57:24.5720203Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:57:24.5740934Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:57:24.5748991Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:57:24.5749504Z ##[endgroup] 2024-08-20T23:57:31.5484501Z Cleaning up orphan processes