2024-08-20T22:10:03.1238038Z Current runner version: '2.319.1' 2024-08-20T22:10:03.1244371Z Runner name: 'i-0ed5a1a6923b38c4c' 2024-08-20T22:10:03.1245264Z Runner group name: 'Default' 2024-08-20T22:10:03.1246232Z Machine name: 'ip-10-0-8-182' 2024-08-20T22:10:03.1262751Z Testing runner upgrade compatibility 2024-08-20T22:10:03.2271873Z ##[group]GITHUB_TOKEN Permissions 2024-08-20T22:10:03.2274106Z Actions: read 2024-08-20T22:10:03.2274676Z Attestations: read 2024-08-20T22:10:03.2275159Z Checks: read 2024-08-20T22:10:03.2275716Z Contents: read 2024-08-20T22:10:03.2276255Z Deployments: read 2024-08-20T22:10:03.2276787Z Discussions: read 2024-08-20T22:10:03.2277372Z Issues: read 2024-08-20T22:10:03.2277893Z Metadata: read 2024-08-20T22:10:03.2278398Z Packages: read 2024-08-20T22:10:03.2278916Z Pages: read 2024-08-20T22:10:03.2279424Z PullRequests: read 2024-08-20T22:10:03.2279987Z RepositoryProjects: read 2024-08-20T22:10:03.2280668Z SecurityEvents: read 2024-08-20T22:10:03.2281174Z Statuses: read 2024-08-20T22:10:03.2281937Z ##[endgroup] 2024-08-20T22:10:03.2285275Z Secret source: Actions 2024-08-20T22:10:03.2286150Z Prepare workflow directory 2024-08-20T22:10:03.7611243Z Prepare all required actions 2024-08-20T22:10:03.7785964Z Getting action download info 2024-08-20T22:10:04.0026089Z Download action repository 'pytorch/test-infra@main' (SHA:0c3a2634aaa2f638c8f640e743f03d696ce1191f) 2024-08-20T22:10:04.6325502Z Download action repository 'pytorch/pytorch@main' (SHA:1ae5d5bb62141d7c9b1b0b66c66a462b4e10b1f2) 2024-08-20T22:10:08.6357730Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-08-20T22:10:08.8555637Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-08-20T22:10:09.1346869Z Getting action download info 2024-08-20T22:10:09.2257713Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-08-20T22:10:09.4186885Z Getting action download info 2024-08-20T22:10:09.5032472Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-08-20T22:10:09.6567354Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/133712 (40ec5f6ddd9787aca0449b24128343ff4c4a88b3) 2024-08-20T22:10:09.6569869Z ##[group] Inputs 2024-08-20T22:10:09.6570335Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:10:09.6579445Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:10:09.6589355Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:10:09.6590509Z sync-tag: 2024-08-20T22:10:09.6591326Z timeout-minutes: 240 2024-08-20T22:10:09.6591657Z use-gha: 2024-08-20T22:10:09.6591956Z dashboard-tag: 2024-08-20T22:10:09.6592275Z s3-bucket: gha-artifacts 2024-08-20T22:10:09.6592619Z aws-role-to-assume: 2024-08-20T22:10:09.6592947Z ##[endgroup] 2024-08-20T22:10:09.6593807Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:10:09.7277554Z A job started hook has been configured by the self-hosted runner administrator 2024-08-20T22:10:09.7417259Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2024-08-20T22:10:09.7428851Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:09.7429412Z ##[endgroup] 2024-08-20T22:10:11.5830856Z Runner Type: amz2023.linux.g5.4xlarge.nvidia.gpu 2024-08-20T22:10:11.5831397Z Instance Type: g5.4xlarge 2024-08-20T22:10:11.5832113Z AMI Name: al2023-ami-2023.5.20240701.0-kernel-6.1-x86_64 2024-08-20T22:10:11.5832878Z AMI ID: ami-06c68f701d8090592 2024-08-20T22:10:17.4177194Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-08-20T22:10:17.4177774Z with: 2024-08-20T22:10:17.4178445Z github-secret: *** 2024-08-20T22:10:17.4179356Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:17.4180324Z activate-with-label: false 2024-08-20T22:10:17.4180693Z label: with-ssh 2024-08-20T22:10:17.4181019Z remove-existing-keys: true 2024-08-20T22:10:17.4181391Z fail-silently: true 2024-08-20T22:10:17.4181715Z env: 2024-08-20T22:10:17.4182024Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:17.4182371Z ##[endgroup] 2024-08-20T22:10:17.5080290Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-08-20T22:10:17.5083466Z ciflow reference detected, attempting to extract PR number 2024-08-20T22:10:17.7573057Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-08-20T22:10:17.8192318Z No SSH keys found for user pytorch-bot[bot] 2024-08-20T22:10:17.8193097Z Grabbing public ssh keys from https://github.com/XuehaiPan.keys 2024-08-20T22:10:17.8997161Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh 2024-08-20T22:10:17.9011539Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys 2024-08-20T22:10:17.9036963Z Login using: ssh ec2-user@ec2-54-242-201-218.compute-1.amazonaws.com 2024-08-20T22:10:17.9038118Z All testing is done inside the container, to start an interactive session run: 2024-08-20T22:10:17.9038931Z docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-08-20T22:10:17.9170563Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-08-20T22:10:17.9171110Z with: 2024-08-20T22:10:17.9171383Z submodules: recursive 2024-08-20T22:10:17.9171709Z fetch-depth: 0 2024-08-20T22:10:17.9171994Z env: 2024-08-20T22:10:17.9172259Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:17.9172591Z ##[endgroup] 2024-08-20T22:10:17.9366745Z ##[group]Run retry () { 2024-08-20T22:10:17.9367134Z retry () { 2024-08-20T22:10:17.9367858Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-08-20T22:10:17.9368424Z } 2024-08-20T22:10:17.9368729Z echo "${GITHUB_WORKSPACE}" 2024-08-20T22:10:17.9369156Z if [ -z "${NO_SUDO}" ]; then 2024-08-20T22:10:17.9369638Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:17.9370097Z else 2024-08-20T22:10:17.9370443Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-08-20T22:10:17.9370873Z fi 2024-08-20T22:10:17.9371212Z mkdir "${GITHUB_WORKSPACE}" 2024-08-20T22:10:17.9381429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:10:17.9381931Z env: 2024-08-20T22:10:17.9382210Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:17.9382547Z NO_SUDO: 2024-08-20T22:10:17.9382825Z ##[endgroup] 2024-08-20T22:10:17.9414894Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:23.7418466Z ##[group]Run malfet/checkout@silent-checkout 2024-08-20T22:10:23.7418910Z with: 2024-08-20T22:10:23.7419238Z ref: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:10:23.7419692Z fetch-depth: 0 2024-08-20T22:10:23.7420007Z submodules: recursive 2024-08-20T22:10:23.7420340Z quiet-checkout: true 2024-08-20T22:10:23.7420684Z repository: pytorch/pytorch 2024-08-20T22:10:23.7421162Z token: *** 2024-08-20T22:10:23.7421453Z ssh-strict: true 2024-08-20T22:10:23.7421774Z persist-credentials: true 2024-08-20T22:10:23.7422131Z clean: true 2024-08-20T22:10:23.7422463Z sparse-checkout-cone-mode: true 2024-08-20T22:10:23.7422854Z lfs: false 2024-08-20T22:10:23.7423152Z set-safe-directory: true 2024-08-20T22:10:23.7423489Z env: 2024-08-20T22:10:23.7423759Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:10:23.7424091Z ##[endgroup] 2024-08-20T22:10:23.8403408Z Syncing repository: pytorch/pytorch 2024-08-20T22:10:23.8405034Z ##[group]Getting Git version info 2024-08-20T22:10:23.8405822Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:23.8406675Z [command]/usr/bin/git version 2024-08-20T22:10:23.8407040Z git version 2.40.1 2024-08-20T22:10:23.8415782Z ##[endgroup] 2024-08-20T22:10:23.8432082Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/158078cf-3f3b-4b4f-9102-71ba59b9fd1a' before making global git config changes 2024-08-20T22:10:23.8433581Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T22:10:23.8436454Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:23.8486527Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-08-20T22:10:23.8490546Z ##[group]Initializing the repository 2024-08-20T22:10:23.8493315Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:10:23.8537716Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-08-20T22:10:23.8538577Z hint: is subject to change. To configure the initial branch name to use in all 2024-08-20T22:10:23.8539379Z hint: of your new repositories, which will suppress this warning, call: 2024-08-20T22:10:23.8539923Z hint: 2024-08-20T22:10:23.8540546Z hint: git config --global init.defaultBranch 2024-08-20T22:10:23.8541014Z hint: 2024-08-20T22:10:23.8541536Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-08-20T22:10:23.8542373Z hint: 'development'. The just-created branch can be renamed via this command: 2024-08-20T22:10:23.8543192Z hint: 2024-08-20T22:10:23.8543529Z hint: git branch -m 2024-08-20T22:10:23.8544266Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-08-20T22:10:23.8551081Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-08-20T22:10:23.8593614Z ##[endgroup] 2024-08-20T22:10:23.8594192Z ##[group]Disabling automatic garbage collection 2024-08-20T22:10:23.8596469Z [command]/usr/bin/git config --local gc.auto 0 2024-08-20T22:10:23.8638406Z ##[endgroup] 2024-08-20T22:10:23.8638927Z ##[group]Setting up auth 2024-08-20T22:10:23.8644272Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T22:10:23.8688434Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T22:10:23.9040146Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T22:10:23.9081515Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T22:10:23.9440182Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:10:23.9500533Z ##[endgroup] 2024-08-20T22:10:23.9501099Z ##[group]Fetching the repository 2024-08-20T22:10:23.9506643Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-08-20T22:10:26.4571468Z remote: Enumerating objects: 1008452 2024-08-20T22:10:26.4572540Z remote: Enumerating objects: 1010229, done. 2024-08-20T22:10:26.4573839Z remote: Counting objects: 0% (1/1777) 2024-08-20T22:10:26.4574343Z remote: Counting objects: 1% (18/1777) 2024-08-20T22:10:26.4574883Z remote: Counting objects: 2% (36/1777) 2024-08-20T22:10:26.4575369Z remote: Counting objects: 3% (54/1777) 2024-08-20T22:10:26.4575853Z remote: Counting objects: 4% (72/1777) 2024-08-20T22:10:26.4576344Z remote: Counting objects: 5% (89/1777) 2024-08-20T22:10:26.4576840Z remote: Counting objects: 6% (107/1777) 2024-08-20T22:10:26.4577339Z remote: Counting objects: 7% (125/1777) 2024-08-20T22:10:26.4577832Z remote: Counting objects: 8% (143/1777) 2024-08-20T22:10:26.4578330Z remote: Counting objects: 9% (160/1777) 2024-08-20T22:10:26.4578821Z remote: Counting objects: 10% (178/1777) 2024-08-20T22:10:26.4579326Z remote: Counting objects: 11% (196/1777) 2024-08-20T22:10:26.4579809Z remote: Counting objects: 12% (214/1777) 2024-08-20T22:10:26.4580444Z remote: Counting objects: 13% (232/1777) 2024-08-20T22:10:26.4581057Z remote: Counting objects: 14% (249/1777) 2024-08-20T22:10:26.4581663Z remote: Counting objects: 15% (267/1777) 2024-08-20T22:10:26.4582148Z remote: Counting objects: 16% (285/1777) 2024-08-20T22:10:26.4582637Z remote: Counting objects: 17% (303/1777) 2024-08-20T22:10:26.4583134Z remote: Counting objects: 18% (320/1777) 2024-08-20T22:10:26.4583662Z remote: Counting objects: 19% (338/1777) 2024-08-20T22:10:26.4584137Z remote: Counting objects: 20% (356/1777) 2024-08-20T22:10:26.4584626Z remote: Counting objects: 21% (374/1777) 2024-08-20T22:10:26.4585113Z remote: Counting objects: 22% (391/1777) 2024-08-20T22:10:26.4585706Z remote: Counting objects: 23% (409/1777) 2024-08-20T22:10:26.4586201Z remote: Counting objects: 24% (427/1777) 2024-08-20T22:10:26.4586686Z remote: Counting objects: 25% (445/1777) 2024-08-20T22:10:26.4587167Z remote: Counting objects: 26% (463/1777) 2024-08-20T22:10:26.4587648Z remote: Counting objects: 27% (480/1777) 2024-08-20T22:10:26.4588615Z remote: Counting objects: 28% (498/1777) 2024-08-20T22:10:26.4589171Z remote: Counting objects: 29% (516/1777) 2024-08-20T22:10:26.4589659Z remote: Counting objects: 30% (534/1777) 2024-08-20T22:10:26.4590263Z remote: Counting objects: 31% (551/1777) 2024-08-20T22:10:26.4590944Z remote: Counting objects: 32% (569/1777) 2024-08-20T22:10:26.4591582Z remote: Counting objects: 33% (587/1777) 2024-08-20T22:10:26.4592514Z remote: Counting objects: 34% (605/1777) 2024-08-20T22:10:26.4593148Z remote: Counting objects: 35% (622/1777) 2024-08-20T22:10:26.4593679Z remote: Counting objects: 36% (640/1777) 2024-08-20T22:10:26.4594171Z remote: Counting objects: 37% (658/1777) 2024-08-20T22:10:26.4594671Z remote: Counting objects: 38% (676/1777) 2024-08-20T22:10:26.4595170Z remote: Counting objects: 39% (694/1777) 2024-08-20T22:10:26.4595656Z remote: Counting objects: 40% (711/1777) 2024-08-20T22:10:26.4596162Z remote: Counting objects: 41% (729/1777) 2024-08-20T22:10:26.4596655Z remote: Counting objects: 42% (747/1777) 2024-08-20T22:10:26.4597143Z remote: Counting objects: 43% (765/1777) 2024-08-20T22:10:26.4597632Z remote: Counting objects: 44% (782/1777) 2024-08-20T22:10:26.4598132Z remote: Counting objects: 45% (800/1777) 2024-08-20T22:10:26.4598622Z remote: Counting objects: 46% (818/1777) 2024-08-20T22:10:26.4599119Z remote: Counting objects: 47% (836/1777) 2024-08-20T22:10:26.4599609Z remote: Counting objects: 48% (853/1777) 2024-08-20T22:10:26.4600097Z remote: Counting objects: 49% (871/1777) 2024-08-20T22:10:26.4600589Z remote: Counting objects: 50% (889/1777) 2024-08-20T22:10:26.4601082Z remote: Counting objects: 51% (907/1777) 2024-08-20T22:10:26.4601568Z remote: Counting objects: 52% (925/1777) 2024-08-20T22:10:26.4602059Z remote: Counting objects: 53% (942/1777) 2024-08-20T22:10:26.4602562Z remote: Counting objects: 54% (960/1777) 2024-08-20T22:10:26.4603054Z remote: Counting objects: 55% (978/1777) 2024-08-20T22:10:26.4603534Z remote: Counting objects: 56% (996/1777) 2024-08-20T22:10:26.4604041Z remote: Counting objects: 57% (1013/1777) 2024-08-20T22:10:26.4604567Z remote: Counting objects: 58% (1031/1777) 2024-08-20T22:10:26.4605189Z remote: Counting objects: 59% (1049/1777) 2024-08-20T22:10:26.4605696Z remote: Counting objects: 60% (1067/1777) 2024-08-20T22:10:26.4606207Z remote: Counting objects: 61% (1084/1777) 2024-08-20T22:10:26.4606712Z remote: Counting objects: 62% (1102/1777) 2024-08-20T22:10:26.4607229Z remote: Counting objects: 63% (1120/1777) 2024-08-20T22:10:26.4607737Z remote: Counting objects: 64% (1138/1777) 2024-08-20T22:10:26.4608232Z remote: Counting objects: 65% (1156/1777) 2024-08-20T22:10:26.4608753Z remote: Counting objects: 66% (1173/1777) 2024-08-20T22:10:26.4609264Z remote: Counting objects: 67% (1191/1777) 2024-08-20T22:10:26.4609776Z remote: Counting objects: 68% (1209/1777) 2024-08-20T22:10:26.4610280Z remote: Counting objects: 69% (1227/1777) 2024-08-20T22:10:26.4610788Z remote: Counting objects: 70% (1244/1777) 2024-08-20T22:10:26.4611300Z remote: Counting objects: 71% (1262/1777) 2024-08-20T22:10:26.4611805Z remote: Counting objects: 72% (1280/1777) 2024-08-20T22:10:26.4612313Z remote: Counting objects: 73% (1298/1777) 2024-08-20T22:10:26.4612821Z remote: Counting objects: 74% (1315/1777) 2024-08-20T22:10:26.4613332Z remote: Counting objects: 75% (1333/1777) 2024-08-20T22:10:26.4613847Z remote: Counting objects: 76% (1351/1777) 2024-08-20T22:10:26.4614343Z remote: Counting objects: 77% (1369/1777) 2024-08-20T22:10:26.4614848Z remote: Counting objects: 78% (1387/1777) 2024-08-20T22:10:26.4615466Z remote: Counting objects: 79% (1404/1777) 2024-08-20T22:10:26.4615971Z remote: Counting objects: 80% (1422/1777) 2024-08-20T22:10:26.4616464Z remote: Counting objects: 81% (1440/1777) 2024-08-20T22:10:26.4616970Z remote: Counting objects: 82% (1458/1777) 2024-08-20T22:10:26.4617473Z remote: Counting objects: 83% (1475/1777) 2024-08-20T22:10:26.4617981Z remote: Counting objects: 84% (1493/1777) 2024-08-20T22:10:26.4618480Z remote: Counting objects: 85% (1511/1777) 2024-08-20T22:10:26.4619077Z remote: Counting objects: 86% (1529/1777) 2024-08-20T22:10:26.4619582Z remote: Counting objects: 87% (1546/1777) 2024-08-20T22:10:26.4620078Z remote: Counting objects: 88% (1564/1777) 2024-08-20T22:10:26.4620592Z remote: Counting objects: 89% (1582/1777) 2024-08-20T22:10:26.4621104Z remote: Counting objects: 90% (1600/1777) 2024-08-20T22:10:26.4621608Z remote: Counting objects: 91% (1618/1777) 2024-08-20T22:10:26.4622113Z remote: Counting objects: 92% (1635/1777) 2024-08-20T22:10:26.4622617Z remote: Counting objects: 93% (1653/1777) 2024-08-20T22:10:26.4623126Z remote: Counting objects: 94% (1671/1777) 2024-08-20T22:10:26.4623640Z remote: Counting objects: 95% (1689/1777) 2024-08-20T22:10:26.4624167Z remote: Counting objects: 96% (1706/1777) 2024-08-20T22:10:26.4624677Z remote: Counting objects: 97% (1724/1777) 2024-08-20T22:10:26.4625184Z remote: Counting objects: 98% (1742/1777) 2024-08-20T22:10:26.4625712Z remote: Counting objects: 99% (1760/1777) 2024-08-20T22:10:26.4626219Z remote: Counting objects: 100% (1777/1777) 2024-08-20T22:10:26.4626775Z remote: Counting objects: 100% (1777/1777), done. 2024-08-20T22:10:26.4725199Z remote: Compressing objects: 0% (1/846) 2024-08-20T22:10:26.4986742Z remote: Compressing objects: 1% (9/846) 2024-08-20T22:10:26.5278622Z remote: Compressing objects: 2% (17/846) 2024-08-20T22:10:26.5633414Z remote: Compressing objects: 3% (26/846) 2024-08-20T22:10:26.6334026Z remote: Compressing objects: 4% (34/846) 2024-08-20T22:10:26.6822997Z remote: Compressing objects: 5% (43/846) 2024-08-20T22:10:26.7326261Z remote: Compressing objects: 6% (51/846) 2024-08-20T22:10:26.7713345Z remote: Compressing objects: 7% (60/846) 2024-08-20T22:10:26.8121333Z remote: Compressing objects: 8% (68/846) 2024-08-20T22:10:26.8330592Z remote: Compressing objects: 9% (77/846) 2024-08-20T22:10:26.8585819Z remote: Compressing objects: 10% (85/846) 2024-08-20T22:10:26.8776138Z remote: Compressing objects: 11% (94/846) 2024-08-20T22:10:26.8962703Z remote: Compressing objects: 12% (102/846) 2024-08-20T22:10:26.9126288Z remote: Compressing objects: 13% (110/846) 2024-08-20T22:10:26.9235535Z remote: Compressing objects: 14% (119/846) 2024-08-20T22:10:26.9336639Z remote: Compressing objects: 15% (127/846) 2024-08-20T22:10:26.9407256Z remote: Compressing objects: 16% (136/846) 2024-08-20T22:10:26.9477487Z remote: Compressing objects: 17% (144/846) 2024-08-20T22:10:26.9520281Z remote: Compressing objects: 18% (153/846) 2024-08-20T22:10:26.9570208Z remote: Compressing objects: 19% (161/846) 2024-08-20T22:10:26.9598476Z remote: Compressing objects: 20% (170/846) 2024-08-20T22:10:26.9619991Z remote: Compressing objects: 21% (178/846) 2024-08-20T22:10:26.9623973Z remote: Compressing objects: 22% (187/846) 2024-08-20T22:10:26.9628890Z remote: Compressing objects: 23% (195/846) 2024-08-20T22:10:26.9634955Z remote: Compressing objects: 24% (204/846) 2024-08-20T22:10:26.9638738Z remote: Compressing objects: 25% (212/846) 2024-08-20T22:10:26.9648598Z remote: Compressing objects: 26% (220/846) 2024-08-20T22:10:26.9651427Z remote: Compressing objects: 27% (229/846) 2024-08-20T22:10:26.9671472Z remote: Compressing objects: 28% (237/846) 2024-08-20T22:10:26.9681842Z remote: Compressing objects: 29% (246/846) 2024-08-20T22:10:26.9692955Z remote: Compressing objects: 30% (254/846) 2024-08-20T22:10:26.9702881Z remote: Compressing objects: 31% (263/846) 2024-08-20T22:10:26.9707167Z remote: Compressing objects: 32% (271/846) 2024-08-20T22:10:26.9715922Z remote: Compressing objects: 33% (280/846) 2024-08-20T22:10:26.9720900Z remote: Compressing objects: 34% (288/846) 2024-08-20T22:10:26.9725923Z remote: Compressing objects: 35% (297/846) 2024-08-20T22:10:26.9734019Z remote: Compressing objects: 36% (305/846) 2024-08-20T22:10:26.9740911Z remote: Compressing objects: 37% (314/846) 2024-08-20T22:10:26.9747421Z remote: Compressing objects: 38% (322/846) 2024-08-20T22:10:26.9754116Z remote: Compressing objects: 39% (330/846) 2024-08-20T22:10:26.9758523Z remote: Compressing objects: 40% (339/846) 2024-08-20T22:10:26.9763892Z remote: Compressing objects: 41% (347/846) 2024-08-20T22:10:26.9774325Z remote: Compressing objects: 42% (356/846) 2024-08-20T22:10:26.9777599Z remote: Compressing objects: 43% (364/846) 2024-08-20T22:10:26.9783805Z remote: Compressing objects: 44% (373/846) 2024-08-20T22:10:26.9789371Z remote: Compressing objects: 45% (381/846) 2024-08-20T22:10:26.9793434Z remote: Compressing objects: 46% (390/846) 2024-08-20T22:10:26.9796877Z remote: Compressing objects: 47% (398/846) 2024-08-20T22:10:26.9799577Z remote: Compressing objects: 48% (407/846) 2024-08-20T22:10:26.9802768Z remote: Compressing objects: 49% (415/846) 2024-08-20T22:10:26.9805513Z remote: Compressing objects: 50% (423/846) 2024-08-20T22:10:26.9808646Z remote: Compressing objects: 51% (432/846) 2024-08-20T22:10:26.9810862Z remote: Compressing objects: 52% (440/846) 2024-08-20T22:10:26.9812745Z remote: Compressing objects: 53% (449/846) 2024-08-20T22:10:26.9814689Z remote: Compressing objects: 54% (457/846) 2024-08-20T22:10:26.9815499Z remote: Compressing objects: 55% (466/846) 2024-08-20T22:10:26.9816733Z remote: Compressing objects: 56% (474/846) 2024-08-20T22:10:26.9817447Z remote: Compressing objects: 57% (483/846) 2024-08-20T22:10:26.9818109Z remote: Compressing objects: 58% (491/846) 2024-08-20T22:10:26.9818722Z remote: Compressing objects: 59% (500/846) 2024-08-20T22:10:26.9819336Z remote: Compressing objects: 60% (508/846) 2024-08-20T22:10:26.9819849Z remote: Compressing objects: 61% (517/846) 2024-08-20T22:10:26.9820368Z remote: Compressing objects: 62% (525/846) 2024-08-20T22:10:26.9820888Z remote: Compressing objects: 63% (533/846) 2024-08-20T22:10:26.9825593Z remote: Compressing objects: 64% (542/846) 2024-08-20T22:10:26.9831611Z remote: Compressing objects: 65% (550/846) 2024-08-20T22:10:26.9837095Z remote: Compressing objects: 66% (559/846) 2024-08-20T22:10:26.9841504Z remote: Compressing objects: 67% (567/846) 2024-08-20T22:10:26.9845858Z remote: Compressing objects: 68% (576/846) 2024-08-20T22:10:26.9850080Z remote: Compressing objects: 69% (584/846) 2024-08-20T22:10:26.9854131Z remote: Compressing objects: 70% (593/846) 2024-08-20T22:10:26.9857318Z remote: Compressing objects: 71% (601/846) 2024-08-20T22:10:26.9860564Z remote: Compressing objects: 72% (610/846) 2024-08-20T22:10:26.9863886Z remote: Compressing objects: 73% (618/846) 2024-08-20T22:10:26.9866323Z remote: Compressing objects: 74% (627/846) 2024-08-20T22:10:26.9869245Z remote: Compressing objects: 75% (635/846) 2024-08-20T22:10:26.9871969Z remote: Compressing objects: 76% (643/846) 2024-08-20T22:10:26.9875600Z remote: Compressing objects: 77% (652/846) 2024-08-20T22:10:26.9876112Z remote: Compressing objects: 78% (660/846) 2024-08-20T22:10:26.9878469Z remote: Compressing objects: 79% (669/846) 2024-08-20T22:10:26.9880523Z remote: Compressing objects: 80% (677/846) 2024-08-20T22:10:26.9881683Z remote: Compressing objects: 81% (686/846) 2024-08-20T22:10:26.9884220Z remote: Compressing objects: 82% (694/846) 2024-08-20T22:10:26.9886034Z remote: Compressing objects: 83% (703/846) 2024-08-20T22:10:26.9887791Z remote: Compressing objects: 84% (711/846) 2024-08-20T22:10:26.9888888Z remote: Compressing objects: 85% (720/846) 2024-08-20T22:10:26.9891469Z remote: Compressing objects: 86% (728/846) 2024-08-20T22:10:26.9892333Z remote: Compressing objects: 87% (737/846) 2024-08-20T22:10:26.9894281Z remote: Compressing objects: 88% (745/846) 2024-08-20T22:10:26.9897990Z remote: Compressing objects: 89% (753/846) 2024-08-20T22:10:26.9898511Z remote: Compressing objects: 90% (762/846) 2024-08-20T22:10:26.9899655Z remote: Compressing objects: 91% (770/846) 2024-08-20T22:10:26.9900156Z remote: Compressing objects: 92% (779/846) 2024-08-20T22:10:26.9900674Z remote: Compressing objects: 93% (787/846) 2024-08-20T22:10:26.9902243Z remote: Compressing objects: 94% (796/846) 2024-08-20T22:10:26.9903129Z remote: Compressing objects: 95% (804/846) 2024-08-20T22:10:26.9904663Z remote: Compressing objects: 96% (813/846) 2024-08-20T22:10:26.9905528Z remote: Compressing objects: 97% (821/846) 2024-08-20T22:10:26.9908122Z remote: Compressing objects: 98% (830/846) 2024-08-20T22:10:26.9908649Z remote: Compressing objects: 99% (838/846) 2024-08-20T22:10:26.9909299Z remote: Compressing objects: 100% (846/846) 2024-08-20T22:10:26.9909849Z remote: Compressing objects: 100% (846/846), done. 2024-08-20T22:10:50.3930994Z remote: Total 1010229 (delta 1169), reused 1411 (delta 928), pack-reused 1008452 (from 1) 2024-08-20T22:11:12.0529330Z [command]/usr/bin/git rev-parse --verify --quiet 40ec5f6ddd9787aca0449b24128343ff4c4a88b3^{object} 2024-08-20T22:11:12.0565062Z 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:12.0570990Z ##[endgroup] 2024-08-20T22:11:12.0573953Z ##[group]Determining the checkout info 2024-08-20T22:11:12.0574677Z ##[endgroup] 2024-08-20T22:11:12.0575394Z ##[group]Checking out the ref 2024-08-20T22:11:12.0576229Z [command]/usr/bin/git checkout --quiet --force 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:11:13.7430819Z ##[endgroup] 2024-08-20T22:11:13.7431406Z ##[group]Setting up auth for fetching submodules 2024-08-20T22:11:13.7437002Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-08-20T22:11:13.7520142Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-08-20T22:11:13.7562732Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-08-20T22:11:13.7601959Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-08-20T22:11:13.7640301Z ##[endgroup] 2024-08-20T22:11:13.7640825Z ##[group]Fetching submodules 2024-08-20T22:11:13.7644323Z [command]/usr/bin/git submodule sync --recursive 2024-08-20T22:11:13.8021679Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-08-20T22:11:13.8391151Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-08-20T22:11:13.8393934Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-08-20T22:11:13.8395934Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-08-20T22:11:13.8398319Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-08-20T22:11:13.8402359Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-08-20T22:11:13.8405196Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-08-20T22:11:13.8408950Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-08-20T22:11:13.8412612Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-08-20T22:11:13.8416305Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-08-20T22:11:13.8420180Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-08-20T22:11:13.8423950Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-08-20T22:11:13.8427861Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-08-20T22:11:13.8433369Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-08-20T22:11:13.8437973Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-08-20T22:11:13.8441771Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-08-20T22:11:13.8446343Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:11:13.8454328Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-08-20T22:11:13.8458798Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-08-20T22:11:13.8463116Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-08-20T22:11:13.8467779Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-08-20T22:11:13.8472316Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-08-20T22:11:13.8476912Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-08-20T22:11:13.8481517Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-08-20T22:11:13.8486619Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-08-20T22:11:13.8491222Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-08-20T22:11:13.8496592Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-08-20T22:11:13.8501121Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-08-20T22:11:13.8506224Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-08-20T22:11:13.8511298Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-08-20T22:11:13.8516519Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-08-20T22:11:13.8521680Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-08-20T22:11:13.8527430Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-08-20T22:11:13.8532544Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-08-20T22:11:13.8543075Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-08-20T22:11:13.8576102Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-08-20T22:11:14.1955074Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-08-20T22:11:14.3837711Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-08-20T22:11:14.5716914Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-08-20T22:11:14.8138505Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-08-20T22:11:16.7885396Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-08-20T22:11:28.8826222Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-08-20T22:11:29.3048147Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-08-20T22:11:29.7582788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-08-20T22:11:30.3820785Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-08-20T22:11:31.7533270Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-08-20T22:11:33.9701685Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-08-20T22:11:40.1163481Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-08-20T22:11:41.6722760Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-08-20T22:11:43.5519972Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-08-20T22:11:44.9835594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-08-20T22:11:45.4211239Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-08-20T22:11:45.7637824Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-08-20T22:11:46.8522920Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-08-20T22:11:47.2052429Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-08-20T22:11:47.4797332Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-08-20T22:11:49.2351344Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-08-20T22:11:49.9979372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-08-20T22:11:50.7022726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-08-20T22:11:57.2039165Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-08-20T22:11:59.3492122Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-08-20T22:12:03.6937510Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-08-20T22:12:03.9752705Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-08-20T22:12:13.2557303Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-08-20T22:12:13.4193588Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-08-20T22:12:13.6078067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-08-20T22:12:14.5616941Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-08-20T22:12:14.8856333Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-08-20T22:12:15.5342005Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-08-20T22:12:15.9865211Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-08-20T22:12:16.0013835Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-08-20T22:12:16.0132501Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-08-20T22:12:16.0441896Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-08-20T22:12:16.0899447Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-08-20T22:12:17.2660458Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-08-20T22:12:17.2940061Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-08-20T22:12:17.3459710Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-08-20T22:12:17.4585629Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-08-20T22:12:17.4989705Z Submodule path 'third_party/cudnn_frontend': checked out '23511ba176243f27b3b275da1fb3814ea805a171' 2024-08-20T22:12:18.1238234Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-08-20T22:12:18.4113966Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-08-20T22:12:18.5045860Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-08-20T22:12:18.5068826Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:12:18.5072336Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:12:18.5075755Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:12:18.5079247Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:12:18.5082922Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:12:18.5114325Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-08-20T22:12:19.5220405Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-08-20T22:12:20.1620878Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-08-20T22:12:22.4733695Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-08-20T22:12:23.5404268Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-08-20T22:12:23.8793301Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-08-20T22:12:23.9902223Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-08-20T22:12:24.5000781Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-08-20T22:12:24.5711336Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-08-20T22:12:24.5863259Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-08-20T22:12:24.7379701Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-08-20T22:12:24.7848255Z Submodule path 'third_party/fmt': checked out '0c9fce2ffefecfdce794e1859584e25877b7b592' 2024-08-20T22:12:24.8309336Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-08-20T22:12:24.8638592Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-08-20T22:12:24.9171109Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:12:24.9331572Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-08-20T22:12:24.9352721Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-08-20T22:12:24.9382491Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-08-20T22:12:38.9405759Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-08-20T22:12:38.9624106Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-08-20T22:12:39.0637493Z Submodule path 'third_party/kineto': checked out 'd9753139d181b9ff42872465aac0e5d3018be415' 2024-08-20T22:12:39.0659272Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:12:39.0662308Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:12:39.0665834Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:12:39.0699040Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-08-20T22:12:39.6421111Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-08-20T22:12:40.8782235Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-08-20T22:12:42.0438264Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-08-20T22:12:42.0459663Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:12:42.0463043Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:12:42.0466601Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:12:42.0470287Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:12:42.0473918Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:12:42.0477797Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:12:42.0481570Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:12:42.0485614Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:12:42.0517807Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-08-20T22:12:42.9303483Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-08-20T22:12:43.2938864Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-08-20T22:12:44.5291504Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-08-20T22:12:44.8340468Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-08-20T22:12:45.3816524Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-08-20T22:12:46.4806014Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-08-20T22:12:53.1418888Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-08-20T22:12:53.5735277Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-08-20T22:12:53.5965690Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-08-20T22:12:53.6396417Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-08-20T22:12:53.6568499Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-08-20T22:12:53.6587640Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:12:53.6620251Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-08-20T22:12:53.9694153Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-08-20T22:12:53.9917375Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-08-20T22:12:54.0397130Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-08-20T22:12:54.1618522Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-08-20T22:12:54.1821092Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-08-20T22:12:54.2277094Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2024-08-20T22:12:54.2941151Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-08-20T22:12:54.3406812Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-08-20T22:12:54.3709433Z Submodule path 'third_party/nccl/nccl': checked out 'ab2b89c4c339bd7f816fbc114a4b05d386b66290' 2024-08-20T22:12:54.4976723Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-08-20T22:12:55.0133362Z Submodule path 'third_party/onnx': checked out '3bf92c03a9f27eba3bda1e5b9e63ea20ec213557' 2024-08-20T22:12:55.0171667Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-08-20T22:12:55.0174775Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-08-20T22:12:55.0206806Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-08-20T22:12:55.5182953Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-08-20T22:12:56.4934373Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-08-20T22:12:56.5356016Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-08-20T22:12:56.6254561Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-08-20T22:12:56.6280267Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:12:56.6283553Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:12:56.6287236Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:12:56.6290867Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:12:56.6294714Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:12:56.6298310Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:12:56.6301962Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:12:56.6305387Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:12:56.6338216Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-08-20T22:12:57.0540493Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-08-20T22:12:58.1017651Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-08-20T22:12:58.4206371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-08-20T22:13:04.9248206Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-08-20T22:13:05.2424751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-08-20T22:13:05.4383941Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-08-20T22:13:05.7460316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-08-20T22:13:12.6664570Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-08-20T22:13:12.7143587Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-08-20T22:13:12.7335905Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-08-20T22:13:12.8543658Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-08-20T22:13:12.8707203Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-08-20T22:13:12.8895093Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-08-20T22:13:12.9099733Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-08-20T22:13:12.9118981Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:12.9122507Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:12.9154041Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-08-20T22:13:14.9298075Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-08-20T22:13:16.3232486Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-08-20T22:13:16.3762588Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-08-20T22:13:17.0498145Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-08-20T22:13:17.0644874Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-08-20T22:13:17.3813959Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-08-20T22:13:17.3841795Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:17.3845200Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:17.3876597Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-08-20T22:13:17.8848988Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-08-20T22:13:19.0057128Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-08-20T22:13:19.0880453Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-08-20T22:13:19.0997319Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-08-20T22:13:19.1155042Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-08-20T22:13:19.1601781Z Submodule path 'third_party/pybind11': checked out '941f45bcb51457884fa1afd6e24a67377d70f75c' 2024-08-20T22:13:19.1949197Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-08-20T22:13:19.2438219Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-08-20T22:13:19.2780804Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-08-20T22:13:19.2801923Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:19.2805297Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:19.2809691Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:19.2813055Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:19.2845083Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-08-20T22:13:20.3726823Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-08-20T22:13:20.5960603Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-08-20T22:13:21.8119676Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-08-20T22:13:22.8774249Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-08-20T22:13:22.8964906Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-08-20T22:13:22.9758404Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-08-20T22:13:23.0113066Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-08-20T22:13:23.0132145Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:23.0164430Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-08-20T22:13:23.2236517Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-08-20T22:13:23.2287741Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-08-20T22:13:23.2655009Z Entering 'android/libs/fbjni' 2024-08-20T22:13:23.2708494Z Entering 'third_party/FP16' 2024-08-20T22:13:23.2761590Z Entering 'third_party/FXdiv' 2024-08-20T22:13:23.2815135Z Entering 'third_party/NNPACK' 2024-08-20T22:13:23.2868684Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:23.2925648Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:23.2999281Z Entering 'third_party/benchmark' 2024-08-20T22:13:23.3053238Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:23.3104745Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:23.3157069Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:23.3208891Z Entering 'third_party/cutlass' 2024-08-20T22:13:23.3268939Z Entering 'third_party/eigen' 2024-08-20T22:13:23.3323050Z Entering 'third_party/fbgemm' 2024-08-20T22:13:23.3378027Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:23.3429412Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:23.3481361Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:23.3538338Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:23.3588424Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:23.3641685Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:23.3697200Z Entering 'third_party/fmt' 2024-08-20T22:13:23.3752738Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:23.3806000Z Entering 'third_party/gloo' 2024-08-20T22:13:23.3860783Z Entering 'third_party/googletest' 2024-08-20T22:13:23.3913101Z Entering 'third_party/ideep' 2024-08-20T22:13:23.3963801Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:23.4023278Z Entering 'third_party/ittapi' 2024-08-20T22:13:23.4077063Z Entering 'third_party/kineto' 2024-08-20T22:13:23.4132068Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:23.4184031Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:23.4236875Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:23.4292711Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:23.4343823Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:23.4392932Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:23.4448010Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:23.4500813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:23.4556983Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:23.4613792Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:23.4669703Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:23.4720612Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:23.4773969Z Entering 'third_party/mimalloc' 2024-08-20T22:13:23.4826248Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:23.4884165Z Entering 'third_party/nlohmann' 2024-08-20T22:13:23.4937194Z Entering 'third_party/onnx' 2024-08-20T22:13:23.5002802Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:23.5055113Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:23.5111159Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:23.5164375Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:23.5214965Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:23.5265922Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:23.5315489Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:23.5366923Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:23.5416203Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:23.5465775Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:23.5513773Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:23.5567174Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:23.5620832Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:23.5695192Z Entering 'third_party/pocketfft' 2024-08-20T22:13:23.5747050Z Entering 'third_party/protobuf' 2024-08-20T22:13:23.5800527Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:23.5852006Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:23.5905649Z Entering 'third_party/psimd' 2024-08-20T22:13:23.5958669Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:23.6015500Z Entering 'third_party/pybind11' 2024-08-20T22:13:23.6067637Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:23.6119040Z Entering 'third_party/sleef' 2024-08-20T22:13:23.6175596Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:23.6226074Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:23.6276998Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:23.6326761Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:23.6377328Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:23.6425235Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:23.6504876Z ##[endgroup] 2024-08-20T22:13:23.6508658Z ##[group]Persisting credentials for submodules 2024-08-20T22:13:23.6510359Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-08-20T22:13:23.6876291Z Entering 'android/libs/fbjni' 2024-08-20T22:13:23.6944850Z Entering 'third_party/FP16' 2024-08-20T22:13:23.7012288Z Entering 'third_party/FXdiv' 2024-08-20T22:13:23.7080533Z Entering 'third_party/NNPACK' 2024-08-20T22:13:23.7148115Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:23.7215534Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:23.7298842Z Entering 'third_party/benchmark' 2024-08-20T22:13:23.7366538Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:23.7434668Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:23.7503016Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:23.7572043Z Entering 'third_party/cutlass' 2024-08-20T22:13:23.7648525Z Entering 'third_party/eigen' 2024-08-20T22:13:23.7718329Z Entering 'third_party/fbgemm' 2024-08-20T22:13:23.7789410Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:23.7855445Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:23.7921576Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:23.7993825Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:23.8062471Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:23.8130644Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:23.8204385Z Entering 'third_party/fmt' 2024-08-20T22:13:23.8273200Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:23.8340455Z Entering 'third_party/gloo' 2024-08-20T22:13:23.8407828Z Entering 'third_party/googletest' 2024-08-20T22:13:23.8475774Z Entering 'third_party/ideep' 2024-08-20T22:13:23.8541637Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:23.8616823Z Entering 'third_party/ittapi' 2024-08-20T22:13:23.8684404Z Entering 'third_party/kineto' 2024-08-20T22:13:23.8750239Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:23.8815583Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:23.8883995Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:23.8950706Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:23.9018289Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:23.9086165Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:23.9156499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:23.9223160Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:23.9290112Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:23.9357499Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:23.9426977Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:23.9497815Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:23.9566422Z Entering 'third_party/mimalloc' 2024-08-20T22:13:23.9635544Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:23.9704946Z Entering 'third_party/nlohmann' 2024-08-20T22:13:23.9775397Z Entering 'third_party/onnx' 2024-08-20T22:13:23.9856392Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:23.9926679Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:23.9999622Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:24.0072695Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:24.0138711Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:24.0204747Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:24.0272242Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:24.0339475Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:24.0405039Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:24.0470748Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:24.0539088Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:24.0608596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:24.0679230Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:24.0771436Z Entering 'third_party/pocketfft' 2024-08-20T22:13:24.0841215Z Entering 'third_party/protobuf' 2024-08-20T22:13:24.0912298Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:24.0977618Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:24.1047410Z Entering 'third_party/psimd' 2024-08-20T22:13:24.1115750Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:24.1182643Z Entering 'third_party/pybind11' 2024-08-20T22:13:24.1250741Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:24.1317628Z Entering 'third_party/sleef' 2024-08-20T22:13:24.1385346Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:24.1450838Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:24.1516314Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:24.1581999Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:24.1648136Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:24.1711848Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:24.1804515Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-08-20T22:13:24.2170377Z Entering 'android/libs/fbjni' 2024-08-20T22:13:24.2233145Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-08-20T22:13:24.2253760Z Entering 'third_party/FP16' 2024-08-20T22:13:24.2315916Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-08-20T22:13:24.2337676Z Entering 'third_party/FXdiv' 2024-08-20T22:13:24.2400455Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-08-20T22:13:24.2421895Z Entering 'third_party/NNPACK' 2024-08-20T22:13:24.2484773Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-08-20T22:13:24.2505525Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:24.2568013Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-08-20T22:13:24.2589286Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:24.2652090Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-08-20T22:13:24.2689506Z Entering 'third_party/benchmark' 2024-08-20T22:13:24.2751187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:24.2772242Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:24.2833235Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-08-20T22:13:24.2853908Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:24.2914466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:24.2936111Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:24.2996704Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-08-20T22:13:24.3017784Z Entering 'third_party/cutlass' 2024-08-20T22:13:24.3082786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:24.3110422Z Entering 'third_party/eigen' 2024-08-20T22:13:24.3170815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-08-20T22:13:24.3193642Z Entering 'third_party/fbgemm' 2024-08-20T22:13:24.3255892Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-08-20T22:13:24.3275168Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:24.3337340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-08-20T22:13:24.3357941Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:24.3419654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-08-20T22:13:24.3440971Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:24.3502108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-08-20T22:13:24.3531046Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:24.3592716Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.3613366Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:24.3674389Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-08-20T22:13:24.3696686Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:24.3759341Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-08-20T22:13:24.3782531Z Entering 'third_party/fmt' 2024-08-20T22:13:24.3843447Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:24.3864403Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:24.3925119Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-08-20T22:13:24.3946782Z Entering 'third_party/gloo' 2024-08-20T22:13:24.4007731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-08-20T22:13:24.4028662Z Entering 'third_party/googletest' 2024-08-20T22:13:24.4089643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.4110312Z Entering 'third_party/ideep' 2024-08-20T22:13:24.4174540Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-08-20T22:13:24.4193020Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:24.4254223Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-08-20T22:13:24.4283039Z Entering 'third_party/ittapi' 2024-08-20T22:13:24.4343163Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-08-20T22:13:24.4364035Z Entering 'third_party/kineto' 2024-08-20T22:13:24.4425278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-08-20T22:13:24.4445677Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:24.4507668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-08-20T22:13:24.4526570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:24.4589183Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-08-20T22:13:24.4616277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:24.4677763Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-08-20T22:13:24.4698657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:24.4760993Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-08-20T22:13:24.4781468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:24.4843108Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-08-20T22:13:24.4861345Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:24.4924083Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-08-20T22:13:24.4950338Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:24.5012217Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-08-20T22:13:24.5032063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:24.5094505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.5114984Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:24.5176462Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-08-20T22:13:24.5197808Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:24.5269118Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-08-20T22:13:24.5294500Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:24.5356781Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-08-20T22:13:24.5376748Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:24.5438260Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.5460902Z Entering 'third_party/mimalloc' 2024-08-20T22:13:24.5523713Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-08-20T22:13:24.5545074Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:24.5606116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-08-20T22:13:24.5627154Z Entering 'third_party/nlohmann' 2024-08-20T22:13:24.5689443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-08-20T22:13:24.5711565Z Entering 'third_party/onnx' 2024-08-20T22:13:24.5773155Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-08-20T22:13:24.5807573Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:24.5868456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:24.5889590Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:24.5951547Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:24.5977941Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:24.6040421Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-08-20T22:13:24.6061170Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:24.6122015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:24.6143812Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:24.6204682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.6225465Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:24.6286731Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-08-20T22:13:24.6307421Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:24.6373702Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-08-20T22:13:24.6395094Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:24.6456292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-08-20T22:13:24.6475911Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:24.6537394Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-08-20T22:13:24.6556962Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:24.6618226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-08-20T22:13:24.6636676Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:24.6698546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-08-20T22:13:24.6720366Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:24.6783802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-08-20T22:13:24.6807045Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:24.6868116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-08-20T22:13:24.6914339Z Entering 'third_party/pocketfft' 2024-08-20T22:13:24.6976382Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-08-20T22:13:24.6997459Z Entering 'third_party/protobuf' 2024-08-20T22:13:24.7060568Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-08-20T22:13:24.7083398Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:24.7144995Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-08-20T22:13:24.7165671Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:24.7225612Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.7250430Z Entering 'third_party/psimd' 2024-08-20T22:13:24.7313614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-08-20T22:13:24.7335282Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:24.7397654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-08-20T22:13:24.7419226Z Entering 'third_party/pybind11' 2024-08-20T22:13:24.7482914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:24.7505192Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:24.7567534Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-08-20T22:13:24.7588746Z Entering 'third_party/sleef' 2024-08-20T22:13:24.7651767Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-08-20T22:13:24.7673390Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:24.7737618Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-08-20T22:13:24.7757612Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:24.7818812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-08-20T22:13:24.7839203Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:24.7899823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-08-20T22:13:24.7920156Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:24.7986158Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-08-20T22:13:24.8006081Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:24.8067837Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-08-20T22:13:24.8085490Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:24.8150764Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-08-20T22:13:24.9244587Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-08-20T22:13:24.9608150Z Entering 'android/libs/fbjni' 2024-08-20T22:13:24.9658608Z Entering 'third_party/FP16' 2024-08-20T22:13:24.9710022Z Entering 'third_party/FXdiv' 2024-08-20T22:13:24.9762806Z Entering 'third_party/NNPACK' 2024-08-20T22:13:24.9814815Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:24.9867557Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:24.9935050Z Entering 'third_party/benchmark' 2024-08-20T22:13:24.9986329Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:25.0037536Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:25.0088959Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:25.0140129Z Entering 'third_party/cutlass' 2024-08-20T22:13:25.0198491Z Entering 'third_party/eigen' 2024-08-20T22:13:25.0251697Z Entering 'third_party/fbgemm' 2024-08-20T22:13:25.0302267Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:25.0351421Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:25.0402119Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:25.0461871Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:25.0511656Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:25.0568388Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:25.0622773Z Entering 'third_party/fmt' 2024-08-20T22:13:25.0674378Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:25.0724645Z Entering 'third_party/gloo' 2024-08-20T22:13:25.0776289Z Entering 'third_party/googletest' 2024-08-20T22:13:25.0826719Z Entering 'third_party/ideep' 2024-08-20T22:13:25.0875893Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:25.0934266Z Entering 'third_party/ittapi' 2024-08-20T22:13:25.0986772Z Entering 'third_party/kineto' 2024-08-20T22:13:25.1037071Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:25.1086021Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:25.1137761Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:25.1187496Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:25.1237512Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:25.1285748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:25.1340236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:25.1390205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:25.1440089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:25.1490704Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:25.1543507Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:25.1593469Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:25.1646948Z Entering 'third_party/mimalloc' 2024-08-20T22:13:25.1699016Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:25.1751475Z Entering 'third_party/nlohmann' 2024-08-20T22:13:25.1804646Z Entering 'third_party/onnx' 2024-08-20T22:13:25.1869287Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:25.1920520Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:25.1978574Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:25.2030460Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:25.2085733Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:25.2135978Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:25.2184596Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:25.2235574Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:25.2284455Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:25.2336404Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:25.2384014Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:25.2436460Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:25.2492425Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:25.2565418Z Entering 'third_party/pocketfft' 2024-08-20T22:13:25.2617398Z Entering 'third_party/protobuf' 2024-08-20T22:13:25.2672109Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:25.2725560Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:25.2779349Z Entering 'third_party/psimd' 2024-08-20T22:13:25.2830248Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:25.2880279Z Entering 'third_party/pybind11' 2024-08-20T22:13:25.2930533Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:25.2986572Z Entering 'third_party/sleef' 2024-08-20T22:13:25.3043038Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:25.3092680Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:25.3145228Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:25.3195089Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:25.3246916Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:25.3300447Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:25.3379220Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-08-20T22:13:25.3746787Z Entering 'android/libs/fbjni' 2024-08-20T22:13:25.3796973Z Entering 'third_party/FP16' 2024-08-20T22:13:25.3848206Z Entering 'third_party/FXdiv' 2024-08-20T22:13:25.3899023Z Entering 'third_party/NNPACK' 2024-08-20T22:13:25.3951068Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T22:13:25.4003237Z Entering 'third_party/XNNPACK' 2024-08-20T22:13:25.4067925Z Entering 'third_party/benchmark' 2024-08-20T22:13:25.4118966Z Entering 'third_party/cpp-httplib' 2024-08-20T22:13:25.4173795Z Entering 'third_party/cpuinfo' 2024-08-20T22:13:25.4224811Z Entering 'third_party/cudnn_frontend' 2024-08-20T22:13:25.4275603Z Entering 'third_party/cutlass' 2024-08-20T22:13:25.4333269Z Entering 'third_party/eigen' 2024-08-20T22:13:25.4387530Z Entering 'third_party/fbgemm' 2024-08-20T22:13:25.4437091Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T22:13:25.4487829Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T22:13:25.4538079Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T22:13:25.4594492Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T22:13:25.4647274Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T22:13:25.4699509Z Entering 'third_party/flatbuffers' 2024-08-20T22:13:25.4754687Z Entering 'third_party/fmt' 2024-08-20T22:13:25.4806057Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T22:13:25.4859504Z Entering 'third_party/gloo' 2024-08-20T22:13:25.4911060Z Entering 'third_party/googletest' 2024-08-20T22:13:25.4962991Z Entering 'third_party/ideep' 2024-08-20T22:13:25.5012741Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T22:13:25.5071301Z Entering 'third_party/ittapi' 2024-08-20T22:13:25.5123894Z Entering 'third_party/kineto' 2024-08-20T22:13:25.5173836Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T22:13:25.5222911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T22:13:25.5276580Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T22:13:25.5327785Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T22:13:25.5380219Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T22:13:25.5429358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T22:13:25.5489271Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T22:13:25.5541322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T22:13:25.5592748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T22:13:25.5646738Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T22:13:25.5700640Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T22:13:25.5750519Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T22:13:25.5804662Z Entering 'third_party/mimalloc' 2024-08-20T22:13:25.5858609Z Entering 'third_party/nccl/nccl' 2024-08-20T22:13:25.5911487Z Entering 'third_party/nlohmann' 2024-08-20T22:13:25.5965919Z Entering 'third_party/onnx' 2024-08-20T22:13:25.6030889Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T22:13:25.6089170Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T22:13:25.6149189Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T22:13:25.6201651Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T22:13:25.6253474Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T22:13:25.6303907Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T22:13:25.6358793Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T22:13:25.6409698Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T22:13:25.6461135Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T22:13:25.6514551Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T22:13:25.6564411Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T22:13:25.6617242Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T22:13:25.6670887Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T22:13:25.6744252Z Entering 'third_party/pocketfft' 2024-08-20T22:13:25.6796417Z Entering 'third_party/protobuf' 2024-08-20T22:13:25.6853945Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T22:13:25.6905216Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T22:13:25.6959872Z Entering 'third_party/psimd' 2024-08-20T22:13:25.7012364Z Entering 'third_party/pthreadpool' 2024-08-20T22:13:25.7065085Z Entering 'third_party/pybind11' 2024-08-20T22:13:25.7116908Z Entering 'third_party/python-peachpy' 2024-08-20T22:13:25.7169676Z Entering 'third_party/sleef' 2024-08-20T22:13:25.7221395Z Entering 'third_party/tensorpipe' 2024-08-20T22:13:25.7272683Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T22:13:25.7322831Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T22:13:25.7374007Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T22:13:25.7424051Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T22:13:25.7476276Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T22:13:25.7548939Z ##[endgroup] 2024-08-20T22:13:25.7599372Z [command]/usr/bin/git log -1 --format='%H' 2024-08-20T22:13:25.7637290Z '40ec5f6ddd9787aca0449b24128343ff4c4a88b3' 2024-08-20T22:13:25.7850592Z Prepare all required actions 2024-08-20T22:13:25.7851038Z Getting action download info 2024-08-20T22:13:25.9305043Z ##[group]Run ./.github/actions/setup-linux 2024-08-20T22:13:25.9305470Z env: 2024-08-20T22:13:25.9305753Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.9306090Z ##[endgroup] 2024-08-20T22:13:25.9367518Z ##[group]Run set -euo pipefail 2024-08-20T22:13:25.9367959Z set -euo pipefail 2024-08-20T22:13:25.9368338Z function get_ec2_metadata() { 2024-08-20T22:13:25.9368866Z  # Pulled from instance metadata endpoint for EC2 2024-08-20T22:13:25.9369782Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-08-20T22:13:25.9370534Z  category=$1 2024-08-20T22:13:25.9371063Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-08-20T22:13:25.9371698Z  runner_name_str=i-0ed5a1a6923b38c4c 2024-08-20T22:13:25.9372201Z  if [[ -f /.inarc ]]; then 2024-08-20T22:13:25.9372690Z  echo "ARC Runner, no info on ec2 metadata" 2024-08-20T22:13:25.9373249Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-08-20T22:13:25.9373943Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-08-20T22:13:25.9374552Z  else 2024-08-20T22:13:25.9375039Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-08-20T22:13:25.9375610Z  fi 2024-08-20T22:13:25.9375903Z } 2024-08-20T22:13:25.9376263Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-08-20T22:13:25.9376838Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-08-20T22:13:25.9377486Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-08-20T22:13:25.9378064Z echo "system info $(uname -a)" 2024-08-20T22:13:25.9387512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.9387991Z env: 2024-08-20T22:13:25.9388269Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.9388608Z ##[endgroup] 2024-08-20T22:13:25.9485055Z ami-id: ami-06c68f701d8090592 2024-08-20T22:13:25.9541572Z instance-id: i-0ed5a1a6923b38c4c 2024-08-20T22:13:25.9597861Z instance-type: g5.4xlarge 2024-08-20T22:13:25.9612528Z system info Linux ip-10-0-8-182.ec2.internal 6.1.94-99.176.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 18 14:57:56 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-08-20T22:13:25.9635540Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:25.9636436Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-08-20T22:13:25.9645100Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.9645755Z env: 2024-08-20T22:13:25.9646033Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.9646378Z ##[endgroup] 2024-08-20T22:13:25.9739305Z ##[group]Run if systemctl is-active --quiet docker; then 2024-08-20T22:13:25.9739942Z if systemctl is-active --quiet docker; then 2024-08-20T22:13:25.9740467Z  echo "Docker daemon is running..."; 2024-08-20T22:13:25.9740914Z else 2024-08-20T22:13:25.9741393Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-08-20T22:13:25.9741965Z fi 2024-08-20T22:13:25.9750143Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:25.9750626Z env: 2024-08-20T22:13:25.9750901Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.9751247Z ##[endgroup] 2024-08-20T22:13:25.9846528Z Docker daemon is running... 2024-08-20T22:13:25.9914255Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:13:25.9914809Z with: 2024-08-20T22:13:25.9915097Z shell: bash 2024-08-20T22:13:25.9915389Z timeout_minutes: 5 2024-08-20T22:13:25.9915712Z max_attempts: 3 2024-08-20T22:13:25.9916031Z retry_wait_seconds: 30 2024-08-20T22:13:25.9919085Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2024-08-20T22:13:25.9922052Z polling_interval_seconds: 1 2024-08-20T22:13:25.9922421Z warning_on_retry: true 2024-08-20T22:13:25.9922779Z continue_on_error: false 2024-08-20T22:13:25.9923114Z env: 2024-08-20T22:13:25.9923403Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:25.9923759Z AWS_RETRY_MODE: standard 2024-08-20T22:13:25.9924110Z AWS_MAX_ATTEMPTS: 5 2024-08-20T22:13:25.9924464Z AWS_DEFAULT_REGION: us-east-1 2024-08-20T22:13:25.9924840Z ##[endgroup] 2024-08-20T22:13:27.2091573Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:27.2092379Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:27.2093222Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:27.2093726Z 2024-08-20T22:13:27.2093846Z Login Succeeded 2024-08-20T22:13:28.0485689Z Command completed after 1 attempt(s). 2024-08-20T22:13:28.0546463Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:28.0547168Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:28.0547808Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-08-20T22:13:28.0557918Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:28.0558403Z env: 2024-08-20T22:13:28.0558680Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.0559022Z ##[endgroup] 2024-08-20T22:13:28.0670671Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:28.0671796Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T22:13:28.0672675Z # shellcheck disable=SC2046 2024-08-20T22:13:28.0673354Z docker stop $(docker ps -q) || true 2024-08-20T22:13:28.0674054Z # Prune all of the docker images 2024-08-20T22:13:28.0674720Z docker system prune -af 2024-08-20T22:13:28.0686445Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:28.0687172Z env: 2024-08-20T22:13:28.0687554Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.0688058Z ##[endgroup] 2024-08-20T22:13:28.1000657Z "docker stop" requires at least 1 argument. 2024-08-20T22:13:28.1001375Z See 'docker stop --help'. 2024-08-20T22:13:28.1001988Z 2024-08-20T22:13:28.1002210Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-08-20T22:13:28.1002593Z 2024-08-20T22:13:28.1005297Z Stop one or more running containers 2024-08-20T22:13:28.1197143Z Total reclaimed space: 0B 2024-08-20T22:13:28.1243691Z ##[group]Run set +e 2024-08-20T22:13:28.1244053Z set +e 2024-08-20T22:13:28.1244359Z set -x 2024-08-20T22:13:28.1244651Z  2024-08-20T22:13:28.1245112Z PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:28.1245956Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-08-20T22:13:28.1246999Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-08-20T22:13:28.1247726Z # one is returned at random 2024-08-20T22:13:28.1248264Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-08-20T22:13:28.1248762Z  2024-08-20T22:13:28.1249098Z if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:28.1249712Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-08-20T22:13:28.1250734Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-08-20T22:13:28.1251278Z  2024-08-20T22:13:28.1251609Z  if [ -z "${RESOLVED_IP}" ]; then 2024-08-20T22:13:28.1252152Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-08-20T22:13:28.1252654Z  exit 1 2024-08-20T22:13:28.1252962Z  fi 2024-08-20T22:13:28.1253248Z fi 2024-08-20T22:13:28.1253532Z  2024-08-20T22:13:28.1253898Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-08-20T22:13:28.1254402Z  # Clean up any old records first 2024-08-20T22:13:28.1254907Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-08-20T22:13:28.1255359Z fi 2024-08-20T22:13:28.1255630Z  2024-08-20T22:13:28.1256065Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-08-20T22:13:28.1256611Z cat /etc/hosts 2024-08-20T22:13:28.1265443Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:28.1265926Z env: 2024-08-20T22:13:28.1266216Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.1266561Z ##[endgroup] 2024-08-20T22:13:28.1295942Z + PT_DOMAIN=download.pytorch.org 2024-08-20T22:13:28.1302424Z ++ dig -4 +short download.pytorch.org 2024-08-20T22:13:28.1302897Z ++ tail -n1 2024-08-20T22:13:28.1641695Z + RESOLVED_IP=18.160.10.76 2024-08-20T22:13:28.1642156Z + '[' -z 18.160.10.76 ']' 2024-08-20T22:13:28.1642596Z + grep -r download.pytorch.org /etc/hosts 2024-08-20T22:13:28.1655258Z 108.138.64.125 download.pytorch.org 2024-08-20T22:13:28.1657477Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-08-20T22:13:28.3142021Z + echo '18.160.10.76 download.pytorch.org' 2024-08-20T22:13:28.3142568Z + sudo tee -a /etc/hosts 2024-08-20T22:13:28.3479857Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:28.3500713Z + cat /etc/hosts 2024-08-20T22:13:28.3515134Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-08-20T22:13:28.3527796Z ::1 localhost6 localhost6.localdomain6 2024-08-20T22:13:28.3528509Z 18.160.10.76 download.pytorch.org 2024-08-20T22:13:28.3696449Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-08-20T22:13:28.3697052Z with: 2024-08-20T22:13:28.3698131Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3699480Z docker-build-dir: .ci/docker 2024-08-20T22:13:28.3699864Z working-directory: . 2024-08-20T22:13:28.3700339Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.3700860Z force-push: false 2024-08-20T22:13:28.3701163Z env: 2024-08-20T22:13:28.3701441Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.3701779Z ##[endgroup] 2024-08-20T22:13:28.3732961Z ##[group]Run set -ex 2024-08-20T22:13:28.3733338Z set -ex 2024-08-20T22:13:28.3733630Z  2024-08-20T22:13:28.3734183Z # If the docker build directory or the build script doesn't exist, the action will 2024-08-20T22:13:28.3735132Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-08-20T22:13:28.3735911Z # job could then download the pre-built image as usual 2024-08-20T22:13:28.3736637Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-08-20T22:13:28.3737300Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3737928Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3738483Z  2024-08-20T22:13:28.3738975Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-08-20T22:13:28.3739590Z  exit 0 2024-08-20T22:13:28.3739904Z else 2024-08-20T22:13:28.3740273Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3740760Z fi 2024-08-20T22:13:28.3741039Z  2024-08-20T22:13:28.3741506Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-08-20T22:13:28.3742334Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-08-20T22:13:28.3743098Z  # use it as it is, but first let's extract the tag 2024-08-20T22:13:28.3743789Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-08-20T22:13:28.3744506Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3745193Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3745742Z else 2024-08-20T22:13:28.3746171Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-08-20T22:13:28.3746818Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3747697Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3748421Z fi 2024-08-20T22:13:28.3758533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:28.3759026Z env: 2024-08-20T22:13:28.3759303Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.3759655Z REPO_NAME: pytorch 2024-08-20T22:13:28.3760767Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3761958Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:28.3762453Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.3762952Z ##[endgroup] 2024-08-20T22:13:28.3793704Z + [[ ! -d .ci/docker ]] 2024-08-20T22:13:28.3794188Z + [[ ! -f .ci/docker/build.sh ]] 2024-08-20T22:13:28.3794554Z + echo skip=false 2024-08-20T22:13:28.3796543Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-08-20T22:13:28.3802285Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3803562Z ++ awk -F '[:,]' '{print $2}' 2024-08-20T22:13:28.3839560Z + DOCKER_TAG=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3840592Z + echo docker-tag=f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3842317Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3874165Z ##[group]Run set +e 2024-08-20T22:13:28.3874784Z set +e 2024-08-20T22:13:28.3875167Z set -x 2024-08-20T22:13:28.3875591Z  2024-08-20T22:13:28.3875971Z login() { 2024-08-20T22:13:28.3876669Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:28.3877569Z } 2024-08-20T22:13:28.3877897Z  2024-08-20T22:13:28.3878300Z retry () { 2024-08-20T22:13:28.3878788Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:28.3879306Z } 2024-08-20T22:13:28.3879678Z  2024-08-20T22:13:28.3880095Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:28.3880582Z  2024-08-20T22:13:28.3881182Z # Check if image already exists, if it does then skip building it 2024-08-20T22:13:28.3881940Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-08-20T22:13:28.3882515Z  exit 0 2024-08-20T22:13:28.3882966Z fi 2024-08-20T22:13:28.3883298Z  2024-08-20T22:13:28.3883872Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-08-20T22:13:28.3884834Z # be empty. The default action would be to continue rebuild the image 2024-08-20T22:13:28.3885766Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-08-20T22:13:28.3886501Z  # if we're on the base branch then use the parent commit 2024-08-20T22:13:28.3887208Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-08-20T22:13:28.3887802Z else 2024-08-20T22:13:28.3888351Z  # otherwise we're on a PR, so use the most recent base commit 2024-08-20T22:13:28.3889193Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-08-20T22:13:28.3889747Z fi 2024-08-20T22:13:28.3890135Z  2024-08-20T22:13:28.3890592Z if [[ -z "${MERGE_BASE}" ]]; then 2024-08-20T22:13:28.3891173Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3891696Z  2024-08-20T22:13:28.3892472Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-08-20T22:13:28.3893294Z  exit 0 2024-08-20T22:13:28.3893748Z fi 2024-08-20T22:13:28.3894160Z  2024-08-20T22:13:28.3894683Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-08-20T22:13:28.3895780Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-08-20T22:13:28.3896658Z  exit 1 2024-08-20T22:13:28.3897029Z fi 2024-08-20T22:13:28.3897444Z  2024-08-20T22:13:28.3898018Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-08-20T22:13:28.3899061Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-08-20T22:13:28.3900002Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-08-20T22:13:28.3901084Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-08-20T22:13:28.3902257Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-08-20T22:13:28.3903262Z fi 2024-08-20T22:13:28.3903623Z  2024-08-20T22:13:28.3904065Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-08-20T22:13:28.3912580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:28.3913169Z env: 2024-08-20T22:13:28.3913575Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:28.3914053Z DOCKER_BUILD_DIR: .ci/docker 2024-08-20T22:13:28.3914604Z BASE_REVISION: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:13:28.3916014Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3917334Z DOCKER_TAG: f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:28.3918156Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.3918845Z ##[endgroup] 2024-08-20T22:13:28.3948430Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.3949320Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.3951191Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:28.3953311Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:28.9383132Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:28.9384316Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:28.9385537Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:28.9386077Z 2024-08-20T22:13:28.9386228Z Login Succeeded 2024-08-20T22:13:28.9408327Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.1473756Z { 2024-08-20T22:13:29.1474311Z "schemaVersion": 2, 2024-08-20T22:13:29.1475852Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-08-20T22:13:29.1476666Z "config": { 2024-08-20T22:13:29.1477593Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-08-20T22:13:29.1478713Z "size": 48711, 2024-08-20T22:13:29.1479460Z "digest": "sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb" 2024-08-20T22:13:29.1480141Z }, 2024-08-20T22:13:29.1480541Z "layers": [ 2024-08-20T22:13:29.1481012Z { 2024-08-20T22:13:29.1481752Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1483011Z "size": 28580681, 2024-08-20T22:13:29.1484102Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-08-20T22:13:29.1485265Z }, 2024-08-20T22:13:29.1485690Z { 2024-08-20T22:13:29.1486448Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1487653Z "size": 7943451, 2024-08-20T22:13:29.1488351Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-08-20T22:13:29.1489031Z }, 2024-08-20T22:13:29.1489430Z { 2024-08-20T22:13:29.1489949Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1490607Z "size": 55728572, 2024-08-20T22:13:29.1491258Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-08-20T22:13:29.1492010Z }, 2024-08-20T22:13:29.1492365Z { 2024-08-20T22:13:29.1492929Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1493890Z "size": 186, 2024-08-20T22:13:29.1494520Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-08-20T22:13:29.1495279Z }, 2024-08-20T22:13:29.1495701Z { 2024-08-20T22:13:29.1496179Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1496867Z "size": 6886, 2024-08-20T22:13:29.1497522Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-08-20T22:13:29.1498176Z }, 2024-08-20T22:13:29.1498902Z { 2024-08-20T22:13:29.1499437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1500084Z "size": 1329001756, 2024-08-20T22:13:29.1500837Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-08-20T22:13:29.1501551Z }, 2024-08-20T22:13:29.1501898Z { 2024-08-20T22:13:29.1502490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1503081Z "size": 62501, 2024-08-20T22:13:29.1503738Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-08-20T22:13:29.1504707Z }, 2024-08-20T22:13:29.1505014Z { 2024-08-20T22:13:29.1505666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1506502Z "size": 1684, 2024-08-20T22:13:29.1507145Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-08-20T22:13:29.1507855Z }, 2024-08-20T22:13:29.1508201Z { 2024-08-20T22:13:29.1509011Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1509719Z "size": 1523, 2024-08-20T22:13:29.1510324Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-08-20T22:13:29.1511056Z }, 2024-08-20T22:13:29.1511442Z { 2024-08-20T22:13:29.1511956Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1512574Z "size": 2528295403, 2024-08-20T22:13:29.1513308Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-08-20T22:13:29.1514024Z }, 2024-08-20T22:13:29.1514325Z { 2024-08-20T22:13:29.1514901Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1515573Z "size": 86016, 2024-08-20T22:13:29.1516271Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-08-20T22:13:29.1516933Z }, 2024-08-20T22:13:29.1517305Z { 2024-08-20T22:13:29.1517883Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1518521Z "size": 1822, 2024-08-20T22:13:29.1519137Z "digest": "sha256:ac23c5fcc1ef310e9cd0d138b180281188e20a3e28c51cd14751622e3c7d92ea" 2024-08-20T22:13:29.1519932Z }, 2024-08-20T22:13:29.1520295Z { 2024-08-20T22:13:29.1520776Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1521473Z "size": 246761563, 2024-08-20T22:13:29.1522149Z "digest": "sha256:ca426ffc9d2bd051ad1ff5c4f02fd5d9be161e70953cc37882c815080c708f4d" 2024-08-20T22:13:29.1522864Z }, 2024-08-20T22:13:29.1523220Z { 2024-08-20T22:13:29.1523765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1524399Z "size": 544, 2024-08-20T22:13:29.1525150Z "digest": "sha256:4373aa743caf6660c671295b837048c8350a2165f2336dc3b7b6c96462579e50" 2024-08-20T22:13:29.1525811Z }, 2024-08-20T22:13:29.1526154Z { 2024-08-20T22:13:29.1526769Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1527401Z "size": 1258, 2024-08-20T22:13:29.1527983Z "digest": "sha256:75b2abb16c685854c017d0cf75c626526c956734c656d9e78fdd4520317e9b3c" 2024-08-20T22:13:29.1528744Z }, 2024-08-20T22:13:29.1529092Z { 2024-08-20T22:13:29.1529576Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1530279Z "size": 484, 2024-08-20T22:13:29.1530898Z "digest": "sha256:ef1e3ef1e85f0124270b8506721219150e230184181f8207d64463bf5c67e656" 2024-08-20T22:13:29.1531579Z }, 2024-08-20T22:13:29.1531955Z { 2024-08-20T22:13:29.1532476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1533454Z "size": 91711766, 2024-08-20T22:13:29.1534198Z "digest": "sha256:0bf61114d37b2d09065aa6852b1e78a32580a1ff011dd5d23f03b0f5c4e710af" 2024-08-20T22:13:29.1534925Z }, 2024-08-20T22:13:29.1535300Z { 2024-08-20T22:13:29.1535894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1536577Z "size": 3390, 2024-08-20T22:13:29.1537332Z "digest": "sha256:2df849f72ba4258df10c9c45ace21a66d93b73196e78e96f12c13098822f60e2" 2024-08-20T22:13:29.1538081Z }, 2024-08-20T22:13:29.1538446Z { 2024-08-20T22:13:29.1538923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1539604Z "size": 1909, 2024-08-20T22:13:29.1540230Z "digest": "sha256:34c47d63c5de27418739aa048b4dd781c3006d65691e9b903c95304d0509d89c" 2024-08-20T22:13:29.1541007Z }, 2024-08-20T22:13:29.1541306Z { 2024-08-20T22:13:29.1541845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1542529Z "size": 701, 2024-08-20T22:13:29.1543181Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:29.1543984Z }, 2024-08-20T22:13:29.1544389Z { 2024-08-20T22:13:29.1544926Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1545522Z "size": 2864254879, 2024-08-20T22:13:29.1546250Z "digest": "sha256:6efb96fc87ea1bca54876d85331e5585e28f325a1d60ee1aef7811eabb74f933" 2024-08-20T22:13:29.1546974Z }, 2024-08-20T22:13:29.1547339Z { 2024-08-20T22:13:29.1547869Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1548522Z "size": 380, 2024-08-20T22:13:29.1549139Z "digest": "sha256:b119c4fc6c58d9e72744201ff4850b34184a94eb3566031b7d36e87ecce950a6" 2024-08-20T22:13:29.1549890Z }, 2024-08-20T22:13:29.1550193Z { 2024-08-20T22:13:29.1550707Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1551406Z "size": 12900, 2024-08-20T22:13:29.1552042Z "digest": "sha256:33b639226eeefa41d7bb26a8ecc20d271ceef45571dabfd80aaee3634926381b" 2024-08-20T22:13:29.1552708Z }, 2024-08-20T22:13:29.1553118Z { 2024-08-20T22:13:29.1553854Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1554452Z "size": 803, 2024-08-20T22:13:29.1555144Z "digest": "sha256:d3547bd43146930aa830d54562c98aa678e355cf5f84527e9717eb72fffe8ca3" 2024-08-20T22:13:29.1555825Z }, 2024-08-20T22:13:29.1556184Z { 2024-08-20T22:13:29.1556712Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1557343Z "size": 106, 2024-08-20T22:13:29.1557981Z "digest": "sha256:5645520256fa6d7f2270608eaed53849a131d03fbf4c5d4472c9cfde1a439e47" 2024-08-20T22:13:29.1558720Z }, 2024-08-20T22:13:29.1559020Z { 2024-08-20T22:13:29.1559556Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1560246Z "size": 503, 2024-08-20T22:13:29.1560854Z "digest": "sha256:829ed7d91e3818d0e11834c71551117c7aeec0a51d798e93d641faa0c02582cb" 2024-08-20T22:13:29.1561574Z }, 2024-08-20T22:13:29.1561997Z { 2024-08-20T22:13:29.1562542Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1563137Z "size": 121477267, 2024-08-20T22:13:29.1563825Z "digest": "sha256:385d0421d67c533d628985c453a186028a61f6ad4209ae519ec3d4ecea661d3e" 2024-08-20T22:13:29.1564529Z }, 2024-08-20T22:13:29.1564918Z { 2024-08-20T22:13:29.1565459Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1566103Z "size": 109, 2024-08-20T22:13:29.1566770Z "digest": "sha256:7cbfce999d76851e07dd71056a6d852629d789ccfd46be44cc405b451fb8bb58" 2024-08-20T22:13:29.1567501Z }, 2024-08-20T22:13:29.1567803Z { 2024-08-20T22:13:29.1568359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1569009Z "size": 490, 2024-08-20T22:13:29.1569579Z "digest": "sha256:6658e558ebe496f0d628947534b0ea1e44808f84f2dfd8a9a704e8aa5329d791" 2024-08-20T22:13:29.1570312Z }, 2024-08-20T22:13:29.1570675Z { 2024-08-20T22:13:29.1571197Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1571837Z "size": 297, 2024-08-20T22:13:29.1572471Z "digest": "sha256:33444a5a7e01a3010fd1cc16b32a66892a08600d7f700747d23faf3c36624f3a" 2024-08-20T22:13:29.1573159Z }, 2024-08-20T22:13:29.1573713Z { 2024-08-20T22:13:29.1574204Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1574848Z "size": 103, 2024-08-20T22:13:29.1575562Z "digest": "sha256:7c47122f4e09b47e891b4ae9e499a35a4eb46faacae87d0e946615adda6e0706" 2024-08-20T22:13:29.1576215Z }, 2024-08-20T22:13:29.1576562Z { 2024-08-20T22:13:29.1577164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1577812Z "size": 1473, 2024-08-20T22:13:29.1578417Z "digest": "sha256:1811ff4d3556ac255604f346d879528aff2aadd7652c2d39baf4fe6eaa60aec3" 2024-08-20T22:13:29.1579185Z }, 2024-08-20T22:13:29.1579534Z { 2024-08-20T22:13:29.1580071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1580855Z "size": 594729516, 2024-08-20T22:13:29.1581517Z "digest": "sha256:7754ee068b33f760d9c320c26a1a3935247ace0c8c463a976105c1586a43fb1c" 2024-08-20T22:13:29.1582203Z }, 2024-08-20T22:13:29.1582598Z { 2024-08-20T22:13:29.1583112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1583738Z "size": 164, 2024-08-20T22:13:29.1584397Z "digest": "sha256:0479fbac3361713c075bf788e778117917dcacdd1ff71ed79a789dd2016a1714" 2024-08-20T22:13:29.1585067Z }, 2024-08-20T22:13:29.1585409Z { 2024-08-20T22:13:29.1585989Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1586650Z "size": 564, 2024-08-20T22:13:29.1587250Z "digest": "sha256:f1d6874dee9c34d154278d2b6935d04447f07fa89d25d4a09398234040d8a7e8" 2024-08-20T22:13:29.1588009Z }, 2024-08-20T22:13:29.1588350Z { 2024-08-20T22:13:29.1588925Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1589544Z "size": 35874203, 2024-08-20T22:13:29.1590189Z "digest": "sha256:afc429e87b593a64f3e0a1be51e09e966ea1aa71e6e8a0bec4edc9f2f7bc1c05" 2024-08-20T22:13:29.1590944Z }, 2024-08-20T22:13:29.1591313Z { 2024-08-20T22:13:29.1591853Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1592526Z "size": 104, 2024-08-20T22:13:29.1593160Z "digest": "sha256:019620471c2592cd22437fe8ed1a94c784114cf7445c5c2ab36dd0bae727fc66" 2024-08-20T22:13:29.1593806Z }, 2024-08-20T22:13:29.1594241Z { 2024-08-20T22:13:29.1594754Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1595388Z "size": 425, 2024-08-20T22:13:29.1596071Z "digest": "sha256:27aeb1b2cd5e28d8f0e54fc1090baf88abc29ebf5fce088b493fcae99c9550d1" 2024-08-20T22:13:29.1596777Z }, 2024-08-20T22:13:29.1597112Z { 2024-08-20T22:13:29.1597719Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1598317Z "size": 20262183, 2024-08-20T22:13:29.1598960Z "digest": "sha256:829c0e19f4b7797e9120a86172fabfd710f9b3971ffd714a616931a3fadc7192" 2024-08-20T22:13:29.1599709Z }, 2024-08-20T22:13:29.1600012Z { 2024-08-20T22:13:29.1600572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1601293Z "size": 438, 2024-08-20T22:13:29.1601937Z "digest": "sha256:560cd6122a3868e7e82e75eb103d182edf6512114d61ccdfae099af00cb16b4c" 2024-08-20T22:13:29.1602613Z }, 2024-08-20T22:13:29.1603003Z { 2024-08-20T22:13:29.1603521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1604180Z "size": 701, 2024-08-20T22:13:29.1604807Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:29.1605561Z }, 2024-08-20T22:13:29.1605920Z { 2024-08-20T22:13:29.1606446Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1607120Z "size": 141, 2024-08-20T22:13:29.1607778Z "digest": "sha256:5a4eb19d72198a01f3babd4711c40fcf7c6e5cb9d5cfedc507b0061408d47097" 2024-08-20T22:13:29.1608548Z }, 2024-08-20T22:13:29.1608876Z { 2024-08-20T22:13:29.1609392Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1610079Z "size": 135, 2024-08-20T22:13:29.1610838Z "digest": "sha256:da6a838a66c72b5de9a5590c233779880ea901c1264940a1c618825d2501ffd1" 2024-08-20T22:13:29.1611493Z }, 2024-08-20T22:13:29.1611907Z { 2024-08-20T22:13:29.1612426Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1613217Z "size": 4838068375, 2024-08-20T22:13:29.1613827Z "digest": "sha256:cbb79a7053537bed2d80b77198ea552b985563b11d3f8616593c6ab575a3a090" 2024-08-20T22:13:29.1614543Z }, 2024-08-20T22:13:29.1614962Z { 2024-08-20T22:13:29.1615438Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1616063Z "size": 194, 2024-08-20T22:13:29.1616765Z "digest": "sha256:4d4fa6802ab3e6b36d8d4adb650046e9370c4a16a19876970559ef8a312ca855" 2024-08-20T22:13:29.1617537Z }, 2024-08-20T22:13:29.1617837Z { 2024-08-20T22:13:29.1618431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1619060Z "size": 564, 2024-08-20T22:13:29.1619705Z "digest": "sha256:0f085d4fa279366fcd52d303b7ad8abddc5962a8872174b8109f3f5d5ca5f2cf" 2024-08-20T22:13:29.1620435Z }, 2024-08-20T22:13:29.1620807Z { 2024-08-20T22:13:29.1621351Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1622002Z "size": 43162921, 2024-08-20T22:13:29.1622639Z "digest": "sha256:4a0412373305517489d891a4749e6d2b80b4aeb7df3b8adca0ed0abb2f3e5300" 2024-08-20T22:13:29.1623351Z }, 2024-08-20T22:13:29.1623740Z { 2024-08-20T22:13:29.1624223Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1624872Z "size": 106, 2024-08-20T22:13:29.1625528Z "digest": "sha256:302c8c1779444e54919f83e91c6d5e1fdea653fc899be802b3a2a34d34d99d5d" 2024-08-20T22:13:29.1626240Z }, 2024-08-20T22:13:29.1626538Z { 2024-08-20T22:13:29.1627122Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1627775Z "size": 1380, 2024-08-20T22:13:29.1628511Z "digest": "sha256:02970347dff622ee45629bc28788716d00528464f71f88ec7b74ae5f44a486e2" 2024-08-20T22:13:29.1629185Z }, 2024-08-20T22:13:29.1629524Z { 2024-08-20T22:13:29.1630092Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1630709Z "size": 701, 2024-08-20T22:13:29.1631357Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:29.1632126Z }, 2024-08-20T22:13:29.1632489Z { 2024-08-20T22:13:29.1633200Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1633885Z "size": 139, 2024-08-20T22:13:29.1634555Z "digest": "sha256:916c5e8580164926a3aec87e88a5d1c2d02756e49fdeed4198e1e6304efa2fd3" 2024-08-20T22:13:29.1635240Z }, 2024-08-20T22:13:29.1635617Z { 2024-08-20T22:13:29.1636146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1636781Z "size": 118, 2024-08-20T22:13:29.1637467Z "digest": "sha256:c42b3602a1fb4df5111224650f15adffb025979ae9c614807379ea84e189533b" 2024-08-20T22:13:29.1638115Z }, 2024-08-20T22:13:29.1638463Z { 2024-08-20T22:13:29.1639059Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1639656Z "size": 4324672728, 2024-08-20T22:13:29.1640317Z "digest": "sha256:11fad972ef6cb9d2dc236504b0ee5b6e5ea996e4576ece0ed3262fbb9ee2de60" 2024-08-20T22:13:29.1641117Z }, 2024-08-20T22:13:29.1641452Z { 2024-08-20T22:13:29.1641948Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1642620Z "size": 174, 2024-08-20T22:13:29.1643217Z "digest": "sha256:f957c16244819132b478272aaf24231699c4fcbd495c229a574f94daab8d6b86" 2024-08-20T22:13:29.1643908Z }, 2024-08-20T22:13:29.1644256Z { 2024-08-20T22:13:29.1644779Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1645504Z "size": 907, 2024-08-20T22:13:29.1646141Z "digest": "sha256:768c31dca016b80bbdc973c8ee818034bdbd17b7ea9bdadbf0ec45ea51005720" 2024-08-20T22:13:29.1646842Z }, 2024-08-20T22:13:29.1647367Z { 2024-08-20T22:13:29.1647941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1648552Z "size": 701, 2024-08-20T22:13:29.1649176Z "digest": "sha256:1fef4bd19792cbf9d3aea12795089ad41164bb6d607c339b7484e1899825b1d2" 2024-08-20T22:13:29.1649916Z }, 2024-08-20T22:13:29.1650270Z { 2024-08-20T22:13:29.1650747Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1651464Z "size": 134, 2024-08-20T22:13:29.1652089Z "digest": "sha256:8c32d0ecebbe2054c96a3a5c1f3d6c8b87c2e826686b369e5242cd55d0f01c2d" 2024-08-20T22:13:29.1652827Z }, 2024-08-20T22:13:29.1653152Z { 2024-08-20T22:13:29.1653672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1654527Z "size": 32, 2024-08-20T22:13:29.1655113Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:29.1655808Z }, 2024-08-20T22:13:29.1656220Z { 2024-08-20T22:13:29.1656733Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1657325Z "size": 158, 2024-08-20T22:13:29.1658018Z "digest": "sha256:78ee77eddef23f20afc78e2924f4467041cadb754634ee3df431c68211cf676d" 2024-08-20T22:13:29.1658710Z }, 2024-08-20T22:13:29.1659038Z { 2024-08-20T22:13:29.1659582Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1660210Z "size": 1841, 2024-08-20T22:13:29.1660931Z "digest": "sha256:37513e9b43d7123e7ddee09100f62a72b57af7b3833eaecd515e247e2f63e13f" 2024-08-20T22:13:29.1661671Z }, 2024-08-20T22:13:29.1661962Z { 2024-08-20T22:13:29.1662499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1663187Z "size": 7529776, 2024-08-20T22:13:29.1663776Z "digest": "sha256:a7a5392e2bf6b73d9be292cc22a99ad4637636c15733beb7afbe6403b5ba21a8" 2024-08-20T22:13:29.1664490Z }, 2024-08-20T22:13:29.1664871Z { 2024-08-20T22:13:29.1665392Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1666004Z "size": 163, 2024-08-20T22:13:29.1666661Z "digest": "sha256:b10de02255c021b2366b85bf12ddf593aac89f88964201c455b71a145bc5ca2e" 2024-08-20T22:13:29.1667396Z }, 2024-08-20T22:13:29.1667733Z { 2024-08-20T22:13:29.1668254Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1668902Z "size": 7943, 2024-08-20T22:13:29.1669542Z "digest": "sha256:e56eeff2db93ffe5002a2bb764795b7333321c6802cc28bef0801a619fe6a07e" 2024-08-20T22:13:29.1670244Z }, 2024-08-20T22:13:29.1670598Z { 2024-08-20T22:13:29.1671121Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1671823Z "size": 8068, 2024-08-20T22:13:29.1672430Z "digest": "sha256:70ac3e38d0fd1f0f46ad710c3fc485fd01406606054e8cf84a63af1570bc9f3a" 2024-08-20T22:13:29.1673122Z }, 2024-08-20T22:13:29.1673531Z { 2024-08-20T22:13:29.1674071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1674669Z "size": 302, 2024-08-20T22:13:29.1675369Z "digest": "sha256:eb09cbf3e4168749cc8605e3188ed35070f6037d13930e443c38d8941734488f" 2024-08-20T22:13:29.1676046Z }, 2024-08-20T22:13:29.1676347Z { 2024-08-20T22:13:29.1676938Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1677568Z "size": 7629638, 2024-08-20T22:13:29.1678236Z "digest": "sha256:ee1273153675e6543b58f5be50321abee5e91f7351701d5d501a6837eaf676ef" 2024-08-20T22:13:29.1678905Z }, 2024-08-20T22:13:29.1679246Z { 2024-08-20T22:13:29.1679839Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1680503Z "size": 108, 2024-08-20T22:13:29.1681092Z "digest": "sha256:e22979f9e0eedfa16f98eb4fa26f30a50c237f5ffd2f6e5b898f2b908058496d" 2024-08-20T22:13:29.1681856Z }, 2024-08-20T22:13:29.1682198Z { 2024-08-20T22:13:29.1682708Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1683373Z "size": 54145661, 2024-08-20T22:13:29.1684126Z "digest": "sha256:6c1f0aabce7c84f9b0b9ab3b96325778dba1c1465b7463c5b4c6be1cf52b74db" 2024-08-20T22:13:29.1684864Z }, 2024-08-20T22:13:29.1685273Z { 2024-08-20T22:13:29.1685800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1686445Z "size": 473, 2024-08-20T22:13:29.1687118Z "digest": "sha256:7ebe49c3a23e71b289ce012b48dad35791a2f2aa7fcee2326a64d763b0d14df8" 2024-08-20T22:13:29.1687819Z }, 2024-08-20T22:13:29.1688188Z { 2024-08-20T22:13:29.1688766Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1689436Z "size": 1374858766, 2024-08-20T22:13:29.1690035Z "digest": "sha256:5d115b29b5d53e5418762bc41ae975dd245bcffc3d058b8902c58146d505cb37" 2024-08-20T22:13:29.1690855Z }, 2024-08-20T22:13:29.1691218Z { 2024-08-20T22:13:29.1691697Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1692383Z "size": 106, 2024-08-20T22:13:29.1693061Z "digest": "sha256:7f2d7a04fbe65bbe1e895138e0cfeeef53ad926c890bab8e7e132ad1e655e48f" 2024-08-20T22:13:29.1693771Z }, 2024-08-20T22:13:29.1694153Z { 2024-08-20T22:13:29.1694703Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1695344Z "size": 559, 2024-08-20T22:13:29.1696035Z "digest": "sha256:d79fd7d526989e554455eda7d459f2c88444030f61710101e52ec8c7a49b7b79" 2024-08-20T22:13:29.1696690Z }, 2024-08-20T22:13:29.1697040Z { 2024-08-20T22:13:29.1697666Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1698298Z "size": 46248488, 2024-08-20T22:13:29.1698897Z "digest": "sha256:9752d6fbbcab4e7403f26777c5b65911320de6389c81b833a47ba45c4f3354b4" 2024-08-20T22:13:29.1699664Z }, 2024-08-20T22:13:29.1700018Z { 2024-08-20T22:13:29.1700503Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1701248Z "size": 111, 2024-08-20T22:13:29.1701887Z "digest": "sha256:83808b2eff4d71ed46db7c29f66fe918f328405c25b314225dfad01b4868d1f4" 2024-08-20T22:13:29.1702646Z }, 2024-08-20T22:13:29.1702946Z { 2024-08-20T22:13:29.1703474Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1704186Z "size": 32, 2024-08-20T22:13:29.1704820Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:29.1705481Z }, 2024-08-20T22:13:29.1705892Z { 2024-08-20T22:13:29.1706411Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1707000Z "size": 32, 2024-08-20T22:13:29.1707717Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:29.1708426Z }, 2024-08-20T22:13:29.1708794Z { 2024-08-20T22:13:29.1709333Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1709962Z "size": 32, 2024-08-20T22:13:29.1710605Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:29.1711351Z }, 2024-08-20T22:13:29.1711650Z { 2024-08-20T22:13:29.1712185Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-08-20T22:13:29.1712865Z "size": 32, 2024-08-20T22:13:29.1713506Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-08-20T22:13:29.1714168Z } 2024-08-20T22:13:29.1714611Z ] 2024-08-20T22:13:29.1714964Z } 2024-08-20T22:13:29.1715285Z + exit 0 2024-08-20T22:13:29.2021155Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:29.2021648Z tag=${ECR_DOCKER_IMAGE##*/} 2024-08-20T22:13:29.2022185Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-08-20T22:13:29.2031627Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:29.2032111Z env: 2024-08-20T22:13:29.2032382Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:29.2033859Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.2035003Z ##[endgroup] 2024-08-20T22:13:29.2067629Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.2118283Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-08-20T22:13:29.2118845Z with: 2024-08-20T22:13:29.2119891Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.2121171Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:29.2121687Z env: 2024-08-20T22:13:29.2122141Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:29.2122495Z ##[endgroup] 2024-08-20T22:13:29.2139888Z ##[group]Run set -x 2024-08-20T22:13:29.2140229Z set -x 2024-08-20T22:13:29.2140531Z set +e 2024-08-20T22:13:29.2140822Z  2024-08-20T22:13:29.2141099Z login() { 2024-08-20T22:13:29.2141749Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-08-20T22:13:29.2142454Z } 2024-08-20T22:13:29.2142731Z  2024-08-20T22:13:29.2143031Z retry () { 2024-08-20T22:13:29.2143429Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-08-20T22:13:29.2143863Z } 2024-08-20T22:13:29.2144146Z  2024-08-20T22:13:29.2144470Z retry login "${DOCKER_REGISTRY}" 2024-08-20T22:13:29.2144878Z  2024-08-20T22:13:29.2145156Z set -e 2024-08-20T22:13:29.2145632Z # ignore output since only exit code is used for conditional 2024-08-20T22:13:29.2146337Z # only pull docker image if it's not available locally 2024-08-20T22:13:29.2147087Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-08-20T22:13:29.2147780Z  retry docker pull "${DOCKER_IMAGE}" 2024-08-20T22:13:29.2148227Z fi 2024-08-20T22:13:29.2156522Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:13:29.2157012Z env: 2024-08-20T22:13:29.2157298Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:13:29.2158463Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.2159749Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:29.2160269Z ##[endgroup] 2024-08-20T22:13:29.2187311Z + set +e 2024-08-20T22:13:29.2187838Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:29.2188469Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:29.2191407Z + aws ecr get-login-password --region us-east-1 2024-08-20T22:13:29.2192723Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-08-20T22:13:29.7635586Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-08-20T22:13:29.7637668Z Configure a credential helper to remove this warning. See 2024-08-20T22:13:29.7639714Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-08-20T22:13:29.7640769Z 2024-08-20T22:13:29.7641021Z Login Succeeded 2024-08-20T22:13:29.7662199Z + set -e 2024-08-20T22:13:29.7664104Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.7826752Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.7829224Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:13:29.9930123Z f6d216893d65c7b8ae43df4daaf247db808378e9: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-08-20T22:13:29.9937242Z 7a2c55901189: Pulling fs layer 2024-08-20T22:13:29.9937888Z 224fe954d725: Pulling fs layer 2024-08-20T22:13:29.9938720Z 75722010b82e: Pulling fs layer 2024-08-20T22:13:29.9939222Z d527cbbb87e3: Pulling fs layer 2024-08-20T22:13:29.9939717Z b57676e46aee: Pulling fs layer 2024-08-20T22:13:29.9940246Z a8c1e85b5e14: Pulling fs layer 2024-08-20T22:13:29.9940753Z a41a8d1c11c8: Pulling fs layer 2024-08-20T22:13:29.9941189Z 0c1227890755: Pulling fs layer 2024-08-20T22:13:29.9941566Z d8d1234baab3: Pulling fs layer 2024-08-20T22:13:29.9941950Z 7ed32bc8e469: Pulling fs layer 2024-08-20T22:13:29.9942444Z ec1e7978c1fe: Pulling fs layer 2024-08-20T22:13:29.9943197Z ac23c5fcc1ef: Pulling fs layer 2024-08-20T22:13:29.9943692Z ca426ffc9d2b: Pulling fs layer 2024-08-20T22:13:29.9944192Z 4373aa743caf: Pulling fs layer 2024-08-20T22:13:29.9944655Z a8c1e85b5e14: Waiting 2024-08-20T22:13:29.9945076Z b57676e46aee: Waiting 2024-08-20T22:13:29.9945515Z 75b2abb16c68: Pulling fs layer 2024-08-20T22:13:29.9946028Z ef1e3ef1e85f: Pulling fs layer 2024-08-20T22:13:29.9946507Z 0bf61114d37b: Pulling fs layer 2024-08-20T22:13:29.9947000Z 2df849f72ba4: Pulling fs layer 2024-08-20T22:13:29.9947460Z a41a8d1c11c8: Waiting 2024-08-20T22:13:29.9947887Z 34c47d63c5de: Pulling fs layer 2024-08-20T22:13:29.9948361Z 1fef4bd19792: Pulling fs layer 2024-08-20T22:13:29.9948848Z 6efb96fc87ea: Pulling fs layer 2024-08-20T22:13:29.9949305Z 0c1227890755: Waiting 2024-08-20T22:13:29.9949716Z d8d1234baab3: Waiting 2024-08-20T22:13:29.9950137Z d527cbbb87e3: Waiting 2024-08-20T22:13:29.9950562Z b119c4fc6c58: Pulling fs layer 2024-08-20T22:13:29.9951050Z 33b639226eee: Pulling fs layer 2024-08-20T22:13:29.9951517Z 7ed32bc8e469: Waiting 2024-08-20T22:13:29.9951932Z d3547bd43146: Pulling fs layer 2024-08-20T22:13:29.9952410Z 5645520256fa: Pulling fs layer 2024-08-20T22:13:29.9952875Z 2df849f72ba4: Waiting 2024-08-20T22:13:29.9953291Z 829ed7d91e38: Pulling fs layer 2024-08-20T22:13:29.9953766Z 385d0421d67c: Pulling fs layer 2024-08-20T22:13:29.9954209Z ec1e7978c1fe: Waiting 2024-08-20T22:13:29.9954541Z 7cbfce999d76: Pulling fs layer 2024-08-20T22:13:29.9954905Z 34c47d63c5de: Waiting 2024-08-20T22:13:29.9955237Z ac23c5fcc1ef: Waiting 2024-08-20T22:13:29.9955570Z 6658e558ebe4: Pulling fs layer 2024-08-20T22:13:29.9955924Z 1fef4bd19792: Waiting 2024-08-20T22:13:29.9956256Z 33444a5a7e01: Pulling fs layer 2024-08-20T22:13:29.9956613Z b119c4fc6c58: Waiting 2024-08-20T22:13:29.9956934Z ca426ffc9d2b: Waiting 2024-08-20T22:13:29.9957250Z 75b2abb16c68: Waiting 2024-08-20T22:13:29.9957583Z 7c47122f4e09: Pulling fs layer 2024-08-20T22:13:29.9957946Z d3547bd43146: Waiting 2024-08-20T22:13:29.9958254Z 33b639226eee: Waiting 2024-08-20T22:13:29.9958581Z 1811ff4d3556: Pulling fs layer 2024-08-20T22:13:29.9958953Z 7754ee068b33: Pulling fs layer 2024-08-20T22:13:29.9959307Z 5645520256fa: Waiting 2024-08-20T22:13:29.9959611Z 385d0421d67c: Waiting 2024-08-20T22:13:29.9959943Z 0479fbac3361: Pulling fs layer 2024-08-20T22:13:29.9960320Z f1d6874dee9c: Pulling fs layer 2024-08-20T22:13:29.9960692Z 6efb96fc87ea: Waiting 2024-08-20T22:13:29.9961043Z 6658e558ebe4: Waiting 2024-08-20T22:13:29.9961412Z afc429e87b59: Pulling fs layer 2024-08-20T22:13:29.9961772Z 1811ff4d3556: Waiting 2024-08-20T22:13:29.9962087Z 7754ee068b33: Waiting 2024-08-20T22:13:29.9962407Z 019620471c25: Pulling fs layer 2024-08-20T22:13:29.9962783Z 27aeb1b2cd5e: Pulling fs layer 2024-08-20T22:13:29.9963158Z 829c0e19f4b7: Pulling fs layer 2024-08-20T22:13:29.9963527Z 560cd6122a38: Pulling fs layer 2024-08-20T22:13:29.9963894Z 5a4eb19d7219: Pulling fs layer 2024-08-20T22:13:29.9964265Z da6a838a66c7: Pulling fs layer 2024-08-20T22:13:29.9964639Z cbb79a705353: Pulling fs layer 2024-08-20T22:13:29.9965102Z 4d4fa6802ab3: Pulling fs layer 2024-08-20T22:13:29.9965476Z 0f085d4fa279: Pulling fs layer 2024-08-20T22:13:29.9965844Z 4a0412373305: Pulling fs layer 2024-08-20T22:13:29.9966209Z 302c8c177944: Pulling fs layer 2024-08-20T22:13:29.9966554Z 0bf61114d37b: Waiting 2024-08-20T22:13:29.9966886Z 02970347dff6: Pulling fs layer 2024-08-20T22:13:29.9967240Z 33444a5a7e01: Waiting 2024-08-20T22:13:29.9967566Z 916c5e858016: Pulling fs layer 2024-08-20T22:13:29.9968081Z 7cbfce999d76: Waiting 2024-08-20T22:13:29.9968418Z c42b3602a1fb: Pulling fs layer 2024-08-20T22:13:29.9968785Z afc429e87b59: Waiting 2024-08-20T22:13:29.9969116Z 11fad972ef6c: Pulling fs layer 2024-08-20T22:13:29.9969474Z cbb79a705353: Waiting 2024-08-20T22:13:29.9969787Z 02970347dff6: Waiting 2024-08-20T22:13:29.9970112Z f957c1624481: Pulling fs layer 2024-08-20T22:13:29.9970477Z 768c31dca016: Pulling fs layer 2024-08-20T22:13:29.9970851Z 8c32d0ecebbe: Pulling fs layer 2024-08-20T22:13:29.9971330Z 4373aa743caf: Waiting 2024-08-20T22:13:29.9971660Z 4f4fb700ef54: Pulling fs layer 2024-08-20T22:13:29.9972009Z 4d4fa6802ab3: Waiting 2024-08-20T22:13:29.9972346Z 78ee77eddef2: Pulling fs layer 2024-08-20T22:13:29.9972709Z 916c5e858016: Waiting 2024-08-20T22:13:29.9973014Z 302c8c177944: Waiting 2024-08-20T22:13:29.9973339Z 37513e9b43d7: Pulling fs layer 2024-08-20T22:13:29.9973690Z 4a0412373305: Waiting 2024-08-20T22:13:29.9974017Z a7a5392e2bf6: Pulling fs layer 2024-08-20T22:13:29.9974376Z 0f085d4fa279: Waiting 2024-08-20T22:13:29.9974707Z b10de02255c0: Pulling fs layer 2024-08-20T22:13:29.9975070Z 11fad972ef6c: Waiting 2024-08-20T22:13:29.9975400Z e56eeff2db93: Pulling fs layer 2024-08-20T22:13:29.9975758Z 829c0e19f4b7: Waiting 2024-08-20T22:13:29.9976103Z 70ac3e38d0fd: Pulling fs layer 2024-08-20T22:13:29.9976482Z eb09cbf3e416: Pulling fs layer 2024-08-20T22:13:29.9976835Z 8c32d0ecebbe: Waiting 2024-08-20T22:13:29.9977151Z 4f4fb700ef54: Waiting 2024-08-20T22:13:29.9977468Z f957c1624481: Waiting 2024-08-20T22:13:29.9977799Z ee1273153675: Pulling fs layer 2024-08-20T22:13:29.9978147Z 768c31dca016: Waiting 2024-08-20T22:13:29.9978481Z e22979f9e0ee: Pulling fs layer 2024-08-20T22:13:29.9978834Z 019620471c25: Waiting 2024-08-20T22:13:29.9979164Z 6c1f0aabce7c: Pulling fs layer 2024-08-20T22:13:29.9979541Z 7ebe49c3a23e: Pulling fs layer 2024-08-20T22:13:29.9979911Z 5d115b29b5d5: Pulling fs layer 2024-08-20T22:13:29.9980269Z a7a5392e2bf6: Waiting 2024-08-20T22:13:29.9980608Z 7f2d7a04fbe6: Pulling fs layer 2024-08-20T22:13:29.9981019Z d79fd7d52698: Pulling fs layer 2024-08-20T22:13:29.9981389Z 9752d6fbbcab: Pulling fs layer 2024-08-20T22:13:29.9981761Z 83808b2eff4d: Pulling fs layer 2024-08-20T22:13:29.9982111Z 70ac3e38d0fd: Waiting 2024-08-20T22:13:29.9982427Z e56eeff2db93: Waiting 2024-08-20T22:13:29.9982749Z da6a838a66c7: Waiting 2024-08-20T22:13:29.9983064Z 6c1f0aabce7c: Waiting 2024-08-20T22:13:29.9983378Z 7ebe49c3a23e: Waiting 2024-08-20T22:13:29.9983694Z eb09cbf3e416: Waiting 2024-08-20T22:13:29.9984013Z 5d115b29b5d5: Waiting 2024-08-20T22:13:29.9984316Z ee1273153675: Waiting 2024-08-20T22:13:29.9984631Z 7f2d7a04fbe6: Waiting 2024-08-20T22:13:29.9984952Z 9752d6fbbcab: Waiting 2024-08-20T22:13:29.9985261Z 78ee77eddef2: Waiting 2024-08-20T22:13:29.9985576Z 829ed7d91e38: Waiting 2024-08-20T22:13:29.9985893Z c42b3602a1fb: Waiting 2024-08-20T22:13:29.9986209Z 27aeb1b2cd5e: Waiting 2024-08-20T22:13:29.9986514Z b10de02255c0: Waiting 2024-08-20T22:13:29.9986830Z ef1e3ef1e85f: Waiting 2024-08-20T22:13:29.9987151Z e22979f9e0ee: Waiting 2024-08-20T22:13:29.9987461Z d79fd7d52698: Waiting 2024-08-20T22:13:29.9987766Z 37513e9b43d7: Waiting 2024-08-20T22:13:30.1267974Z 224fe954d725: Verifying Checksum 2024-08-20T22:13:30.1268539Z 224fe954d725: Download complete 2024-08-20T22:13:30.2401466Z d527cbbb87e3: Verifying Checksum 2024-08-20T22:13:30.2402036Z d527cbbb87e3: Download complete 2024-08-20T22:13:30.3075105Z b57676e46aee: Download complete 2024-08-20T22:13:30.3418739Z 7a2c55901189: Verifying Checksum 2024-08-20T22:13:30.3419294Z 7a2c55901189: Download complete 2024-08-20T22:13:30.4130504Z a41a8d1c11c8: Verifying Checksum 2024-08-20T22:13:30.4131023Z a41a8d1c11c8: Download complete 2024-08-20T22:13:30.4699783Z 0c1227890755: Download complete 2024-08-20T22:13:30.5393970Z d8d1234baab3: Verifying Checksum 2024-08-20T22:13:30.5394566Z d8d1234baab3: Download complete 2024-08-20T22:13:30.6049642Z 75722010b82e: Verifying Checksum 2024-08-20T22:13:30.6050219Z 75722010b82e: Download complete 2024-08-20T22:13:30.6919481Z ec1e7978c1fe: Verifying Checksum 2024-08-20T22:13:30.6919962Z ec1e7978c1fe: Download complete 2024-08-20T22:13:30.7704347Z ac23c5fcc1ef: Verifying Checksum 2024-08-20T22:13:30.7704971Z ac23c5fcc1ef: Download complete 2024-08-20T22:13:31.4945968Z 7a2c55901189: Pull complete 2024-08-20T22:13:31.8237837Z 224fe954d725: Pull complete 2024-08-20T22:13:32.7403323Z 75722010b82e: Pull complete 2024-08-20T22:13:32.7676725Z d527cbbb87e3: Pull complete 2024-08-20T22:13:32.7930955Z b57676e46aee: Pull complete 2024-08-20T22:13:33.2830073Z ca426ffc9d2b: Verifying Checksum 2024-08-20T22:13:33.2830707Z ca426ffc9d2b: Download complete 2024-08-20T22:13:33.3672937Z 4373aa743caf: Verifying Checksum 2024-08-20T22:13:33.3673499Z 4373aa743caf: Download complete 2024-08-20T22:13:33.4536288Z 75b2abb16c68: Verifying Checksum 2024-08-20T22:13:33.4536830Z 75b2abb16c68: Download complete 2024-08-20T22:13:33.5279009Z ef1e3ef1e85f: Verifying Checksum 2024-08-20T22:13:33.5279564Z ef1e3ef1e85f: Download complete 2024-08-20T22:13:34.5049326Z 0bf61114d37b: Verifying Checksum 2024-08-20T22:13:34.5049890Z 0bf61114d37b: Download complete 2024-08-20T22:13:34.5729657Z 2df849f72ba4: Verifying Checksum 2024-08-20T22:13:34.5730220Z 2df849f72ba4: Download complete 2024-08-20T22:13:34.6473480Z 34c47d63c5de: Verifying Checksum 2024-08-20T22:13:34.6474052Z 34c47d63c5de: Download complete 2024-08-20T22:13:34.9692628Z 1fef4bd19792: Verifying Checksum 2024-08-20T22:13:34.9693204Z 1fef4bd19792: Download complete 2024-08-20T22:13:43.6528718Z a8c1e85b5e14: Verifying Checksum 2024-08-20T22:13:43.6529344Z a8c1e85b5e14: Download complete 2024-08-20T22:13:43.7156877Z b119c4fc6c58: Verifying Checksum 2024-08-20T22:13:43.7157378Z b119c4fc6c58: Download complete 2024-08-20T22:13:43.7907614Z 33b639226eee: Verifying Checksum 2024-08-20T22:13:43.7908062Z 33b639226eee: Download complete 2024-08-20T22:13:43.8625739Z d3547bd43146: Download complete 2024-08-20T22:13:43.9552619Z 5645520256fa: Verifying Checksum 2024-08-20T22:13:43.9553215Z 5645520256fa: Download complete 2024-08-20T22:13:44.0266417Z 829ed7d91e38: Download complete 2024-08-20T22:13:45.3127220Z 385d0421d67c: Verifying Checksum 2024-08-20T22:13:45.3127817Z 385d0421d67c: Download complete 2024-08-20T22:13:45.3994393Z 7cbfce999d76: Verifying Checksum 2024-08-20T22:13:45.3994918Z 7cbfce999d76: Download complete 2024-08-20T22:13:45.4742178Z 6658e558ebe4: Verifying Checksum 2024-08-20T22:13:45.4742661Z 6658e558ebe4: Download complete 2024-08-20T22:13:45.5592543Z 33444a5a7e01: Download complete 2024-08-20T22:13:45.6326755Z 7c47122f4e09: Verifying Checksum 2024-08-20T22:13:45.6327236Z 7c47122f4e09: Download complete 2024-08-20T22:13:45.7394864Z 1811ff4d3556: Verifying Checksum 2024-08-20T22:13:51.7294674Z 7754ee068b33: Verifying Checksum 2024-08-20T22:13:51.7295209Z 7754ee068b33: Download complete 2024-08-20T22:13:51.8116588Z 0479fbac3361: Download complete 2024-08-20T22:13:51.8970262Z f1d6874dee9c: Verifying Checksum 2024-08-20T22:13:51.8970850Z f1d6874dee9c: Download complete 2024-08-20T22:13:52.3079595Z afc429e87b59: Verifying Checksum 2024-08-20T22:13:52.3080103Z afc429e87b59: Download complete 2024-08-20T22:13:52.3761020Z 019620471c25: Download complete 2024-08-20T22:13:52.4546382Z 27aeb1b2cd5e: Verifying Checksum 2024-08-20T22:13:52.4546804Z 27aeb1b2cd5e: Download complete 2024-08-20T22:13:52.7089916Z 829c0e19f4b7: Verifying Checksum 2024-08-20T22:13:52.7090555Z 829c0e19f4b7: Download complete 2024-08-20T22:13:52.7710815Z 560cd6122a38: Download complete 2024-08-20T22:13:52.8485557Z 5a4eb19d7219: Verifying Checksum 2024-08-20T22:13:52.8486060Z 5a4eb19d7219: Download complete 2024-08-20T22:13:52.9276437Z da6a838a66c7: Download complete 2024-08-20T22:13:55.9191477Z 7ed32bc8e469: Verifying Checksum 2024-08-20T22:13:55.9191949Z 7ed32bc8e469: Download complete 2024-08-20T22:13:55.9888124Z 4d4fa6802ab3: Verifying Checksum 2024-08-20T22:13:55.9888697Z 4d4fa6802ab3: Download complete 2024-08-20T22:13:56.0451705Z 0f085d4fa279: Download complete 2024-08-20T22:13:56.5592164Z 4a0412373305: Verifying Checksum 2024-08-20T22:13:56.5592920Z 4a0412373305: Download complete 2024-08-20T22:13:56.6243409Z 302c8c177944: Verifying Checksum 2024-08-20T22:13:56.6244027Z 302c8c177944: Download complete 2024-08-20T22:13:56.6936534Z 02970347dff6: Verifying Checksum 2024-08-20T22:13:56.6937161Z 02970347dff6: Download complete 2024-08-20T22:13:56.7406585Z a8c1e85b5e14: Pull complete 2024-08-20T22:13:56.7634667Z 916c5e858016: Verifying Checksum 2024-08-20T22:13:56.7635097Z 916c5e858016: Download complete 2024-08-20T22:13:56.8356493Z c42b3602a1fb: Verifying Checksum 2024-08-20T22:13:56.8357330Z c42b3602a1fb: Download complete 2024-08-20T22:13:56.8697710Z a41a8d1c11c8: Pull complete 2024-08-20T22:13:56.9482652Z 0c1227890755: Pull complete 2024-08-20T22:13:57.0244693Z d8d1234baab3: Pull complete 2024-08-20T22:14:03.6630669Z 6efb96fc87ea: Verifying Checksum 2024-08-20T22:14:03.6631151Z 6efb96fc87ea: Download complete 2024-08-20T22:14:03.7400894Z f957c1624481: Verifying Checksum 2024-08-20T22:14:03.7401374Z f957c1624481: Download complete 2024-08-20T22:14:03.8238485Z 768c31dca016: Download complete 2024-08-20T22:14:03.9029831Z 8c32d0ecebbe: Verifying Checksum 2024-08-20T22:14:03.9030287Z 8c32d0ecebbe: Download complete 2024-08-20T22:14:03.9122199Z 4f4fb700ef54: Verifying Checksum 2024-08-20T22:14:03.9122730Z 4f4fb700ef54: Download complete 2024-08-20T22:14:03.9894088Z 78ee77eddef2: Download complete 2024-08-20T22:14:04.0572982Z 37513e9b43d7: Download complete 2024-08-20T22:14:04.2147098Z a7a5392e2bf6: Verifying Checksum 2024-08-20T22:14:04.2147750Z a7a5392e2bf6: Download complete 2024-08-20T22:14:04.2816748Z b10de02255c0: Download complete 2024-08-20T22:14:04.3581142Z e56eeff2db93: Verifying Checksum 2024-08-20T22:14:04.3581657Z e56eeff2db93: Download complete 2024-08-20T22:14:04.4994053Z eb09cbf3e416: Verifying Checksum 2024-08-20T22:14:04.4994686Z eb09cbf3e416: Download complete 2024-08-20T22:14:04.6370066Z ee1273153675: Verifying Checksum 2024-08-20T22:14:04.6370667Z ee1273153675: Download complete 2024-08-20T22:14:04.7124641Z e22979f9e0ee: Verifying Checksum 2024-08-20T22:14:04.7125305Z e22979f9e0ee: Download complete 2024-08-20T22:14:05.3044654Z 6c1f0aabce7c: Verifying Checksum 2024-08-20T22:14:05.3045257Z 6c1f0aabce7c: Download complete 2024-08-20T22:14:05.3773077Z 7ebe49c3a23e: Verifying Checksum 2024-08-20T22:14:05.3773685Z 7ebe49c3a23e: Download complete 2024-08-20T22:14:32.3981233Z 5d115b29b5d5: Verifying Checksum 2024-08-20T22:14:32.3981694Z 5d115b29b5d5: Download complete 2024-08-20T22:14:32.4720840Z 7f2d7a04fbe6: Download complete 2024-08-20T22:14:32.5497217Z d79fd7d52698: Verifying Checksum 2024-08-20T22:14:32.5497699Z d79fd7d52698: Download complete 2024-08-20T22:14:34.0656122Z 9752d6fbbcab: Verifying Checksum 2024-08-20T22:14:34.0656764Z 9752d6fbbcab: Download complete 2024-08-20T22:14:34.1468650Z 83808b2eff4d: Verifying Checksum 2024-08-20T22:14:34.1469150Z 83808b2eff4d: Download complete 2024-08-20T22:15:11.0109807Z 11fad972ef6c: Verifying Checksum 2024-08-20T22:15:11.0110267Z 11fad972ef6c: Download complete 2024-08-20T22:15:12.6516734Z cbb79a705353: Verifying Checksum 2024-08-20T22:15:12.6517296Z cbb79a705353: Download complete 2024-08-20T22:15:27.2365427Z 7ed32bc8e469: Pull complete 2024-08-20T22:15:27.4268941Z ec1e7978c1fe: Pull complete 2024-08-20T22:15:27.6181953Z ac23c5fcc1ef: Pull complete 2024-08-20T22:15:36.0506173Z ca426ffc9d2b: Pull complete 2024-08-20T22:15:36.2190516Z 4373aa743caf: Pull complete 2024-08-20T22:15:36.3035222Z 75b2abb16c68: Pull complete 2024-08-20T22:15:36.3536521Z ef1e3ef1e85f: Pull complete 2024-08-20T22:15:38.8981774Z 0bf61114d37b: Pull complete 2024-08-20T22:15:39.0230924Z 2df849f72ba4: Pull complete 2024-08-20T22:15:39.0695419Z 34c47d63c5de: Pull complete 2024-08-20T22:15:39.1105960Z 1fef4bd19792: Pull complete 2024-08-20T22:16:37.3764463Z 6efb96fc87ea: Pull complete 2024-08-20T22:16:37.5832793Z b119c4fc6c58: Pull complete 2024-08-20T22:16:37.7764396Z 33b639226eee: Pull complete 2024-08-20T22:16:37.8563045Z d3547bd43146: Pull complete 2024-08-20T22:16:37.9227219Z 5645520256fa: Pull complete 2024-08-20T22:16:37.9782073Z 829ed7d91e38: Pull complete 2024-08-20T22:16:41.4192602Z 385d0421d67c: Pull complete 2024-08-20T22:16:41.5245300Z 7cbfce999d76: Pull complete 2024-08-20T22:16:41.6091219Z 6658e558ebe4: Pull complete 2024-08-20T22:16:41.7815614Z 33444a5a7e01: Pull complete 2024-08-20T22:16:41.8891887Z 7c47122f4e09: Pull complete 2024-08-20T22:16:41.9770417Z 1811ff4d3556: Pull complete 2024-08-20T22:16:52.4686188Z 7754ee068b33: Pull complete 2024-08-20T22:16:52.5328473Z 0479fbac3361: Pull complete 2024-08-20T22:16:52.6891874Z f1d6874dee9c: Pull complete 2024-08-20T22:16:53.5975183Z afc429e87b59: Pull complete 2024-08-20T22:16:53.7642616Z 019620471c25: Pull complete 2024-08-20T22:16:53.8422833Z 27aeb1b2cd5e: Pull complete 2024-08-20T22:16:54.2444909Z 829c0e19f4b7: Pull complete 2024-08-20T22:16:54.3721911Z 560cd6122a38: Pull complete 2024-08-20T22:16:54.5288885Z 5a4eb19d7219: Pull complete 2024-08-20T22:16:54.6111584Z da6a838a66c7: Pull complete 2024-08-20T22:17:33.0603938Z cbb79a705353: Pull complete 2024-08-20T22:17:33.1927344Z 4d4fa6802ab3: Pull complete 2024-08-20T22:17:33.2914298Z 0f085d4fa279: Pull complete 2024-08-20T22:17:35.6346690Z 4a0412373305: Pull complete 2024-08-20T22:17:35.6609580Z 302c8c177944: Pull complete 2024-08-20T22:17:35.6889654Z 02970347dff6: Pull complete 2024-08-20T22:17:35.7443617Z 916c5e858016: Pull complete 2024-08-20T22:17:35.7764516Z c42b3602a1fb: Pull complete 2024-08-20T22:18:57.5925619Z 11fad972ef6c: Pull complete 2024-08-20T22:18:57.7766495Z f957c1624481: Pull complete 2024-08-20T22:18:57.8656179Z 768c31dca016: Pull complete 2024-08-20T22:18:58.1096619Z 8c32d0ecebbe: Pull complete 2024-08-20T22:18:58.2703965Z 4f4fb700ef54: Pull complete 2024-08-20T22:18:58.4834423Z 78ee77eddef2: Pull complete 2024-08-20T22:18:58.6280076Z 37513e9b43d7: Pull complete 2024-08-20T22:18:58.9101414Z a7a5392e2bf6: Pull complete 2024-08-20T22:18:59.0519715Z b10de02255c0: Pull complete 2024-08-20T22:18:59.1267562Z e56eeff2db93: Pull complete 2024-08-20T22:18:59.1806256Z 70ac3e38d0fd: Pull complete 2024-08-20T22:18:59.2440041Z eb09cbf3e416: Pull complete 2024-08-20T22:19:00.5971845Z ee1273153675: Pull complete 2024-08-20T22:19:00.7750266Z e22979f9e0ee: Pull complete 2024-08-20T22:19:02.9164811Z 6c1f0aabce7c: Pull complete 2024-08-20T22:19:03.0924519Z 7ebe49c3a23e: Pull complete 2024-08-20T22:19:16.2626081Z 5d115b29b5d5: Pull complete 2024-08-20T22:19:16.3360973Z 7f2d7a04fbe6: Pull complete 2024-08-20T22:19:16.3919018Z d79fd7d52698: Pull complete 2024-08-20T22:19:17.2394444Z 9752d6fbbcab: Pull complete 2024-08-20T22:19:17.4219830Z 83808b2eff4d: Pull complete 2024-08-20T22:19:18.0063053Z Digest: sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T22:19:18.0409133Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:18.0643166Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:19:18.0730617Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:18.0731560Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-08-20T22:19:18.0743818Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:19:18.0744319Z env: 2024-08-20T22:19:18.0744593Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:18.0744940Z ##[endgroup] 2024-08-20T22:19:18.0938847Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-08-20T22:19:18.0939400Z with: 2024-08-20T22:19:18.0939689Z driver-version: 550.54.15 2024-08-20T22:19:18.0940026Z env: 2024-08-20T22:19:18.0940306Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:18.0940667Z ##[endgroup] 2024-08-20T22:19:18.1025854Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:19:18.1026397Z with: 2024-08-20T22:19:18.1026674Z timeout_minutes: 10 2024-08-20T22:19:18.1026991Z max_attempts: 3 2024-08-20T22:19:18.1059161Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" # Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with # more than one GPUs. This just needs to be run once. The command fails # on subsequent runs and complains that the mode is already on, but that's # ok sudo nvidia-persistenced || true # This should show persistence mode ON nvidia-smi 2024-08-20T22:19:18.1091314Z retry_wait_seconds: 10 2024-08-20T22:19:18.1091685Z polling_interval_seconds: 1 2024-08-20T22:19:18.1092047Z warning_on_retry: true 2024-08-20T22:19:18.1092398Z continue_on_error: false 2024-08-20T22:19:18.1092727Z env: 2024-08-20T22:19:18.1092998Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:19:18.1093345Z DRIVER_VERSION: 550.54.15 2024-08-20T22:19:18.1093693Z ##[endgroup] 2024-08-20T22:19:18.1911178Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-08-20T22:19:18.1912026Z + pre_install_nvidia_driver_amzn2 2024-08-20T22:19:18.1913013Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-08-20T22:19:18.7838605Z No match for argument: nvidia-driver-latest-dkms 2024-08-20T22:19:18.7839750Z No packages marked for removal. 2024-08-20T22:19:18.7919062Z Dependencies resolved. 2024-08-20T22:19:18.7928815Z Nothing to do. 2024-08-20T22:19:18.7929302Z Complete! 2024-08-20T22:19:18.8500801Z + install_nvidia_driver_common 2024-08-20T22:19:18.8508115Z + echo 'Before installing NVIDIA driver' 2024-08-20T22:19:18.8508691Z + lspci 2024-08-20T22:19:18.8509096Z Before installing NVIDIA driver 2024-08-20T22:19:18.8618626Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:18.8619367Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:18.8620281Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:18.8621110Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:18.8621982Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:18.8623002Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:18.8623871Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:18.8624957Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:18.8625537Z + lsmod 2024-08-20T22:19:18.8862140Z Module Size Used by 2024-08-20T22:19:18.8862700Z ib_core 454656 0 2024-08-20T22:19:18.8863202Z veth 36864 0 2024-08-20T22:19:18.8863692Z nvidia_modeset 1351680 0 2024-08-20T22:19:18.8864155Z video 65536 1 nvidia_modeset 2024-08-20T22:19:18.8864664Z wmi 36864 1 video 2024-08-20T22:19:18.8865179Z nvidia_uvm 4706304 0 2024-08-20T22:19:18.8865749Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:18.8866315Z drm 602112 1 nvidia 2024-08-20T22:19:18.8866752Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:18.8867267Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:18.8867772Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:18.8870063Z xt_conntrack 16384 1 2024-08-20T22:19:18.8871106Z nft_chain_nat 16384 3 2024-08-20T22:19:18.8871785Z xt_MASQUERADE 20480 1 2024-08-20T22:19:18.8872283Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:18.8872993Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:18.8873559Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:18.8874188Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:18.8874636Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:18.8875055Z xfrm_user 57344 1 2024-08-20T22:19:18.8875442Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:18.8875980Z xt_addrtype 16384 2 2024-08-20T22:19:18.8876425Z nft_compat 20480 4 2024-08-20T22:19:18.8877002Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:18.8877691Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:18.8878218Z br_netfilter 36864 0 2024-08-20T22:19:18.8878619Z bridge 307200 1 br_netfilter 2024-08-20T22:19:18.8879046Z stp 16384 1 bridge 2024-08-20T22:19:18.8879470Z llc 16384 2 bridge,stp 2024-08-20T22:19:18.8879872Z overlay 167936 0 2024-08-20T22:19:18.8880233Z tls 114688 0 2024-08-20T22:19:18.8880596Z nls_ascii 16384 1 2024-08-20T22:19:18.8880965Z nls_cp437 20480 1 2024-08-20T22:19:18.8881341Z vfat 24576 1 2024-08-20T22:19:18.8881725Z fat 86016 1 vfat 2024-08-20T22:19:18.8882120Z ena 167936 0 2024-08-20T22:19:18.8882598Z ptp 36864 1 ena 2024-08-20T22:19:18.8883096Z pps_core 24576 1 ptp 2024-08-20T22:19:18.8883600Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:18.8884081Z aesni_intel 393216 0 2024-08-20T22:19:18.8884537Z i8042 45056 0 2024-08-20T22:19:18.8885140Z serio 28672 3 i8042 2024-08-20T22:19:18.8885679Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:18.8886342Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:18.8886971Z button 24576 0 2024-08-20T22:19:18.8887439Z sunrpc 692224 1 2024-08-20T22:19:18.8887888Z sch_fq_codel 20480 17 2024-08-20T22:19:18.8888253Z dm_mod 188416 0 2024-08-20T22:19:18.8888603Z fuse 163840 1 2024-08-20T22:19:18.8889043Z configfs 57344 1 2024-08-20T22:19:18.8889515Z loop 36864 0 2024-08-20T22:19:18.8889974Z dax 45056 1 dm_mod 2024-08-20T22:19:18.8890359Z dmi_sysfs 20480 0 2024-08-20T22:19:18.8890719Z crc32_pclmul 16384 0 2024-08-20T22:19:18.8891080Z crc32c_intel 24576 0 2024-08-20T22:19:18.8891432Z efivarfs 24576 1 2024-08-20T22:19:18.8891807Z + modinfo nvidia 2024-08-20T22:19:18.8892732Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:18.8893527Z alias: char-major-195-* 2024-08-20T22:19:18.8894029Z version: 550.54.15 2024-08-20T22:19:18.8894372Z supported: external 2024-08-20T22:19:18.8894704Z license: NVIDIA 2024-08-20T22:19:18.8895070Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:18.8895695Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:18.8896266Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:18.8896726Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:18.8897216Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:18.8897687Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:18.8898189Z depends: i2c-core,drm 2024-08-20T22:19:18.8898542Z retpoline: Y 2024-08-20T22:19:18.8898849Z name: nvidia 2024-08-20T22:19:18.8899441Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:18.8900118Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:18.8900758Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:18.8901433Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:18.8901864Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:18.8902291Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:18.8902736Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:18.8903166Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:18.8903713Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:18.8904232Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:18.8904780Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:18.8905242Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:18.8905670Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:18.8906075Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:18.8906522Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:18.8907033Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:18.8907608Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:18.8908161Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:18.8908764Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:18.8909340Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:18.8909936Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:18.8910514Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:18.8910995Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:18.8911514Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:18.8912047Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:18.8912532Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:18.8912983Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:18.8913448Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:18.8913905Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:18.8914345Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:18.8914844Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:18.8915352Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:18.8915821Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:18.8916295Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:18.8916785Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:18.8917249Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:18.8917737Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:18.8918204Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:18.8918603Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:18.8919062Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:18.8919521Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:18.8919972Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:18.8920430Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:18.8921006Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:18.8921458Z parm: rm_firmware_active:charp 2024-08-20T22:19:18.8921859Z + HAS_NVIDIA_DRIVER=0 2024-08-20T22:19:18.8922232Z ++ command -v nvidia-smi 2024-08-20T22:19:18.8922622Z + '[' -x /usr/bin/nvidia-smi ']' 2024-08-20T22:19:18.8922988Z + set +e 2024-08-20T22:19:18.8923486Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-08-20T22:19:18.9180673Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-08-20T22:19:18.9181242Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:18.9181778Z + '[' 0 -ne 0 ']' 2024-08-20T22:19:18.9182228Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-08-20T22:19:18.9182605Z + HAS_NVIDIA_DRIVER=1 2024-08-20T22:19:18.9183310Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-08-20T22:19:18.9184003Z + set -e 2024-08-20T22:19:18.9184305Z + '[' 1 -eq 0 ']' 2024-08-20T22:19:18.9185013Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-08-20T22:19:18.9185969Z + post_install_nvidia_driver_common 2024-08-20T22:19:18.9187992Z + sudo modprobe nvidia 2024-08-20T22:19:19.0636102Z + echo 'After installing NVIDIA driver' 2024-08-20T22:19:19.0636680Z + lspci 2024-08-20T22:19:19.0636966Z After installing NVIDIA driver 2024-08-20T22:19:19.0741409Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-08-20T22:19:19.0742354Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-08-20T22:19:19.0743279Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-08-20T22:19:19.0744140Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-08-20T22:19:19.0744943Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller 2024-08-20T22:19:19.0745717Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-08-20T22:19:19.0746431Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1) 2024-08-20T22:19:19.0747243Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-08-20T22:19:19.0747821Z + lsmod 2024-08-20T22:19:19.0774487Z Module Size Used by 2024-08-20T22:19:19.0775033Z ib_core 454656 0 2024-08-20T22:19:19.0775545Z veth 36864 0 2024-08-20T22:19:19.0776016Z nvidia_modeset 1351680 0 2024-08-20T22:19:19.0776502Z video 65536 1 nvidia_modeset 2024-08-20T22:19:19.0776937Z wmi 36864 1 video 2024-08-20T22:19:19.0777323Z nvidia_uvm 4706304 0 2024-08-20T22:19:19.0777758Z nvidia 54071296 7 nvidia_uvm,nvidia_modeset 2024-08-20T22:19:19.0778226Z drm 602112 1 nvidia 2024-08-20T22:19:19.0778661Z drm_panel_orientation_quirks 28672 1 drm 2024-08-20T22:19:19.0779176Z backlight 24576 3 video,drm,nvidia_modeset 2024-08-20T22:19:19.0779663Z i2c_core 106496 2 nvidia,drm 2024-08-20T22:19:19.0780077Z xt_conntrack 16384 1 2024-08-20T22:19:19.0780460Z nft_chain_nat 16384 3 2024-08-20T22:19:19.0780830Z xt_MASQUERADE 20480 1 2024-08-20T22:19:19.0781265Z nf_nat 57344 2 nft_chain_nat,xt_MASQUERADE 2024-08-20T22:19:19.0781746Z nf_conntrack_netlink 57344 0 2024-08-20T22:19:19.0782312Z nf_conntrack 184320 4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE 2024-08-20T22:19:19.0782944Z nf_defrag_ipv6 24576 1 nf_conntrack 2024-08-20T22:19:19.0783395Z nf_defrag_ipv4 16384 1 nf_conntrack 2024-08-20T22:19:19.0783818Z xfrm_user 57344 1 2024-08-20T22:19:19.0784207Z xfrm_algo 16384 1 xfrm_user 2024-08-20T22:19:19.0784612Z xt_addrtype 16384 2 2024-08-20T22:19:19.0784978Z nft_compat 20480 4 2024-08-20T22:19:19.0785428Z nf_tables 307200 57 nft_compat,nft_chain_nat 2024-08-20T22:19:19.0786027Z nfnetlink 20480 4 nft_compat,nf_conntrack_netlink,nf_tables 2024-08-20T22:19:19.0786837Z br_netfilter 36864 0 2024-08-20T22:19:19.0787255Z bridge 307200 1 br_netfilter 2024-08-20T22:19:19.0787751Z stp 16384 1 bridge 2024-08-20T22:19:19.0788178Z llc 16384 2 bridge,stp 2024-08-20T22:19:19.0788581Z overlay 167936 0 2024-08-20T22:19:19.0788943Z tls 114688 0 2024-08-20T22:19:19.0789301Z nls_ascii 16384 1 2024-08-20T22:19:19.0789659Z nls_cp437 20480 1 2024-08-20T22:19:19.0790010Z vfat 24576 1 2024-08-20T22:19:19.0790370Z fat 86016 1 vfat 2024-08-20T22:19:19.0790751Z ena 167936 0 2024-08-20T22:19:19.0791103Z ptp 36864 1 ena 2024-08-20T22:19:19.0791487Z pps_core 24576 1 ptp 2024-08-20T22:19:19.0791877Z ghash_clmulni_intel 16384 0 2024-08-20T22:19:19.0792247Z aesni_intel 393216 0 2024-08-20T22:19:19.0792615Z i8042 45056 0 2024-08-20T22:19:19.0793029Z serio 28672 3 i8042 2024-08-20T22:19:19.0793736Z crypto_simd 16384 1 aesni_intel 2024-08-20T22:19:19.0794398Z cryptd 28672 2 crypto_simd,ghash_clmulni_intel 2024-08-20T22:19:19.0795034Z button 24576 0 2024-08-20T22:19:19.0795499Z sunrpc 692224 1 2024-08-20T22:19:19.0795978Z sch_fq_codel 20480 17 2024-08-20T22:19:19.0796346Z dm_mod 188416 0 2024-08-20T22:19:19.0796706Z fuse 163840 1 2024-08-20T22:19:19.0797072Z configfs 57344 1 2024-08-20T22:19:19.0797431Z loop 36864 0 2024-08-20T22:19:19.0797792Z dax 45056 1 dm_mod 2024-08-20T22:19:19.0798183Z dmi_sysfs 20480 0 2024-08-20T22:19:19.0798548Z crc32_pclmul 16384 0 2024-08-20T22:19:19.0798917Z crc32c_intel 24576 0 2024-08-20T22:19:19.0799279Z efivarfs 24576 1 2024-08-20T22:19:19.0799738Z + modinfo nvidia 2024-08-20T22:19:19.0800419Z filename: /lib/modules/6.1.94-99.176.amzn2023.x86_64/kernel/drivers/video/nvidia.ko 2024-08-20T22:19:19.0801114Z alias: char-major-195-* 2024-08-20T22:19:19.0801485Z version: 550.54.15 2024-08-20T22:19:19.0801830Z supported: external 2024-08-20T22:19:19.0802163Z license: NVIDIA 2024-08-20T22:19:19.0802537Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-08-20T22:19:19.0803006Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-08-20T22:19:19.0803456Z srcversion: 833721318DA517F0C2FEC97 2024-08-20T22:19:19.0803913Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-08-20T22:19:19.0804406Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-08-20T22:19:19.0804997Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-08-20T22:19:19.0805487Z depends: i2c-core,drm 2024-08-20T22:19:19.0805838Z retpoline: Y 2024-08-20T22:19:19.0806134Z name: nvidia 2024-08-20T22:19:19.0806740Z vermagic: 6.1.94-99.176.amzn2023.x86_64 SMP preempt mod_unload modversions 2024-08-20T22:19:19.0807440Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-08-20T22:19:19.0808079Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-08-20T22:19:19.0808651Z parm: NVreg_ResmanDebugLevel:int 2024-08-20T22:19:19.0809093Z parm: NVreg_RmLogonRC:int 2024-08-20T22:19:19.0809518Z parm: NVreg_ModifyDeviceFiles:int 2024-08-20T22:19:19.0809964Z parm: NVreg_DeviceFileUID:int 2024-08-20T22:19:19.0810392Z parm: NVreg_DeviceFileGID:int 2024-08-20T22:19:19.0810823Z parm: NVreg_DeviceFileMode:int 2024-08-20T22:19:19.0811340Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-08-20T22:19:19.0811935Z parm: NVreg_UsePageAttributeTable:int 2024-08-20T22:19:19.0812427Z parm: NVreg_EnablePCIeGen3:int 2024-08-20T22:19:19.0812854Z parm: NVreg_EnableMSI:int 2024-08-20T22:19:19.0813264Z parm: NVreg_TCEBypassMode:int 2024-08-20T22:19:19.0813846Z parm: NVreg_EnableStreamMemOPs:int 2024-08-20T22:19:19.0814372Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-08-20T22:19:19.0814982Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-08-20T22:19:19.0815552Z parm: NVreg_EnableS0ixPowerManagement:int 2024-08-20T22:19:19.0816153Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:19.0816729Z parm: NVreg_DynamicPowerManagement:int 2024-08-20T22:19:19.0817328Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-08-20T22:19:19.0817909Z parm: NVreg_EnableGpuFirmware:int 2024-08-20T22:19:19.0818391Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-08-20T22:19:19.0818910Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-08-20T22:19:19.0819444Z parm: NVreg_EnableUserNUMAManagement:int 2024-08-20T22:19:19.0819931Z parm: NVreg_MemoryPoolSize:int 2024-08-20T22:19:19.0820387Z parm: NVreg_KMallocHeapMaxSize:int 2024-08-20T22:19:19.0820870Z parm: NVreg_VMallocHeapMaxSize:int 2024-08-20T22:19:19.0821424Z parm: NVreg_IgnoreMMIOCheck:int 2024-08-20T22:19:19.0821868Z parm: NVreg_NvLinkDisable:int 2024-08-20T22:19:19.0822357Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-08-20T22:19:19.0822872Z parm: NVreg_RegisterPCIDriver:int 2024-08-20T22:19:19.0823341Z parm: NVreg_EnableResizableBar:int 2024-08-20T22:19:19.0823826Z parm: NVreg_EnableDbgBreakpoint:int 2024-08-20T22:19:19.0824308Z parm: NVreg_EnableNonblockingOpen:int 2024-08-20T22:19:19.0824785Z parm: NVreg_RegistryDwords:charp 2024-08-20T22:19:19.0825281Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-08-20T22:19:19.0825752Z parm: NVreg_RmMsg:charp 2024-08-20T22:19:19.0826149Z parm: NVreg_GpuBlacklist:charp 2024-08-20T22:19:19.0826609Z parm: NVreg_TemporaryFilePath:charp 2024-08-20T22:19:19.0827070Z parm: NVreg_ExcludedGpus:charp 2024-08-20T22:19:19.0827522Z parm: NVreg_DmaRemapPeerMmio:int 2024-08-20T22:19:19.0827994Z parm: NVreg_RmNvlinkBandwidth:charp 2024-08-20T22:19:19.0828460Z parm: NVreg_ImexChannelCount:int 2024-08-20T22:19:19.0828905Z parm: rm_firmware_active:charp 2024-08-20T22:19:19.0829292Z + set +e 2024-08-20T22:19:19.0829608Z + nvidia-smi 2024-08-20T22:19:19.1005440Z Tue Aug 20 22:19:19 2024 2024-08-20T22:19:19.1006134Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:19.1006958Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:19:19.1007748Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:19.1008532Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:19:19.1009412Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:19:19.1010104Z | | | MIG M. | 2024-08-20T22:19:19.1010644Z |=========================================+========================+======================| 2024-08-20T22:19:19.1215605Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:19:19.1216329Z | 0% 21C P8 19W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:19:19.1216949Z | | | N/A | 2024-08-20T22:19:19.1217639Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:19:19.1220651Z 2024-08-20T22:19:19.1221331Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:19.1222146Z | Processes: | 2024-08-20T22:19:19.1222852Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:19:19.1223530Z | ID ID Usage | 2024-08-20T22:19:19.1224090Z |=========================================================================================| 2024-08-20T22:19:19.1225939Z | No running processes found | 2024-08-20T22:19:19.1226713Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:19:19.4327108Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-08-20T22:19:19.4516947Z NVIDIA A10G 2024-08-20T22:19:19.4580915Z + NVIDIA_SMI_STATUS=0 2024-08-20T22:19:19.4581456Z + '[' 0 -eq 0 ']' 2024-08-20T22:19:19.4581987Z + echo 'INFO: Ignoring allowed status 0' 2024-08-20T22:19:19.4582526Z + set -e 2024-08-20T22:19:19.4582835Z INFO: Ignoring allowed status 0 2024-08-20T22:19:19.4590154Z == Installing nvidia container toolkit for amzn2023 == 2024-08-20T22:19:19.4593834Z + sudo yum install -y yum-utils 2024-08-20T22:19:19.9433595Z Last metadata expiration check: 2:31:18 ago on Tue Aug 20 19:48:01 2024. 2024-08-20T22:19:19.9654403Z Package dnf-utils-4.3.0-13.amzn2023.0.4.noarch is already installed. 2024-08-20T22:19:19.9957026Z Dependencies resolved. 2024-08-20T22:19:20.0083405Z Nothing to do. 2024-08-20T22:19:20.0083813Z Complete! 2024-08-20T22:19:20.0971308Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]] 2024-08-20T22:19:20.0972554Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:20.0973885Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:20.4357857Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo 2024-08-20T22:19:20.5137797Z + sudo yum install -y nvidia-docker2 2024-08-20T22:19:20.9932916Z nvidia-container-toolkit 10 kB/s | 833 B 00:00 2024-08-20T22:19:21.0154245Z Package nvidia-docker2-2.14.0-1.noarch is already installed. 2024-08-20T22:19:21.0452939Z Dependencies resolved. 2024-08-20T22:19:21.0577821Z Nothing to do. 2024-08-20T22:19:21.0578173Z Complete! 2024-08-20T22:19:21.1590090Z + sudo systemctl restart docker 2024-08-20T22:20:22.0402948Z nvidia-persistenced failed to initialize. Check syslog for more details. 2024-08-20T22:20:22.0644422Z Tue Aug 20 22:20:22 2024 2024-08-20T22:20:22.0647158Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:22.0648093Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-08-20T22:20:22.0648875Z |-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:22.0649859Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-08-20T22:20:22.0650923Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-08-20T22:20:22.0651816Z | | | MIG M. | 2024-08-20T22:20:22.0652354Z |=========================================+========================+======================| 2024-08-20T22:20:22.0837212Z | 0 NVIDIA A10G On | 00000000:00:1E.0 Off | 0 | 2024-08-20T22:20:22.0838024Z | 0% 21C P8 22W / 300W | 0MiB / 23028MiB | 0% Default | 2024-08-20T22:20:22.0838660Z | | | N/A | 2024-08-20T22:20:22.0839404Z +-----------------------------------------+------------------------+----------------------+ 2024-08-20T22:20:22.0842052Z 2024-08-20T22:20:22.0843248Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:22.0843890Z | Processes: | 2024-08-20T22:20:22.0844590Z | GPU GI CI PID Type Process name GPU Memory | 2024-08-20T22:20:22.0845400Z | ID ID Usage | 2024-08-20T22:20:22.0845957Z |=========================================================================================| 2024-08-20T22:20:22.0847180Z | No running processes found | 2024-08-20T22:20:22.0847961Z +-----------------------------------------------------------------------------------------+ 2024-08-20T22:20:23.2333328Z Command completed after 1 attempt(s). 2024-08-20T22:20:23.2424941Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:23.2425830Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-08-20T22:20:23.2426858Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-08-20T22:20:23.2427608Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:23.2444531Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:23.2445225Z env: 2024-08-20T22:20:23.2445561Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:23.2446084Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:23.2446656Z ##[endgroup] 2024-08-20T22:20:23.5546288Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:23.5805951Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1) 2024-08-20T22:20:23.5812681Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84) 2024-08-20T22:20:23.7127196Z Prepare all required actions 2024-08-20T22:20:23.7127710Z Getting action download info 2024-08-20T22:20:23.8230077Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-08-20T22:20:24.0212868Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-08-20T22:20:24.1373734Z ##[group]Run ./.github/actions/download-build-artifacts 2024-08-20T22:20:24.1374214Z with: 2024-08-20T22:20:24.1374538Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:24.1375003Z s3-bucket: gha-artifacts 2024-08-20T22:20:24.1375343Z env: 2024-08-20T22:20:24.1375620Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:24.1376054Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:24.1376530Z ##[endgroup] 2024-08-20T22:20:24.1410171Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:24.1410601Z with: 2024-08-20T22:20:24.1410975Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:24.1411429Z s3-bucket: gha-artifacts 2024-08-20T22:20:24.1411786Z region: us-east-1 2024-08-20T22:20:24.1412090Z env: 2024-08-20T22:20:24.1412367Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:24.1412824Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:24.1413297Z ##[endgroup] 2024-08-20T22:20:24.6181022Z (node:233738) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:24.6181892Z 2024-08-20T22:20:24.6182216Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:24.6182958Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:24.6183980Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:24.6703067Z Found 1 objects with prefix pytorch/pytorch/10479310725/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-08-20T22:20:24.6704170Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:32.2002023Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-08-20T22:20:32.2008486Z Artifact download has finished successfully 2024-08-20T22:20:32.2393505Z ##[group]Run unzip -o artifacts.zip 2024-08-20T22:20:32.2393942Z unzip -o artifacts.zip 2024-08-20T22:20:32.2402910Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:32.2403399Z env: 2024-08-20T22:20:32.2403680Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:32.2404127Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:32.2404597Z ##[endgroup] 2024-08-20T22:20:32.2499587Z Archive: artifacts.zip 2024-08-20T22:20:32.2501581Z creating: dist/ 2024-08-20T22:20:34.3846110Z inflating: dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:34.3847002Z creating: build/custom_test_artifacts/ 2024-08-20T22:20:34.3847815Z creating: build/custom_test_artifacts/custom-op-build/ 2024-08-20T22:20:34.3848853Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-08-20T22:20:34.3850384Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:34.3856582Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:34.3857890Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:34.3859201Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:34.3860644Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:34.3861983Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:34.3863559Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:34.3865351Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:34.3866800Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:34.3868211Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:34.3869855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:34.3871486Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:34.3873707Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:34.3875326Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:34.3877128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:34.3878999Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:34.3880481Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:34.3881945Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:34.3922133Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:34.3962921Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:34.3965142Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:34.4010981Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:34.4012961Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:34.4014955Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:34.4017016Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:34.4019005Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:34.4020950Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:34.4022879Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:34.4024812Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:34.4026757Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:34.4028779Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:34.4030655Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:34.4032422Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:34.4034492Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:34.4036221Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:34.4037967Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:34.4092985Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:34.4154421Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:34.4156044Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:34.4157412Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:34.4158637Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:34.4159897Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:34.4161207Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-08-20T22:20:34.4162636Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:34.4164260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-08-20T22:20:34.4165901Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-08-20T22:20:34.4167342Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-08-20T22:20:34.4168875Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:34.4170431Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-08-20T22:20:34.4171916Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:34.4173463Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-08-20T22:20:34.4174948Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-08-20T22:20:34.4191551Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-08-20T22:20:34.4336370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-08-20T22:20:34.4337796Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-08-20T22:20:34.4339316Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-08-20T22:20:34.4340995Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-08-20T22:20:34.4342658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-08-20T22:20:34.4344193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-08-20T22:20:34.4345758Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-08-20T22:20:34.4347376Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-08-20T22:20:34.4349007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-08-20T22:20:34.4350853Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-08-20T22:20:34.4352452Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-08-20T22:20:34.4369672Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-08-20T22:20:34.4453503Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-08-20T22:20:34.4454712Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:34.4455783Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:34.4456980Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-08-20T22:20:34.4458095Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-08-20T22:20:34.4459375Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:34.4460635Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-08-20T22:20:34.4464370Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-08-20T22:20:34.4465498Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-08-20T22:20:34.4466653Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-08-20T22:20:34.4588416Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-08-20T22:20:34.4652849Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-08-20T22:20:34.4653895Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-08-20T22:20:34.4654894Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-08-20T22:20:34.4656057Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:34.4662759Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:34.4664039Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:34.4665316Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:34.4666728Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:34.4668040Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:34.4669598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:34.4671137Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:34.4672539Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:34.4673932Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:34.4675550Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:34.4677410Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:34.4679814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:34.4681420Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:34.4683203Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:34.4685157Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:34.4686599Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:34.4688242Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:34.4728243Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:34.4769326Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:34.4771489Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:34.4817235Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:34.4819164Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:34.4821382Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:34.4823401Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:34.4825354Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:34.4827276Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:34.4829158Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:34.4831099Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:34.4833193Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:34.4835011Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:34.4836731Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:34.4838528Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:34.4840259Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:34.4841884Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:34.4843609Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:34.4899527Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:34.4961007Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:34.4962621Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:34.4963984Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:34.4965274Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:34.4966523Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:34.4967847Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-08-20T22:20:34.4969317Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-08-20T22:20:34.4971042Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-08-20T22:20:34.4972649Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-08-20T22:20:34.4974333Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-08-20T22:20:34.4976118Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-08-20T22:20:34.4977683Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-08-20T22:20:34.4979219Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-08-20T22:20:34.4980792Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-08-20T22:20:34.4982376Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-08-20T22:20:34.4998115Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-08-20T22:20:34.5064829Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-08-20T22:20:34.5066487Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:34.5068151Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:34.5069525Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-08-20T22:20:34.5070752Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-08-20T22:20:34.5072005Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:34.5073283Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-08-20T22:20:34.5074897Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-08-20T22:20:34.5076288Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-08-20T22:20:34.5077593Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-08-20T22:20:34.5128514Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-08-20T22:20:34.5129554Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-08-20T22:20:34.5130647Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-08-20T22:20:34.5131968Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-08-20T22:20:34.5139078Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-08-20T22:20:34.5140433Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-08-20T22:20:34.5141843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-08-20T22:20:34.5143358Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-08-20T22:20:34.5144769Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-08-20T22:20:34.5146409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-08-20T22:20:34.5148004Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-08-20T22:20:34.5149533Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-08-20T22:20:34.5151049Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-08-20T22:20:34.5152773Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-08-20T22:20:34.5154488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-08-20T22:20:34.5156390Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-08-20T22:20:34.5158093Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-08-20T22:20:34.5159978Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-08-20T22:20:34.5162228Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-08-20T22:20:34.5163381Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-08-20T22:20:34.5164452Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-08-20T22:20:34.5203740Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-08-20T22:20:34.5244292Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-08-20T22:20:34.5246402Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-08-20T22:20:34.5291884Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-08-20T22:20:34.5293337Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-08-20T22:20:34.5294806Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-08-20T22:20:34.5296323Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-08-20T22:20:34.5297772Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-08-20T22:20:34.5299224Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-08-20T22:20:34.5300652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-08-20T22:20:34.5302081Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-08-20T22:20:34.5303485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-08-20T22:20:34.5304817Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-08-20T22:20:34.5306107Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-08-20T22:20:34.5307370Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-08-20T22:20:34.5308645Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-08-20T22:20:34.5309883Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-08-20T22:20:34.5311140Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-08-20T22:20:34.5371140Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-08-20T22:20:34.5432112Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-08-20T22:20:34.5433555Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-08-20T22:20:34.5434595Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-08-20T22:20:34.5435516Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-08-20T22:20:34.5436476Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-08-20T22:20:34.5437677Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-08-20T22:20:34.5438796Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:34.5440033Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-08-20T22:20:34.5441248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-08-20T22:20:34.5442403Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-08-20T22:20:34.5443547Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:34.5444894Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-08-20T22:20:34.5446086Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:34.5447247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-08-20T22:20:34.5448409Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-08-20T22:20:34.5451247Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-08-20T22:20:34.5577964Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-08-20T22:20:34.5579117Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-08-20T22:20:34.5580286Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-08-20T22:20:34.5581583Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-08-20T22:20:34.5582825Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-08-20T22:20:34.5584003Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-08-20T22:20:34.5585223Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-08-20T22:20:34.5586443Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-08-20T22:20:34.5587668Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-08-20T22:20:34.5588881Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-08-20T22:20:34.5590084Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-08-20T22:20:34.5609691Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-08-20T22:20:34.5666120Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-08-20T22:20:34.5667389Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-08-20T22:20:34.5668512Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-08-20T22:20:34.5669524Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-08-20T22:20:34.5671026Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-08-20T22:20:34.5672108Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-08-20T22:20:34.5673055Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-08-20T22:20:34.5676324Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-08-20T22:20:34.5677446Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-08-20T22:20:34.5678330Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-08-20T22:20:34.5782753Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-08-20T22:20:34.5826766Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-08-20T22:20:34.5827998Z creating: build/lib/ 2024-08-20T22:20:34.5915430Z inflating: build/lib/libprotobuf-lite.a 2024-08-20T22:20:34.6371907Z inflating: build/lib/libprotobuf.a 2024-08-20T22:20:34.6382034Z inflating: build/lib/libpthreadpool.a 2024-08-20T22:20:34.6390402Z inflating: build/lib/libcpuinfo.a 2024-08-20T22:20:34.6398585Z inflating: build/lib/libcpuinfo_internals.a 2024-08-20T22:20:34.6399302Z inflating: build/lib/libclog.a 2024-08-20T22:20:34.6417818Z inflating: build/lib/libnnpack.a 2024-08-20T22:20:34.6420431Z inflating: build/lib/libnnpack_reference_layers.a 2024-08-20T22:20:34.6486151Z inflating: build/lib/libgtest.a 2024-08-20T22:20:34.6561608Z inflating: build/lib/libbenchmark.a 2024-08-20T22:20:34.6625094Z inflating: build/lib/libasmjit.a 2024-08-20T22:20:34.6632752Z inflating: build/lib/libittnotify.a 2024-08-20T22:20:34.6661070Z inflating: build/lib/libtensorpipe_uv.a 2024-08-20T22:20:34.6789325Z inflating: build/lib/libgloo.a 2024-08-20T22:20:34.6809783Z inflating: build/lib/libfmt.a 2024-08-20T22:20:34.6906265Z inflating: build/lib/libc10.so 2024-08-20T22:20:34.6907664Z inflating: build/lib/libcaffe2_nvrtc.so 2024-08-20T22:20:34.6909365Z inflating: build/lib/libtorch_global_deps.so 2024-08-20T22:20:34.6929256Z inflating: build/lib/libpytorch_qnnpack.a 2024-08-20T22:20:34.7433878Z inflating: build/lib/libprotoc.a 2024-08-20T22:20:34.7452163Z inflating: build/lib/libgmock.a 2024-08-20T22:20:34.7452810Z inflating: build/lib/libgtest_main.a 2024-08-20T22:20:34.7453806Z inflating: build/lib/libbenchmark_main.a 2024-08-20T22:20:34.7844260Z inflating: build/lib/libgloo_cuda.a 2024-08-20T22:20:35.8052865Z inflating: build/lib/libdnnl.a 2024-08-20T22:20:35.8626765Z inflating: build/lib/libtensorpipe.a 2024-08-20T22:20:35.8685077Z inflating: build/lib/libc10_cuda.so 2024-08-20T22:20:35.8685600Z inflating: build/lib/libgmock_main.a 2024-08-20T22:20:35.9947474Z inflating: build/lib/libfbgemm.a 2024-08-20T22:20:36.0203863Z inflating: build/lib/libtensorpipe_cuda.a 2024-08-20T22:20:36.0711291Z inflating: build/lib/libkineto.a 2024-08-20T22:20:36.0895701Z inflating: build/lib/libXNNPACK.a 2024-08-20T22:20:36.0938660Z inflating: build/lib/libonnx_proto.a 2024-08-20T22:20:36.1633737Z inflating: build/lib/libonnx.a 2024-08-20T22:20:38.6170340Z inflating: build/lib/libtorch_cpu.so 2024-08-20T22:20:38.6175870Z inflating: build/lib/libunbox_lib.a 2024-08-20T22:20:38.6180088Z inflating: build/lib/libshm.so 2024-08-20T22:20:40.6864940Z inflating: build/lib/libtorch_cuda.so 2024-08-20T22:20:40.6866021Z inflating: build/lib/libtorch.so 2024-08-20T22:20:40.6870308Z inflating: build/lib/libc10d_cuda_test.so 2024-08-20T22:20:41.5064604Z inflating: build/lib/libtorch_cuda_linalg.so 2024-08-20T22:20:41.7072330Z inflating: build/lib/libtorch_python.so 2024-08-20T22:20:41.7145184Z inflating: build/lib/libtorchbind_test.so 2024-08-20T22:20:41.7166178Z inflating: build/lib/libjitbackend_test.so 2024-08-20T22:20:41.7192183Z inflating: build/lib/libaoti_custom_ops.so 2024-08-20T22:20:41.7217714Z inflating: build/lib/libbackend_with_compiler.so 2024-08-20T22:20:41.7252716Z inflating: build/lib/libnnapi_backend.so 2024-08-20T22:20:41.7253187Z creating: build/bin/ 2024-08-20T22:20:41.7304162Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-08-20T22:20:41.7356130Z inflating: build/bin/c10_DeviceGuard_test 2024-08-20T22:20:41.7407383Z inflating: build/bin/c10_Device_test 2024-08-20T22:20:41.7466991Z inflating: build/bin/c10_DispatchKeySet_test 2024-08-20T22:20:41.7520537Z inflating: build/bin/c10_Scalar_test 2024-08-20T22:20:41.7570212Z inflating: build/bin/c10_StreamGuard_test 2024-08-20T22:20:41.7622209Z inflating: build/bin/c10_SymInt_test 2024-08-20T22:20:41.7677457Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-08-20T22:20:41.7733626Z inflating: build/bin/c10_InlineStreamGuard_test 2024-08-20T22:20:41.7790237Z inflating: build/bin/c10_SizesAndStrides_test 2024-08-20T22:20:41.7862205Z inflating: build/bin/c10_cow_test 2024-08-20T22:20:41.7915274Z inflating: build/bin/c10_Bitset_test 2024-08-20T22:20:41.7964835Z inflating: build/bin/c10_ConstexprCrc_test 2024-08-20T22:20:41.8015365Z inflating: build/bin/c10_DeadlockDetection_test 2024-08-20T22:20:41.8066735Z inflating: build/bin/c10_Half_test 2024-08-20T22:20:41.8123075Z inflating: build/bin/c10_LeftRight_test 2024-08-20T22:20:41.8178319Z inflating: build/bin/c10_Metaprogramming_test 2024-08-20T22:20:41.8228885Z inflating: build/bin/c10_Synchronized_test 2024-08-20T22:20:41.8284623Z inflating: build/bin/c10_ThreadLocal_test 2024-08-20T22:20:41.8336635Z inflating: build/bin/c10_TypeIndex_test 2024-08-20T22:20:41.8388022Z inflating: build/bin/c10_TypeList_test 2024-08-20T22:20:41.8437936Z inflating: build/bin/c10_TypeTraits_test 2024-08-20T22:20:41.8491277Z inflating: build/bin/c10_accumulate_test 2024-08-20T22:20:41.8547137Z inflating: build/bin/c10_bfloat16_test 2024-08-20T22:20:41.8597917Z inflating: build/bin/c10_bit_cast_test 2024-08-20T22:20:41.8654967Z inflating: build/bin/c10_complex_math_test 2024-08-20T22:20:41.8710806Z inflating: build/bin/c10_complex_test 2024-08-20T22:20:41.8764432Z inflating: build/bin/c10_exception_test 2024-08-20T22:20:41.8814684Z inflating: build/bin/c10_flags_test 2024-08-20T22:20:41.8865367Z inflating: build/bin/c10_generic_math_test 2024-08-20T22:20:41.9031142Z inflating: build/bin/c10_intrusive_ptr_test 2024-08-20T22:20:41.9082096Z inflating: build/bin/c10_irange_test 2024-08-20T22:20:41.9136068Z inflating: build/bin/c10_lazy_test 2024-08-20T22:20:41.9193485Z inflating: build/bin/c10_logging_test 2024-08-20T22:20:41.9256413Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-08-20T22:20:41.9331837Z inflating: build/bin/c10_optional_test 2024-08-20T22:20:41.9484156Z inflating: build/bin/c10_small_vector_test 2024-08-20T22:20:41.9538558Z inflating: build/bin/c10_registry_test 2024-08-20T22:20:41.9590865Z inflating: build/bin/c10_ssize_test 2024-08-20T22:20:41.9642816Z inflating: build/bin/c10_string_util_test 2024-08-20T22:20:41.9693570Z inflating: build/bin/c10_tempfile_test 2024-08-20T22:20:41.9753070Z inflating: build/bin/c10_string_view_test 2024-08-20T22:20:41.9801651Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-08-20T22:20:41.9858543Z inflating: build/bin/c10_typeid_test 2024-08-20T22:20:42.0306730Z inflating: build/bin/protoc-3.13.0.0 2024-08-20T22:20:42.0757483Z inflating: build/bin/protoc 2024-08-20T22:20:42.0812397Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-08-20T22:20:42.0866187Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-08-20T22:20:42.0920086Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-08-20T22:20:42.0972632Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-08-20T22:20:42.1025131Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-08-20T22:20:42.1078346Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-08-20T22:20:42.1127714Z inflating: build/bin/c10_cuda_CUDATest 2024-08-20T22:20:42.1180921Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-08-20T22:20:42.1514113Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-08-20T22:20:42.1861050Z inflating: build/bin/vec_test_all_types_AVX512 2024-08-20T22:20:42.2222959Z inflating: build/bin/vec_test_all_types_AVX2 2024-08-20T22:20:42.2275696Z inflating: build/bin/BackoffTest 2024-08-20T22:20:42.2329860Z inflating: build/bin/FileStoreTest 2024-08-20T22:20:42.2386339Z inflating: build/bin/TCPStoreTest 2024-08-20T22:20:42.2440271Z inflating: build/bin/HashStoreTest 2024-08-20T22:20:42.2453688Z inflating: build/bin/ProcessGroupMPITest 2024-08-20T22:20:42.2508106Z inflating: build/bin/test_edge_op_registration 2024-08-20T22:20:42.2512426Z inflating: build/bin/torch_shm_manager 2024-08-20T22:20:42.2515139Z inflating: build/bin/example_allreduce 2024-08-20T22:20:42.2570861Z inflating: build/bin/test_dist_autograd 2024-08-20T22:20:42.2640262Z inflating: build/bin/test_cpp_rpc 2024-08-20T22:20:42.2642976Z inflating: build/bin/parallel_benchmark 2024-08-20T22:20:42.2710360Z inflating: build/bin/test_mobile_nnc 2024-08-20T22:20:42.2719971Z inflating: build/bin/aot_model_compiler_test 2024-08-20T22:20:42.3069410Z inflating: build/bin/test_lazy 2024-08-20T22:20:42.4256989Z inflating: build/bin/test_api 2024-08-20T22:20:42.4332360Z inflating: build/bin/Dict_test 2024-08-20T22:20:42.4384477Z inflating: build/bin/Dimname_test 2024-08-20T22:20:42.4450219Z inflating: build/bin/MaybeOwned_test 2024-08-20T22:20:42.4507923Z inflating: build/bin/NamedTensor_test 2024-08-20T22:20:42.4570385Z inflating: build/bin/apply_utils_test 2024-08-20T22:20:42.4629087Z inflating: build/bin/atest 2024-08-20T22:20:42.4692819Z inflating: build/bin/basic 2024-08-20T22:20:42.4748221Z inflating: build/bin/broadcast_test 2024-08-20T22:20:42.4799511Z inflating: build/bin/cpu_allocator_test 2024-08-20T22:20:42.4858213Z inflating: build/bin/cpu_generator_test 2024-08-20T22:20:42.4911742Z inflating: build/bin/cpu_profiling_allocator_test 2024-08-20T22:20:42.5006347Z inflating: build/bin/cpu_rng_test 2024-08-20T22:20:42.5056851Z inflating: build/bin/dispatch_key_set_test 2024-08-20T22:20:42.5107812Z inflating: build/bin/dlconvertor_test 2024-08-20T22:20:42.5166782Z inflating: build/bin/extension_backend_test 2024-08-20T22:20:42.5221643Z inflating: build/bin/half_test 2024-08-20T22:20:42.5318590Z inflating: build/bin/ivalue_test 2024-08-20T22:20:42.5369171Z inflating: build/bin/lazy_tensor_test 2024-08-20T22:20:42.5424680Z inflating: build/bin/math_kernel_test 2024-08-20T22:20:42.5479603Z inflating: build/bin/memory_format_test 2024-08-20T22:20:42.5534199Z inflating: build/bin/memory_overlapping_test 2024-08-20T22:20:42.5587463Z inflating: build/bin/mobile_memory_cleanup 2024-08-20T22:20:42.5644447Z inflating: build/bin/native_test 2024-08-20T22:20:42.5695371Z inflating: build/bin/operator_name_test 2024-08-20T22:20:42.5747512Z inflating: build/bin/operators_test 2024-08-20T22:20:42.5799355Z inflating: build/bin/packedtensoraccessor_test 2024-08-20T22:20:42.5867589Z inflating: build/bin/pow_test 2024-08-20T22:20:42.5925234Z inflating: build/bin/quantized_test 2024-08-20T22:20:42.5975924Z inflating: build/bin/reduce_ops_test 2024-08-20T22:20:42.6027390Z inflating: build/bin/reportMemoryUsage_test 2024-08-20T22:20:42.6084741Z inflating: build/bin/scalar_tensor_test 2024-08-20T22:20:42.6143836Z inflating: build/bin/scalar_test 2024-08-20T22:20:42.6195782Z inflating: build/bin/StorageUtils_test 2024-08-20T22:20:42.6248965Z inflating: build/bin/stride_properties_test 2024-08-20T22:20:42.6328611Z inflating: build/bin/tensor_iterator_test 2024-08-20T22:20:42.6383572Z inflating: build/bin/test_parallel 2024-08-20T22:20:42.6387024Z inflating: build/bin/thread_init_test 2024-08-20T22:20:42.6443063Z inflating: build/bin/type_ptr_test 2024-08-20T22:20:42.6506432Z inflating: build/bin/type_test 2024-08-20T22:20:42.6559713Z inflating: build/bin/undefined_tensor_test 2024-08-20T22:20:42.6561024Z inflating: build/bin/verify_api_visibility 2024-08-20T22:20:42.6630968Z inflating: build/bin/legacy_vmap_test 2024-08-20T22:20:42.6682641Z inflating: build/bin/weakref_test 2024-08-20T22:20:42.6734713Z inflating: build/bin/wrapdim_test 2024-08-20T22:20:42.6787121Z inflating: build/bin/xla_tensor_test 2024-08-20T22:20:42.6847903Z inflating: build/bin/IListRef_test 2024-08-20T22:20:42.6954755Z inflating: build/bin/List_test 2024-08-20T22:20:42.7021890Z inflating: build/bin/KernelFunction_test 2024-08-20T22:20:42.7144725Z inflating: build/bin/kernel_function_legacy_test 2024-08-20T22:20:42.7243173Z inflating: build/bin/kernel_function_test 2024-08-20T22:20:42.7372929Z inflating: build/bin/kernel_lambda_legacy_test 2024-08-20T22:20:42.7476959Z inflating: build/bin/kernel_lambda_test 2024-08-20T22:20:42.7538831Z inflating: build/bin/kernel_stackbased_test 2024-08-20T22:20:42.7635918Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-08-20T22:20:42.7687260Z inflating: build/bin/CppSignature_test 2024-08-20T22:20:42.7743704Z inflating: build/bin/backend_fallback_test 2024-08-20T22:20:42.7792830Z inflating: build/bin/op_allowlist_test 2024-08-20T22:20:42.8100140Z inflating: build/bin/op_registration_test 2024-08-20T22:20:42.8164242Z inflating: build/bin/inline_container_test 2024-08-20T22:20:42.8217577Z inflating: build/bin/cuda_apply_test 2024-08-20T22:20:42.8271553Z inflating: build/bin/cuda_allocator_test 2024-08-20T22:20:42.8330420Z inflating: build/bin/cuda_atomic_ops_test 2024-08-20T22:20:42.8386432Z inflating: build/bin/cuda_caching_host_allocator_test 2024-08-20T22:20:42.8456450Z inflating: build/bin/cuda_complex_math_test 2024-08-20T22:20:42.8516096Z inflating: build/bin/cuda_complex_test 2024-08-20T22:20:42.8566368Z inflating: build/bin/cuda_device_test 2024-08-20T22:20:42.8623983Z inflating: build/bin/cuda_cub_test 2024-08-20T22:20:42.8675370Z inflating: build/bin/cuda_dlconvertor_test 2024-08-20T22:20:42.8740599Z inflating: build/bin/cuda_distributions_test 2024-08-20T22:20:42.8797690Z inflating: build/bin/cuda_generator_test 2024-08-20T22:20:42.8847826Z inflating: build/bin/cuda_half_test 2024-08-20T22:20:42.8899393Z inflating: build/bin/cuda_integer_divider_test 2024-08-20T22:20:42.8949105Z inflating: build/bin/cuda_optional_test 2024-08-20T22:20:42.9001212Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-08-20T22:20:42.9054611Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-08-20T22:20:42.9104604Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-08-20T22:20:42.9166747Z inflating: build/bin/cuda_stream_test 2024-08-20T22:20:42.9215992Z inflating: build/bin/cuda_cudnn_test 2024-08-20T22:20:42.9268495Z inflating: build/bin/cuda_vectorized_test 2024-08-20T22:20:42.9283089Z inflating: build/bin/tutorial_tensorexpr 2024-08-20T22:20:42.9349930Z inflating: build/bin/ProcessGroupGlooTest 2024-08-20T22:20:42.9407672Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-08-20T22:20:42.9471800Z inflating: build/bin/ProcessGroupNCCLTest 2024-08-20T22:20:42.9534533Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-08-20T22:20:43.0369803Z inflating: build/bin/test_tensorexpr 2024-08-20T22:20:43.0953341Z inflating: build/bin/test_jit 2024-08-20T22:20:43.0954441Z creating: .additional_ci_files/ 2024-08-20T22:20:43.1014674Z inflating: .additional_ci_files/test-times.json 2024-08-20T22:20:43.1255364Z inflating: .additional_ci_files/test-class-times.json 2024-08-20T22:20:43.1291913Z ##[group]Run rm artifacts.zip 2024-08-20T22:20:43.1292300Z rm artifacts.zip 2024-08-20T22:20:43.1301388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:43.1301884Z env: 2024-08-20T22:20:43.1302341Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.1302791Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.1303269Z ##[endgroup] 2024-08-20T22:20:43.2414732Z ##[group]Run df -H 2024-08-20T22:20:43.2415467Z df -H 2024-08-20T22:20:43.2424451Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:43.2424948Z env: 2024-08-20T22:20:43.2425220Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.2425669Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.2426148Z ##[endgroup] 2024-08-20T22:20:43.2499573Z Filesystem Size Used Avail Use% Mounted on 2024-08-20T22:20:43.2500159Z devtmpfs 4.2M 0 4.2M 0% /dev 2024-08-20T22:20:43.2500697Z tmpfs 34G 0 34G 0% /dev/shm 2024-08-20T22:20:43.2501164Z tmpfs 14G 553k 14G 1% /run 2024-08-20T22:20:43.2501604Z /dev/nvme0n1p1 161G 53G 109G 33% / 2024-08-20T22:20:43.2502390Z tmpfs 34G 21k 34G 1% /tmp 2024-08-20T22:20:43.2502891Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2024-08-20T22:20:43.2503721Z tmpfs 6.7G 0 6.7G 0% /run/user/0 2024-08-20T22:20:43.2542968Z Prepare all required actions 2024-08-20T22:20:43.2543475Z Getting action download info 2024-08-20T22:20:43.3689936Z ##[group]Run ./.github/actions/download-td-artifacts 2024-08-20T22:20:43.3690388Z with: 2024-08-20T22:20:43.3690650Z env: 2024-08-20T22:20:43.3690928Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.3691375Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.3691854Z ##[endgroup] 2024-08-20T22:20:43.3724442Z ##[group]Run seemethere/download-artifact-s3@v4 2024-08-20T22:20:43.3724878Z with: 2024-08-20T22:20:43.3725215Z name: td_results 2024-08-20T22:20:43.3725526Z s3-bucket: gha-artifacts 2024-08-20T22:20:43.3725867Z region: us-east-1 2024-08-20T22:20:43.3726157Z env: 2024-08-20T22:20:43.3726431Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.3726865Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.3727353Z ##[endgroup] 2024-08-20T22:20:43.8458204Z (node:233757) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-08-20T22:20:43.8458881Z 2024-08-20T22:20:43.8459142Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-08-20T22:20:43.8459837Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-08-20T22:20:43.8460698Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-08-20T22:20:43.9064344Z Found 0 objects with prefix pytorch/pytorch/10479310725/td_results/ 2024-08-20T22:20:43.9072419Z Artifact download has finished successfully 2024-08-20T22:20:43.9391921Z ##[group]Run mkdir -p .additional_ci_files 2024-08-20T22:20:43.9392403Z mkdir -p .additional_ci_files 2024-08-20T22:20:43.9392961Z mv td_results.json .additional_ci_files/td_results.json 2024-08-20T22:20:43.9402916Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:43.9403469Z env: 2024-08-20T22:20:43.9403751Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.9404212Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.9404693Z ##[endgroup] 2024-08-20T22:20:43.9471915Z mv: cannot stat 'td_results.json': No such file or directory 2024-08-20T22:20:43.9483967Z ##[error]Process completed with exit code 1. 2024-08-20T22:20:43.9518501Z ##[group]Run .github/scripts/parse_ref.py 2024-08-20T22:20:43.9519053Z .github/scripts/parse_ref.py 2024-08-20T22:20:43.9528346Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:43.9528708Z env: 2024-08-20T22:20:43.9528992Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.9529452Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.9529936Z ##[endgroup] 2024-08-20T22:20:43.9817975Z Prepare all required actions 2024-08-20T22:20:43.9856645Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-08-20T22:20:43.9857103Z with: 2024-08-20T22:20:43.9857634Z github-token: *** 2024-08-20T22:20:43.9858176Z env: 2024-08-20T22:20:43.9858454Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.9858911Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.9859396Z ##[endgroup] 2024-08-20T22:20:43.9879029Z ##[group]Run set -eux 2024-08-20T22:20:43.9879362Z set -eux 2024-08-20T22:20:43.9879942Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-08-20T22:20:43.9889139Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:43.9889629Z env: 2024-08-20T22:20:43.9889904Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:43.9890348Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:43.9890976Z GITHUB_TOKEN: *** 2024-08-20T22:20:43.9891279Z ##[endgroup] 2024-08-20T22:20:43.9922426Z + python3 .github/scripts/get_workflow_job_id.py 10479310725 i-0ed5a1a6923b38c4c 2024-08-20T22:20:45.6874945Z setting job-id=29026527906 2024-08-20T22:20:45.6875837Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:45.7116362Z Prepare all required actions 2024-08-20T22:20:45.7116800Z Getting action download info 2024-08-20T22:20:45.8186099Z ##[group]Run ./.github/actions/filter-test-configs 2024-08-20T22:20:45.8186554Z with: 2024-08-20T22:20:45.8187004Z github-token: *** 2024-08-20T22:20:45.8195724Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]} 2024-08-20T22:20:45.8205172Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:45.8205881Z env: 2024-08-20T22:20:45.8206163Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.8206608Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.8207082Z ##[endgroup] 2024-08-20T22:20:45.8249153Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-08-20T22:20:45.8249693Z with: 2024-08-20T22:20:45.8250128Z shell: bash 2024-08-20T22:20:45.8250427Z timeout_minutes: 10 2024-08-20T22:20:45.8250751Z max_attempts: 5 2024-08-20T22:20:45.8251069Z retry_wait_seconds: 30 2024-08-20T22:20:45.8252146Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:45.8253280Z polling_interval_seconds: 1 2024-08-20T22:20:45.8253649Z warning_on_retry: true 2024-08-20T22:20:45.8253985Z continue_on_error: false 2024-08-20T22:20:45.8254318Z env: 2024-08-20T22:20:45.8254589Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:45.8255022Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:45.8255644Z GITHUB_TOKEN: *** 2024-08-20T22:20:45.8255963Z ##[endgroup] 2024-08-20T22:20:45.9012029Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-08-20T22:20:46.1400489Z Defaulting to user installation because normal site-packages is not writeable 2024-08-20T22:20:46.1577368Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1) 2024-08-20T22:20:46.1580803Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1) 2024-08-20T22:20:46.1699530Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2024-08-20T22:20:46.1703559Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2024.7.4) 2024-08-20T22:20:46.1707262Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2024-08-20T22:20:46.1717227Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12) 2024-08-20T22:20:46.8802214Z Command completed after 1 attempt(s). 2024-08-20T22:20:46.8860720Z ##[group]Run set -x 2024-08-20T22:20:46.8861070Z set -x 2024-08-20T22:20:46.8861386Z  2024-08-20T22:20:46.8861921Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:46.8862582Z # in runner workspace 2024-08-20T22:20:46.8863105Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-08-20T22:20:46.8873053Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:46.8873549Z env: 2024-08-20T22:20:46.8873819Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.8874275Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.8874754Z ##[endgroup] 2024-08-20T22:20:46.8904598Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-08-20T22:20:46.9147692Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:46.9148225Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-08-20T22:20:46.9148701Z echo "Job name: ${JOB_NAME}" 2024-08-20T22:20:46.9149101Z  2024-08-20T22:20:46.9149635Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-08-20T22:20:46.9150294Z # in runner workspace 2024-08-20T22:20:46.9150868Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-08-20T22:20:46.9151484Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-08-20T22:20:46.9151927Z  --job-name "${JOB_NAME}" \ 2024-08-20T22:20:46.9160913Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-08-20T22:20:46.9170222Z  --selected-test-configs "" \ 2024-08-20T22:20:46.9170676Z  --pr-number "${PR_NUMBER}" \ 2024-08-20T22:20:46.9171089Z  --tag "${TAG}" \ 2024-08-20T22:20:46.9171471Z  --event-name "${EVENT_NAME}" \ 2024-08-20T22:20:46.9171900Z  --schedule "${SCHEDULE}" \ 2024-08-20T22:20:46.9172338Z  --branch "${HEAD_BRANCH}" 2024-08-20T22:20:46.9181131Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:46.9181625Z env: 2024-08-20T22:20:46.9181892Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:46.9182339Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:46.9182999Z GITHUB_TOKEN: *** 2024-08-20T22:20:46.9183647Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:46.9184364Z PR_NUMBER: 2024-08-20T22:20:46.9184670Z TAG: ciflow/inductor/133712 2024-08-20T22:20:46.9185041Z EVENT_NAME: push 2024-08-20T22:20:46.9185356Z SCHEDULE: 2024-08-20T22:20:46.9185639Z HEAD_BRANCH: 2024-08-20T22:20:46.9185939Z ##[endgroup] 2024-08-20T22:20:46.9215322Z Workflow: inductor 2024-08-20T22:20:46.9216264Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.2152992Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-08-20T22:20:47.3635615Z ##[group]Run echo "Filtered matrix:" 2024-08-20T22:20:47.3636064Z echo "Filtered matrix:" 2024-08-20T22:20:47.3644912Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "amz2023.linux.g5.4xlarge.nvidia.gpu"}]}" 2024-08-20T22:20:47.3654055Z  2024-08-20T22:20:47.3654337Z echo 2024-08-20T22:20:47.3654717Z echo "Is the current job unstable? False" 2024-08-20T22:20:47.3655161Z  2024-08-20T22:20:47.3655436Z echo 2024-08-20T22:20:47.3655792Z echo "Is keep-going label set? False" 2024-08-20T22:20:47.3656227Z  2024-08-20T22:20:47.3656489Z echo 2024-08-20T22:20:47.3656802Z echo "Renabled issues? " 2024-08-20T22:20:47.3665771Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:47.3666267Z env: 2024-08-20T22:20:47.3666543Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.3666992Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.3667467Z ##[endgroup] 2024-08-20T22:20:47.3696840Z Filtered matrix: 2024-08-20T22:20:47.3706906Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: amz2023.linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: amz2023.linux.g5.4xlarge.nvidia.gpu}]} 2024-08-20T22:20:47.3717596Z 2024-08-20T22:20:47.3717753Z Is the current job unstable? False 2024-08-20T22:20:47.3718064Z 2024-08-20T22:20:47.3718371Z Is keep-going label set? False 2024-08-20T22:20:47.3718624Z 2024-08-20T22:20:47.3718753Z Renabled issues? 2024-08-20T22:20:47.3761368Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:47.3762070Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-08-20T22:20:47.3770386Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T22:20:47.3770877Z env: 2024-08-20T22:20:47.3771164Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.3771619Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.3772101Z JOB_TIMEOUT: 240 2024-08-20T22:20:47.3772404Z ##[endgroup] 2024-08-20T22:20:47.3855459Z ##[group]Run set -x 2024-08-20T22:20:47.3855861Z set -x 2024-08-20T22:20:47.3856157Z  2024-08-20T22:20:47.3856508Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-08-20T22:20:47.3857044Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-08-20T22:20:47.3857594Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-08-20T22:20:47.3858101Z  TEST_COMMAND=.ci/onnx/test.sh 2024-08-20T22:20:47.3858509Z else 2024-08-20T22:20:47.3858845Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:47.3859265Z fi 2024-08-20T22:20:47.3859541Z  2024-08-20T22:20:47.3860008Z # detached container should get cleaned up by teardown_ec2_linux 2024-08-20T22:20:47.3860752Z # TODO: Stop building test binaries as part of the build phase 2024-08-20T22:20:47.3861422Z # Used for GPU_FLAG since that doesn't play nice 2024-08-20T22:20:47.3861993Z # shellcheck disable=SC2086,SC2090 2024-08-20T22:20:47.3862449Z container_name=$(docker run \ 2024-08-20T22:20:47.3862865Z  ${GPU_FLAG:-} \ 2024-08-20T22:20:47.3863231Z  -e BUILD_ENVIRONMENT \ 2024-08-20T22:20:47.3863624Z  -e PR_NUMBER \ 2024-08-20T22:20:47.3863981Z  -e GITHUB_ACTIONS \ 2024-08-20T22:20:47.3864374Z  -e GITHUB_REPOSITORY \ 2024-08-20T22:20:47.3864772Z  -e GITHUB_WORKFLOW \ 2024-08-20T22:20:47.3865160Z  -e GITHUB_JOB \ 2024-08-20T22:20:47.3865522Z  -e GITHUB_RUN_ID \ 2024-08-20T22:20:47.3865918Z  -e GITHUB_RUN_NUMBER \ 2024-08-20T22:20:47.3866329Z  -e GITHUB_RUN_ATTEMPT \ 2024-08-20T22:20:47.3866730Z  -e JOB_ID \ 2024-08-20T22:20:47.3867075Z  -e JOB_NAME \ 2024-08-20T22:20:47.3867427Z  -e BASE_SHA \ 2024-08-20T22:20:47.3867781Z  -e BRANCH \ 2024-08-20T22:20:47.3868108Z  -e SHA1 \ 2024-08-20T22:20:47.3868456Z  -e AWS_DEFAULT_REGION \ 2024-08-20T22:20:47.3868867Z  -e IN_WHEEL_TEST \ 2024-08-20T22:20:47.3869260Z  -e SHARD_NUMBER \ 2024-08-20T22:20:47.3869624Z  -e TEST_CONFIG \ 2024-08-20T22:20:47.3869998Z  -e NUM_TEST_SHARDS \ 2024-08-20T22:20:47.3870398Z  -e REENABLED_ISSUES \ 2024-08-20T22:20:47.3870814Z  -e CONTINUE_THROUGH_ERROR \ 2024-08-20T22:20:47.3871234Z  -e VERBOSE_TEST_LOGS \ 2024-08-20T22:20:47.3871641Z  -e TEST_SHOWLOCALS \ 2024-08-20T22:20:47.3872033Z  -e NO_TEST_TIMEOUT \ 2024-08-20T22:20:47.3872416Z  -e NO_TD \ 2024-08-20T22:20:47.3872752Z  -e TD_DISTRIBUTED \ 2024-08-20T22:20:47.3873138Z  -e PR_LABELS \ 2024-08-20T22:20:47.3873550Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-08-20T22:20:47.3874004Z  -e SCCACHE_BUCKET \ 2024-08-20T22:20:47.3874411Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-08-20T22:20:47.3874827Z  -e XLA_CUDA \ 2024-08-20T22:20:47.3875244Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-08-20T22:20:47.3875919Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-08-20T22:20:47.3876448Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-08-20T22:20:47.3876966Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-08-20T22:20:47.3877437Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-08-20T22:20:47.3877888Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2024-08-20T22:20:47.3878334Z  -e DASHBOARD_TAG \ 2024-08-20T22:20:47.3878814Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-08-20T22:20:47.3879377Z  --security-opt seccomp=unconfined \ 2024-08-20T22:20:47.3879834Z  --cap-add=SYS_PTRACE \ 2024-08-20T22:20:47.3880230Z  --ipc=host \ 2024-08-20T22:20:47.3880597Z  --shm-size="${SHM_SIZE}" \ 2024-08-20T22:20:47.3881005Z  --tty \ 2024-08-20T22:20:47.3881321Z  --detach \ 2024-08-20T22:20:47.3881806Z  --name="${container_name}" \ 2024-08-20T22:20:47.3882224Z  --user jenkins \ 2024-08-20T22:20:47.3882717Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-08-20T22:20:47.3883271Z  -w /var/lib/jenkins/workspace \ 2024-08-20T22:20:47.3883726Z  "${DOCKER_IMAGE}" 2024-08-20T22:20:47.3884094Z ) 2024-08-20T22:20:47.3884519Z # Propagate download.pytorch.org IP to container 2024-08-20T22:20:47.3885489Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-08-20T22:20:47.3886451Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-08-20T22:20:47.3887347Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-08-20T22:20:47.3895776Z shell: /usr/bin/bash -e {0} 2024-08-20T22:20:47.3896131Z env: 2024-08-20T22:20:47.3896427Z GIT_DEFAULT_BRANCH: main 2024-08-20T22:20:47.3896910Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:20:47.3897523Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:20:47.3898026Z PR_NUMBER: 2024-08-20T22:20:47.3898362Z GITHUB_REPOSITORY: pytorch/pytorch 2024-08-20T22:20:47.3898796Z GITHUB_WORKFLOW: inductor 2024-08-20T22:20:47.3899178Z GITHUB_JOB: test 2024-08-20T22:20:47.3899507Z GITHUB_RUN_ID: 10479310725 2024-08-20T22:20:47.3899885Z GITHUB_RUN_NUMBER: 88082 2024-08-20T22:20:47.3900249Z GITHUB_RUN_ATTEMPT: 1 2024-08-20T22:20:47.3900597Z JOB_ID: 29026527906 2024-08-20T22:20:47.3901249Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:20:47.3901983Z BRANCH: 2024-08-20T22:20:47.3902331Z SHA1: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:47.3902868Z BASE_SHA: 40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:20:47.3903336Z TEST_CONFIG: inductor_timm 2024-08-20T22:20:47.3903701Z SHARD_NUMBER: 1 2024-08-20T22:20:47.3904031Z NUM_TEST_SHARDS: 2 2024-08-20T22:20:47.3904360Z REENABLED_ISSUES: 2024-08-20T22:20:47.3904703Z CONTINUE_THROUGH_ERROR: False 2024-08-20T22:20:47.3905111Z VERBOSE_TEST_LOGS: False 2024-08-20T22:20:47.3905474Z TEST_SHOWLOCALS: False 2024-08-20T22:20:47.3905825Z NO_TEST_TIMEOUT: False 2024-08-20T22:20:47.3906173Z NO_TD: False 2024-08-20T22:20:47.3906485Z TD_DISTRIBUTED: False 2024-08-20T22:20:47.3906918Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-08-20T22:20:47.3907414Z SCCACHE_S3_KEY_PREFIX: inductor 2024-08-20T22:20:47.3907813Z SHM_SIZE: 2g 2024-08-20T22:20:47.3908890Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:47.3910037Z XLA_CUDA: 2024-08-20T22:20:47.3910515Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:20:47.3911128Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-08-20T22:20:47.3911575Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-08-20T22:20:47.3912092Z DASHBOARD_TAG: 2024-08-20T22:20:47.3912600Z HUGGING_FACE_HUB_TOKEN: *** 2024-08-20T22:20:47.3912992Z SCRIBE_GRAPHQL_ACCESS_TOKEN: 2024-08-20T22:20:47.3913378Z ##[endgroup] 2024-08-20T22:20:47.3941110Z + [[ inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-08-20T22:20:47.3941829Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-08-20T22:20:47.3942337Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-08-20T22:20:47.3950471Z +++ nproc --ignore=2 2024-08-20T22:20:47.3976969Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_10479310725 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T22:20:58.8174914Z + container_name=a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T22:20:58.8176345Z + grep download.pytorch.org /etc/hosts 2024-08-20T22:20:58.8179521Z + docker exec -i a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 sudo bash -c '/bin/cat >> /etc/hosts' 2024-08-20T22:20:58.9798545Z + echo DOCKER_CONTAINER_ID=a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T22:20:58.9802919Z ++ echo dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl 2024-08-20T22:20:58.9806339Z + docker exec -t a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 sh -c 'pip install dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-08-20T22:20:59.3790871Z Processing ./dist/torch-2.5.0a0+git40ec5f6-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+git40ec5f6) 2024-08-20T22:20:59.7055576Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.13.1) 2024-08-20T22:20:59.7058493Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (4.12.2) 2024-08-20T22:20:59.7060680Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.8.8) 2024-08-20T22:20:59.7062761Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.1.4) 2024-08-20T22:20:59.7065287Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2024.6.1) 2024-08-20T22:20:59.7070533Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.13.1) 2024-08-20T22:20:59.7086679Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.3.0) 2024-08-20T22:20:59.7099974Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (3.3.0) 2024-08-20T22:20:59.7117151Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (1.22.4) 2024-08-20T22:20:59.7506085Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+git40ec5f6->torch==2.5.0a0+git40ec5f6) (2.1.5) 2024-08-20T22:21:00.3285438Z Installing collected packages: torch 2024-08-20T22:21:10.0988950Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-08-20T22:21:10.0990798Z timm 1.0.9.dev0 requires torchvision, which is not installed. 2024-08-20T22:21:10.0992095Z Successfully installed torch-2.5.0a0+git40ec5f6 2024-08-20T22:21:10.1787997Z + export TERM=vt100 2024-08-20T22:21:10.1788476Z + TERM=vt100 2024-08-20T22:21:10.1790229Z ++ dirname .ci/pytorch/test.sh 2024-08-20T22:21:10.1801841Z + source .ci/pytorch/common.sh 2024-08-20T22:21:10.1804739Z +++ dirname .ci/pytorch/common.sh 2024-08-20T22:21:10.1816064Z ++ source .ci/pytorch/common_utils.sh 2024-08-20T22:21:10.1818307Z +++ declare -f -t trap_add 2024-08-20T22:21:10.1824608Z ++ set -ex 2024-08-20T22:21:10.1825253Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:10.1826989Z ++ BUILD_TEST_LIBTORCH=0 2024-08-20T22:21:10.1827781Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-08-20T22:21:10.1830557Z ++ stat -c %u /var/lib/jenkins/workspace 2024-08-20T22:21:10.1851266Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-08-20T22:21:10.1851868Z + trap_add cleanup_workspace EXIT 2024-08-20T22:21:10.1852442Z + trap_add_cmd=cleanup_workspace 2024-08-20T22:21:10.1852906Z + shift 2024-08-20T22:21:10.1853249Z + for trap_add_name in "$@" 2024-08-20T22:21:10.1860435Z +++ trap -p EXIT 2024-08-20T22:21:10.1863151Z ++ eval 'extract_trap_cmd ' 2024-08-20T22:21:10.1863670Z +++ extract_trap_cmd 2024-08-20T22:21:10.1864150Z +++ printf '%s\n' '' 2024-08-20T22:21:10.1864564Z ++ printf '%s\n' cleanup_workspace 2024-08-20T22:21:10.1867020Z + trap -- ' 2024-08-20T22:21:10.1867480Z cleanup_workspace' EXIT 2024-08-20T22:21:10.1868079Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-08-20T22:21:10.9033374Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-08-20T22:21:10.9055707Z + echo 'Environment variables:' 2024-08-20T22:21:10.9056218Z Environment variables: 2024-08-20T22:21:10.9056549Z + env 2024-08-20T22:21:10.9065859Z INSTALLED_DB=yes 2024-08-20T22:21:10.9066455Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:10.9066917Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:10.9067465Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:10.9068311Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:10.9071148Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:10.9071860Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:10.9072465Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:10.9073063Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:10.9073719Z HOSTNAME=a553b99c8e72 2024-08-20T22:21:10.9074719Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9075749Z GITHUB_ACTION=__self 2024-08-20T22:21:10.9076195Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:10.9080285Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:10.9084409Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:10.9085030Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:10.9085381Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:10.9085720Z TEST_CONFIG=inductor_timm 2024-08-20T22:21:10.9086078Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:10.9086554Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:10.9087015Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:10.9087449Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:10.9087895Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:10.9088445Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:10.9088898Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:10.9089392Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:10.9089840Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:10.9090163Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:10.9090536Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:10.9090925Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9091357Z XLA_CUDA= 2024-08-20T22:21:10.9091845Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:10.9094104Z *** 2024-08-20T22:21:10.9094400Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:10.9094791Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:10.9095153Z GITHUB_ACTIONS=true 2024-08-20T22:21:10.9095488Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:10.9096009Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:10.9096541Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:10.9097007Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9097529Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:10.9098021Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9098798Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:10.9099495Z UCC_HOME=/usr 2024-08-20T22:21:10.9099832Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:10.9100276Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:10.9100612Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:10.9101074Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:10.9101568Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:10.9102033Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:10.9102403Z SHARD_NUMBER=1 2024-08-20T22:21:10.9102710Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:10.9103055Z HOME=/var/lib/jenkins 2024-08-20T22:21:10.9103413Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:10.9103847Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:10.9104323Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:10.9104801Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:10.9105165Z CUDA_VERSION=12.1.1 2024-08-20T22:21:10.9105592Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:10.9106025Z NUM_TEST_SHARDS=2 2024-08-20T22:21:10.9106326Z UCX_HOME=/usr 2024-08-20T22:21:10.9106833Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:10.9107912Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9109138Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:10.9110394Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9111491Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:10.9112132Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:10.9112467Z DASHBOARD_TAG= 2024-08-20T22:21:10.9112770Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:10.9113340Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:10.9113869Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:10.9114854Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9115748Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:10.9116141Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:10.9116511Z PR_NUMBER= 2024-08-20T22:21:10.9116788Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:10.9117117Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:10.9117549Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:10.9118000Z TERM=vt100 2024-08-20T22:21:10.9118332Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:10.9118735Z INSTALLED_VISION=yes 2024-08-20T22:21:10.9119041Z BRANCH= 2024-08-20T22:21:10.9119322Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:10.9119815Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:10.9120289Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:10.9121037Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:10.9121766Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:10.9122234Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:10.9122690Z REENABLED_ISSUES= 2024-08-20T22:21:10.9122982Z SHLVL=1 2024-08-20T22:21:10.9123237Z MAX_JOBS=14 2024-08-20T22:21:10.9123539Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:10.9123887Z NVARCH=x86_64 2024-08-20T22:21:10.9124178Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:10.9124624Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9125265Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:10.9125759Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:10.9126414Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:10.9126965Z GITHUB_JOB=test 2024-08-20T22:21:10.9127369Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:10.9127920Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:10.9128424Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:10.9128742Z TD_DISTRIBUTED=False 2024-08-20T22:21:10.9129113Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:10.9129540Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:10.9129967Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:10.9130351Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:10.9130694Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:10.9131035Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:10.9132045Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:10.9133442Z GITHUB_BASE_REF= 2024-08-20T22:21:10.9133768Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:10.9134137Z CI=true 2024-08-20T22:21:10.9134454Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:10.9134872Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:10.9135242Z JOB_ID=29026527906 2024-08-20T22:21:10.9135548Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:10.9135866Z GITHUB_HEAD_REF= 2024-08-20T22:21:10.9136165Z GITHUB_ACTION_REF= 2024-08-20T22:21:10.9136598Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:10.9137051Z TEST_SHOWLOCALS=False 2024-08-20T22:21:10.9137369Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:10.9137718Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:10.9138614Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9139412Z NO_TD=False 2024-08-20T22:21:10.9139705Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:10.9140057Z _=/usr/bin/env 2024-08-20T22:21:10.9140528Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-08-20T22:21:10.9307813Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-08-20T22:21:10.9309065Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-08-20T22:21:10.9310193Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-08-20T22:21:10.9311047Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-08-20T22:21:10.9311637Z + BUILD_DIR=build 2024-08-20T22:21:10.9311963Z + BUILD_RENAMED_DIR=build_renamed 2024-08-20T22:21:10.9312339Z + BUILD_BIN_DIR=build/bin 2024-08-20T22:21:10.9312675Z + SHARD_NUMBER=1 2024-08-20T22:21:10.9312975Z + NUM_TEST_SHARDS=2 2024-08-20T22:21:10.9313370Z + export VALGRIND=ON 2024-08-20T22:21:10.9313684Z + VALGRIND=ON 2024-08-20T22:21:10.9314225Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-08-20T22:21:10.9314717Z + [[ 0 == \1 ]] 2024-08-20T22:21:10.9315008Z + [[ False == \1 ]] 2024-08-20T22:21:10.9315486Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-08-20T22:21:10.9316006Z ++ realpath build/custom_test_artifacts 2024-08-20T22:21:10.9325789Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-08-20T22:21:10.9326501Z + [[ -n '' ]] 2024-08-20T22:21:10.9326850Z + echo 'Environment variables' 2024-08-20T22:21:10.9327223Z Environment variables 2024-08-20T22:21:10.9327537Z + env 2024-08-20T22:21:10.9335253Z INSTALLED_DB=yes 2024-08-20T22:21:10.9335794Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-08-20T22:21:10.9336287Z NVIDIA_VISIBLE_DEVICES=all 2024-08-20T22:21:10.9336779Z NV_NVML_DEV_VERSION=12.1.105-1 2024-08-20T22:21:10.9337531Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T22:21:10.9338279Z CONTINUE_THROUGH_ERROR=False 2024-08-20T22:21:10.9338965Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-08-20T22:21:10.9339636Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:10.9340411Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-08-20T22:21:10.9341037Z HOSTNAME=a553b99c8e72 2024-08-20T22:21:10.9341939Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9342744Z GITHUB_ACTION=__self 2024-08-20T22:21:10.9343085Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-08-20T22:21:10.9347587Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-08-20T22:21:10.9351890Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-08-20T22:21:10.9352396Z NV_NVTX_VERSION=12.1.105-1 2024-08-20T22:21:10.9352744Z GITHUB_RUN_NUMBER=88082 2024-08-20T22:21:10.9353091Z TEST_CONFIG=inductor_timm 2024-08-20T22:21:10.9353452Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-08-20T22:21:10.9353918Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-08-20T22:21:10.9354373Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-08-20T22:21:10.9354813Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-08-20T22:21:10.9355208Z SCRIBE_GRAPHQL_ACCESS_TOKEN= 2024-08-20T22:21:10.9355597Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-08-20T22:21:10.9356041Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:10.9356528Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-08-20T22:21:10.9356979Z GITHUB_REF_TYPE=tag 2024-08-20T22:21:10.9357301Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-08-20T22:21:10.9357676Z NCCL_VERSION=2.17.1-1 2024-08-20T22:21:10.9358063Z BASE_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9358493Z XLA_CUDA= 2024-08-20T22:21:10.9358953Z HUGGING_FACE_HUB_TOKEN=*** 2024-08-20T22:21:10.9359539Z *** 2024-08-20T22:21:10.9359821Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-08-20T22:21:10.9360231Z GITHUB_REPOSITORY_ID=65600975 2024-08-20T22:21:10.9360624Z GITHUB_ACTIONS=true 2024-08-20T22:21:10.9360959Z NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T22:21:10.9361456Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-08-20T22:21:10.9361987Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-08-20T22:21:10.9362455Z SHA1=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9362969Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-08-20T22:21:10.9363445Z GITHUB_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9364227Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/133712 2024-08-20T22:21:10.9365094Z UCC_HOME=/usr 2024-08-20T22:21:10.9365442Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-08-20T22:21:10.9365839Z VERBOSE_TEST_LOGS=False 2024-08-20T22:21:10.9366177Z NVIDIA_PRODUCT_NAME=CUDA 2024-08-20T22:21:10.9366780Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-08-20T22:21:10.9367283Z GITHUB_REF=refs/tags/ciflow/inductor/133712 2024-08-20T22:21:10.9367758Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-08-20T22:21:10.9368131Z SHARD_NUMBER=1 2024-08-20T22:21:10.9368441Z GITHUB_REF_PROTECTED=false 2024-08-20T22:21:10.9368788Z HOME=/var/lib/jenkins 2024-08-20T22:21:10.9369156Z GITHUB_API_URL=https://api.github.com 2024-08-20T22:21:10.9369583Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-08-20T22:21:10.9370062Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-08-20T22:21:10.9370539Z SCCACHE_S3_KEY_PREFIX=inductor 2024-08-20T22:21:10.9370900Z CUDA_VERSION=12.1.1 2024-08-20T22:21:10.9371331Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-08-20T22:21:10.9371766Z NUM_TEST_SHARDS=2 2024-08-20T22:21:10.9372064Z UCX_HOME=/usr 2024-08-20T22:21:10.9372575Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-08-20T22:21:10.9373656Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9374882Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (inductor_timm, 1, 2, amz2023.linux.g5.4xlarge.nvidia.gpu) 2024-08-20T22:21:10.9376096Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9377204Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-08-20T22:21:10.9377837Z GITHUB_EVENT_NAME=push 2024-08-20T22:21:10.9378173Z DASHBOARD_TAG= 2024-08-20T22:21:10.9378481Z GITHUB_RUN_ID=10479310725 2024-08-20T22:21:10.9378939Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-08-20T22:21:10.9379479Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-08-20T22:21:10.9380472Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9381373Z GITHUB_ACTOR=pytorch-bot[bot] 2024-08-20T22:21:10.9381783Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-08-20T22:21:10.9382163Z PR_NUMBER= 2024-08-20T22:21:10.9382444Z GITHUB_RUN_ATTEMPT=1 2024-08-20T22:21:10.9382761Z VALGRIND=ON 2024-08-20T22:21:10.9383055Z ANACONDA_PYTHON_VERSION=3.10 2024-08-20T22:21:10.9383494Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-08-20T22:21:10.9383946Z TERM=vt100 2024-08-20T22:21:10.9384279Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-08-20T22:21:10.9384691Z INSTALLED_VISION=yes 2024-08-20T22:21:10.9385003Z BRANCH= 2024-08-20T22:21:10.9385288Z OPENSSL_ROOT_DIR=/opt/openssl 2024-08-20T22:21:10.9385681Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-08-20T22:21:10.9386100Z CUDA_PATH=/usr/local/cuda 2024-08-20T22:21:10.9386848Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-08-20T22:21:10.9387586Z GITHUB_SERVER_URL=https://github.com 2024-08-20T22:21:10.9388060Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-08-20T22:21:10.9388523Z REENABLED_ISSUES= 2024-08-20T22:21:10.9388828Z SHLVL=1 2024-08-20T22:21:10.9389087Z MAX_JOBS=14 2024-08-20T22:21:10.9389496Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-08-20T22:21:10.9389848Z NVARCH=x86_64 2024-08-20T22:21:10.9390166Z GITHUB_ACTOR_ID=54816060 2024-08-20T22:21:10.9390639Z GITHUB_WORKFLOW_SHA=40ec5f6ddd9787aca0449b24128343ff4c4a88b3 2024-08-20T22:21:10.9391170Z GITHUB_REF_NAME=ciflow/inductor/133712 2024-08-20T22:21:10.9391662Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-08-20T22:21:10.9392326Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-08-20T22:21:10.9392887Z GITHUB_JOB=test 2024-08-20T22:21:10.9393290Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-08-20T22:21:10.9393838Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-08-20T22:21:10.9394343Z NO_TEST_TIMEOUT=False 2024-08-20T22:21:10.9394667Z TD_DISTRIBUTED=False 2024-08-20T22:21:10.9395046Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-08-20T22:21:10.9395480Z GITHUB_REPOSITORY=pytorch/pytorch 2024-08-20T22:21:10.9395990Z NV_NVPROF_VERSION=12.1.105-1 2024-08-20T22:21:10.9396359Z GITHUB_RETENTION_DAYS=90 2024-08-20T22:21:10.9396713Z OPENSSL_DIR=/opt/openssl 2024-08-20T22:21:10.9397062Z GITHUB_ACTION_REPOSITORY= 2024-08-20T22:21:10.9398067Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:10.9399100Z GITHUB_BASE_REF= 2024-08-20T22:21:10.9399428Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-08-20T22:21:10.9399796Z CI=true 2024-08-20T22:21:10.9400116Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-08-20T22:21:10.9400579Z GITHUB_REPOSITORY_OWNER=pytorch 2024-08-20T22:21:10.9400952Z JOB_ID=29026527906 2024-08-20T22:21:10.9401263Z INSTALLED_PROTOBUF=yes 2024-08-20T22:21:10.9401589Z GITHUB_HEAD_REF= 2024-08-20T22:21:10.9401891Z GITHUB_ACTION_REF= 2024-08-20T22:21:10.9402327Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-08-20T22:21:10.9402785Z TEST_SHOWLOCALS=False 2024-08-20T22:21:10.9403126Z GITHUB_WORKFLOW=inductor 2024-08-20T22:21:10.9403484Z DEBIAN_FRONTEND=noninteractive 2024-08-20T22:21:10.9404396Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_25d2b960-ded4-4b31-b4ec-dedcf823a79b 2024-08-20T22:21:10.9405286Z NO_TD=False 2024-08-20T22:21:10.9405586Z SKIP_SCCACHE_INITIALIZATION=1 2024-08-20T22:21:10.9405947Z _=/usr/bin/env 2024-08-20T22:21:10.9406295Z + echo 'Testing pytorch' 2024-08-20T22:21:10.9406635Z Testing pytorch 2024-08-20T22:21:10.9406959Z + export LANG=C.UTF-8 2024-08-20T22:21:10.9407301Z + LANG=C.UTF-8 2024-08-20T22:21:10.9407586Z + PR_NUMBER= 2024-08-20T22:21:10.9407904Z + [[ inductor_timm == \d\e\f\a\u\l\t ]] 2024-08-20T22:21:10.9408368Z + [[ inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-08-20T22:21:10.9408821Z + [[ inductor_timm == \s\l\o\w ]] 2024-08-20T22:21:10.9409416Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-08-20T22:21:10.9410109Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-08-20T22:21:10.9410655Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:10.9411132Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-08-20T22:21:10.9411554Z + [[ inductor_timm == *crossref* ]] 2024-08-20T22:21:10.9412098Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-08-20T22:21:10.9412727Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-08-20T22:21:10.9413372Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:10.9413920Z + pip_install --user ninja==1.10.2 2024-08-20T22:21:10.9414450Z + pip install --progress-bar off --user ninja==1.10.2 2024-08-20T22:21:11.6731151Z Collecting ninja==1.10.2 2024-08-20T22:21:11.6933264Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-08-20T22:21:11.7293544Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-08-20T22:21:12.3052637Z Installing collected packages: ninja 2024-08-20T22:21:12.3131232Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-08-20T22:21:12.3132984Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-08-20T22:21:12.3566905Z Successfully installed ninja-1.10.2 2024-08-20T22:21:12.4341198Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:12.4343293Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:12.4344851Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:12.4345365Z + install_tlparse 2024-08-20T22:21:12.4345761Z + pip_install --user tlparse==0.3.25 2024-08-20T22:21:12.4346322Z + pip install --progress-bar off --user tlparse==0.3.25 2024-08-20T22:21:12.8637062Z Collecting tlparse==0.3.25 2024-08-20T22:21:12.8804091Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.7 kB) 2024-08-20T22:21:12.9295483Z Downloading tlparse-0.3.25-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-08-20T22:21:13.5666668Z Installing collected packages: tlparse 2024-08-20T22:21:13.6389288Z Successfully installed tlparse-0.3.25 2024-08-20T22:21:13.7150901Z ++ python -m site --user-base 2024-08-20T22:21:13.7391192Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-08-20T22:21:13.7392751Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-08-20T22:21:13.7393527Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-08-20T22:21:13.7394351Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-08-20T22:21:13.7395436Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-08-20T22:21:13.7396653Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-08-20T22:21:13.7397343Z + cd test 2024-08-20T22:21:13.7397869Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-08-20T22:21:15.3835887Z + [[ inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-08-20T22:21:15.3837115Z + [[ inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-08-20T22:21:15.3839913Z + DYNAMO_BENCHMARK_FLAGS=() 2024-08-20T22:21:15.3840974Z + [[ inductor_timm == *pr_time_benchmarks* ]] 2024-08-20T22:21:15.3841616Z + [[ inductor_timm == *dynamo_eager* ]] 2024-08-20T22:21:15.3842084Z + [[ inductor_timm == *aot_eager* ]] 2024-08-20T22:21:15.3842507Z + [[ inductor_timm == *aot_inductor* ]] 2024-08-20T22:21:15.3842937Z + [[ inductor_timm == *inductor* ]] 2024-08-20T22:21:15.3843341Z + [[ inductor_timm != *perf* ]] 2024-08-20T22:21:15.3843868Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-08-20T22:21:15.3844304Z + [[ inductor_timm == *dynamic* ]] 2024-08-20T22:21:15.3844695Z + [[ inductor_timm == *cpu* ]] 2024-08-20T22:21:15.3845242Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-08-20T22:21:15.3874327Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:15.3875027Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-08-20T22:21:15.3878234Z + cd test 2024-08-20T22:21:15.3878785Z + python -c 'import torch; print(torch.__config__.show())' 2024-08-20T22:21:16.8628915Z PyTorch built with: 2024-08-20T22:21:16.8630237Z - GCC 9.4 2024-08-20T22:21:16.8631093Z - C++ Version: 201703 2024-08-20T22:21:16.8632221Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:16.8633549Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:16.8634214Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:16.8634766Z - LAPACK is enabled (usually provided by MKL) 2024-08-20T22:21:16.8635581Z - NNPACK is enabled 2024-08-20T22:21:16.8635982Z - CPU capability usage: AVX2 2024-08-20T22:21:16.8636399Z - CUDA Runtime 12.1 2024-08-20T22:21:16.8636917Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-08-20T22:21:16.8637515Z - CuDNN 90.1 (built against CUDA 12.4) 2024-08-20T22:21:16.8637959Z - Magma 2.6.1 2024-08-20T22:21:16.8644775Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-08-20T22:21:16.8651063Z 2024-08-20T22:21:17.1819750Z + cd test 2024-08-20T22:21:17.1820531Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-08-20T22:21:18.5338994Z ATen/Parallel: 2024-08-20T22:21:18.5339404Z at::get_num_threads() : 8 2024-08-20T22:21:18.5339897Z at::get_num_interop_threads() : 16 2024-08-20T22:21:18.5340397Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-08-20T22:21:18.5340799Z omp_get_max_threads() : 8 2024-08-20T22:21:18.5341702Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-08-20T22:21:18.5342534Z mkl_get_max_threads() : 8 2024-08-20T22:21:18.5343150Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-08-20T22:21:18.5343779Z std::thread::hardware_concurrency() : 16 2024-08-20T22:21:18.5344194Z Environment variables: 2024-08-20T22:21:18.5344540Z OMP_NUM_THREADS : [not set] 2024-08-20T22:21:18.5344910Z MKL_NUM_THREADS : [not set] 2024-08-20T22:21:18.5345279Z ATen parallel backend: OpenMP 2024-08-20T22:21:18.5345535Z 2024-08-20T22:21:18.8101716Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-08-20T22:21:18.8102433Z + [[ inductor_timm == *backward* ]] 2024-08-20T22:21:18.8102981Z + [[ inductor_timm == *xla* ]] 2024-08-20T22:21:18.8103417Z + [[ inductor_timm == *executorch* ]] 2024-08-20T22:21:18.8103954Z + [[ inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-08-20T22:21:18.8104795Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-08-20T22:21:18.8105406Z + [[ inductor_timm == distributed ]] 2024-08-20T22:21:18.8105857Z + [[ inductor_timm == *inductor_distributed* ]] 2024-08-20T22:21:18.8106389Z + [[ inductor_timm == *inductor-halide* ]] 2024-08-20T22:21:18.8106941Z + [[ inductor_timm == *inductor-micro-benchmark* ]] 2024-08-20T22:21:18.8107492Z + [[ inductor_timm == *huggingface* ]] 2024-08-20T22:21:18.8108021Z + [[ inductor_timm == *timm* ]] 2024-08-20T22:21:18.8108462Z + install_torchvision 2024-08-20T22:21:18.8108796Z + local orig_preload 2024-08-20T22:21:18.8109114Z + local commit 2024-08-20T22:21:18.8109425Z ++ get_pinned_commit vision 2024-08-20T22:21:18.8109815Z ++ cat .github/ci_commit_pins/vision.txt 2024-08-20T22:21:18.8126350Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:18.8126845Z + orig_preload= 2024-08-20T22:21:18.8127398Z + '[' -n '' ']' 2024-08-20T22:21:18.8128262Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:18.8130034Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:19.1487292Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:19.1492031Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-ttc1odbk 2024-08-20T22:21:19.1519723Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-ttc1odbk 2024-08-20T22:21:20.7791477Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-08-20T22:21:20.7818588Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:22.1076246Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:22.4329673Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-08-20T22:21:24.9257152Z Preparing metadata (setup.py) ... [?25l- \ done 2024-08-20T22:21:24.9290590Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4) 2024-08-20T22:21:24.9294828Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+git40ec5f6) 2024-08-20T22:21:24.9299107Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-08-20T22:21:24.9369872Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-08-20T22:21:24.9374339Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-08-20T22:21:24.9377667Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-08-20T22:21:24.9380866Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-08-20T22:21:24.9384289Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.6.1) 2024-08-20T22:21:24.9389389Z Requirement already satisfied: sympy==1.13.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.1) 2024-08-20T22:21:24.9403889Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy==1.13.1->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-08-20T22:21:24.9884700Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-08-20T22:21:24.9949767Z Building wheels for collected packages: torchvision 2024-08-20T22:22:41.1438712Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done 2024-08-20T22:22:41.1468121Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2116298 sha256=fe4ee5e9d675fbaf5d8d202b5ddf9caaf22ab8c5ce9e2a86e3ca965e71b4735d 2024-08-20T22:22:41.1471065Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-08-20T22:22:41.1503770Z Successfully built torchvision 2024-08-20T22:22:41.6297307Z Installing collected packages: torchvision 2024-08-20T22:22:42.0520006Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-08-20T22:22:42.1894101Z + '[' -n '' ']' 2024-08-20T22:22:42.1894425Z + id=0 2024-08-20T22:22:42.1897928Z + test_dynamo_benchmark timm_models 0 2024-08-20T22:22:42.1898486Z ++ pwd 2024-08-20T22:22:42.1902009Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:42.1902755Z + local suite=timm_models 2024-08-20T22:22:42.1903123Z + shift 2024-08-20T22:22:42.1903450Z + local shard_id=0 2024-08-20T22:22:42.1903858Z + shift 2024-08-20T22:22:42.1904261Z + [[ inductor_timm == *perf_compare* ]] 2024-08-20T22:22:42.1904806Z + [[ inductor_timm == *perf* ]] 2024-08-20T22:22:42.1905219Z + [[ inductor_timm == *cpu* ]] 2024-08-20T22:22:42.1908664Z + [[ inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:42.1909618Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-08-20T22:22:42.1911506Z ++ pwd 2024-08-20T22:22:42.1912023Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:42.1912689Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-08-20T22:22:42.1934419Z + local name=inference 2024-08-20T22:22:42.1934918Z + shift 2024-08-20T22:22:42.1935281Z + local suite=timm_models 2024-08-20T22:22:42.1935710Z + shift 2024-08-20T22:22:42.1936060Z + local shard_id=0 2024-08-20T22:22:42.1936432Z + shift 2024-08-20T22:22:42.1936775Z + partition_flags=() 2024-08-20T22:22:42.1937169Z + local partition_flags 2024-08-20T22:22:42.1937571Z + [[ -n 2 ]] 2024-08-20T22:22:42.1937873Z + [[ -n 0 ]] 2024-08-20T22:22:42.1938468Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-08-20T22:22:42.1939119Z + [[ inductor_timm == *perf_compare* ]] 2024-08-20T22:22:42.1939547Z + [[ inductor_timm == *perf* ]] 2024-08-20T22:22:42.1939939Z + [[ inductor_timm == *aot_inductor* ]] 2024-08-20T22:22:42.1940355Z + [[ inductor_timm == *_avx2* ]] 2024-08-20T22:22:42.1940748Z + [[ inductor_timm == *_avx512* ]] 2024-08-20T22:22:42.1942275Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-08-20T22:22:48.6286560Z 2024-08-20T22:22:48.9641043Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:22:48.9641456Z 2024-08-20T22:22:49.0711360Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-08-20T22:44:17.2984383Z pass 2024-08-20T22:44:17.3487568Z TIMING: entire_frame_compile:62.81013 code_gen:27.1712 inductor_compile:44.25507 backend_compile:50.19689 total_wall_time:72.9788 2024-08-20T22:44:17.3491044Z STATS: call_* op count: 989 | FakeTensorMode.__torch_dispatch__:49959 | FakeTensor.__torch_dispatch__:15634 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-08-20T22:44:17.3493344Z Dynamo produced 3 graphs covering 989 ops with 7 graph breaks (5 unique) 2024-08-20T22:44:24.3606241Z 2024-08-20T22:44:25.3292225Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:25.3293257Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:44:25.3294110Z cuda train botnet26t_256 2024-08-20T22:45:14.8682124Z pass 2024-08-20T22:45:14.8788313Z TIMING: entire_frame_compile:29.6557 code_gen:22.72832 inductor_compile:31.63814 backend_compile:24.05858 total_wall_time:45.62267 2024-08-20T22:45:14.8790150Z STATS: call_* op count: 264 | FakeTensorMode.__torch_dispatch__:23667 | FakeTensor.__torch_dispatch__:5542 | ProxyTorchDispatchMode.__torch_dispatch__:9029 2024-08-20T22:45:14.8791266Z Dynamo produced 2 graphs covering 264 ops with 6 graph breaks (5 unique) 2024-08-20T22:45:20.3544787Z 2024-08-20T22:45:24.3420264Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:24.3420775Z loading model: 0it [00:03, ?it/s] 2024-08-20T22:45:24.3421210Z cuda train cait_m36_384 2024-08-20T22:45:24.3426950Z Traceback (most recent call last): 2024-08-20T22:45:24.3427834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2507, in validate_model 2024-08-20T22:45:24.3428595Z self.model_iter_fn(model, example_inputs) 2024-08-20T22:45:24.3429461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in forward_and_backward_pass 2024-08-20T22:45:24.3431882Z pred = mod(*cloned_inputs) 2024-08-20T22:45:24.3433561Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:45:24.3434511Z return self._call_impl(*args, **kwargs) 2024-08-20T22:45:24.3435754Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:45:24.3436752Z return forward_call(*args, **kwargs) 2024-08-20T22:45:24.3437629Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 438, in forward 2024-08-20T22:45:24.3438380Z x = self.forward_features(x) 2024-08-20T22:45:24.3439243Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 423, in forward_features 2024-08-20T22:45:24.3440192Z x = self.blocks(x) 2024-08-20T22:45:24.3441093Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:45:24.3441977Z return self._call_impl(*args, **kwargs) 2024-08-20T22:45:24.3443198Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:45:24.3444069Z return forward_call(*args, **kwargs) 2024-08-20T22:45:24.3445072Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 250, in forward 2024-08-20T22:45:24.3446112Z input = module(input) 2024-08-20T22:45:24.3447081Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:45:24.3448209Z return self._call_impl(*args, **kwargs) 2024-08-20T22:45:24.3449435Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:45:24.3450497Z return forward_call(*args, **kwargs) 2024-08-20T22:45:24.3451395Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 199, in forward 2024-08-20T22:45:24.3452291Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-08-20T22:45:24.3453373Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl 2024-08-20T22:45:24.3455633Z return self._call_impl(*args, **kwargs) 2024-08-20T22:45:24.3456888Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl 2024-08-20T22:45:24.3457762Z return forward_call(*args, **kwargs) 2024-08-20T22:45:24.3458887Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 153, in forward 2024-08-20T22:45:24.3459804Z x = (attn @ v).transpose(1, 2).reshape(B, N, C) 2024-08-20T22:45:24.3463207Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 248462 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-08-20T22:45:24.3465901Z 2024-08-20T22:45:24.3466191Z The above exception was the direct cause of the following exception: 2024-08-20T22:45:24.3466632Z 2024-08-20T22:45:24.3466777Z Traceback (most recent call last): 2024-08-20T22:45:24.3467410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4607, in run 2024-08-20T22:45:24.3468055Z ) = runner.load_model( 2024-08-20T22:45:24.3468707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 328, in load_model 2024-08-20T22:45:24.3469454Z self.validate_model(model, example_inputs) 2024-08-20T22:45:24.3470196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2509, in validate_model 2024-08-20T22:45:24.3470945Z raise RuntimeError("Eager run failed") from e 2024-08-20T22:45:24.3471407Z RuntimeError: Eager run failed 2024-08-20T22:45:24.3471662Z 2024-08-20T22:45:24.3471783Z eager_fail_to_run 2024-08-20T22:45:28.2561749Z 2024-08-20T22:45:29.2255827Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:29.2256298Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:45:29.2256733Z cuda train coat_lite_mini 2024-08-20T22:46:20.8951068Z pass 2024-08-20T22:46:20.9068705Z TIMING: entire_frame_compile:27.67023 code_gen:22.13233 inductor_compile:32.0145 backend_compile:22.87552 total_wall_time:48.00982 2024-08-20T22:46:20.9070075Z STATS: call_* op count: 456 | FakeTensorMode.__torch_dispatch__:35432 | FakeTensor.__torch_dispatch__:5893 | ProxyTorchDispatchMode.__torch_dispatch__:14386 2024-08-20T22:46:20.9075080Z Dynamo produced 2 graphs covering 456 ops with 6 graph breaks (5 unique) 2024-08-20T22:46:26.6803876Z 2024-08-20T22:46:28.3452174Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:46:28.3453174Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:46:28.3454076Z cuda train convit_base 2024-08-20T22:46:28.3771386Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-08-20T22:47:54.5179741Z pass 2024-08-20T22:47:54.5426425Z TIMING: entire_frame_compile:52.53476 code_gen:32.60405 inductor_compile:51.14832 backend_compile:42.18795 total_wall_time:83.74031 2024-08-20T22:47:54.5430512Z STATS: call_* op count: 1142 | FakeTensorMode.__torch_dispatch__:85759 | FakeTensor.__torch_dispatch__:15622 | ProxyTorchDispatchMode.__torch_dispatch__:40209 2024-08-20T22:47:54.5431655Z Dynamo produced 3 graphs covering 1142 ops with 7 graph breaks (5 unique) 2024-08-20T22:48:01.5442028Z 2024-08-20T22:48:02.4243284Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:02.4243879Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:48:02.4244330Z cuda train convmixer_768_32 2024-08-20T22:49:00.4156526Z pass 2024-08-20T22:49:00.4345397Z TIMING: entire_frame_compile:36.00082 code_gen:16.20555 inductor_compile:28.85137 backend_compile:26.45929 total_wall_time:48.85616 2024-08-20T22:49:00.4347929Z STATS: call_* op count: 300 | FakeTensorMode.__torch_dispatch__:38513 | FakeTensor.__torch_dispatch__:11061 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-08-20T22:49:00.4353247Z Dynamo produced 2 graphs covering 300 ops with 5 graph breaks (4 unique) 2024-08-20T22:49:06.6781481Z 2024-08-20T22:49:08.4235877Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:49:08.4236606Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:49:08.4237283Z cuda train convnext_base 2024-08-20T22:50:07.9893888Z W0820 22:50:07.988000 15430 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:51:30.5333395Z pass 2024-08-20T22:51:30.5822840Z TIMING: entire_frame_compile:107.02701 code_gen:48.44709 inductor_compile:78.51005 backend_compile:83.28712 total_wall_time:129.91745 2024-08-20T22:51:30.5827097Z STATS: call_* op count: 1508 | FakeTensorMode.__torch_dispatch__:80534 | FakeTensor.__torch_dispatch__:27882 | ProxyTorchDispatchMode.__torch_dispatch__:35464 2024-08-20T22:51:30.5828269Z Dynamo produced 3 graphs covering 1508 ops with 7 graph breaks (5 unique) 2024-08-20T22:51:40.0338664Z 2024-08-20T22:51:40.8033919Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:40.8034729Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:51:40.8035217Z cuda train crossvit_9_240 2024-08-20T22:52:37.7884615Z W0820 22:52:37.787000 16249 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:53:41.7251308Z pass 2024-08-20T22:53:41.7386392Z TIMING: entire_frame_compile:94.22733 code_gen:41.81326 inductor_compile:67.49618 backend_compile:76.12849 total_wall_time:113.32725 2024-08-20T22:53:41.7389957Z STATS: call_* op count: 1312 | FakeTensorMode.__torch_dispatch__:77628 | FakeTensor.__torch_dispatch__:23568 | ProxyTorchDispatchMode.__torch_dispatch__:34361 2024-08-20T22:53:41.7391111Z Dynamo produced 3 graphs covering 1312 ops with 7 graph breaks (5 unique) 2024-08-20T22:53:50.2579299Z 2024-08-20T22:53:51.5587666Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:53:51.5588189Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:53:51.5588636Z cuda train cspdarknet53 2024-08-20T22:54:55.3003813Z W0820 22:54:55.299000 17018 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:55:48.6877102Z pass 2024-08-20T22:55:48.7010820Z TIMING: entire_frame_compile:89.22637 code_gen:40.00177 inductor_compile:65.50169 backend_compile:69.61378 total_wall_time:108.13511 2024-08-20T22:55:48.7012207Z STATS: call_* op count: 950 | FakeTensorMode.__torch_dispatch__:60180 | FakeTensor.__torch_dispatch__:22264 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-08-20T22:55:48.7013319Z Dynamo produced 3 graphs covering 950 ops with 7 graph breaks (5 unique) 2024-08-20T22:55:57.2319463Z 2024-08-20T22:55:58.7963433Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:55:58.7964037Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:55:58.7965618Z cuda train deit_base_distilled_patch16_224 2024-08-20T22:56:28.4747488Z W0820 22:56:28.473000 17945 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T22:57:05.3796119Z pass 2024-08-20T22:57:05.4309535Z TIMING: entire_frame_compile:49.31859 code_gen:21.16237 inductor_compile:35.27533 backend_compile:39.72096 total_wall_time:58.22038 2024-08-20T22:57:05.4310915Z STATS: call_* op count: 753 | FakeTensorMode.__torch_dispatch__:40529 | FakeTensor.__torch_dispatch__:13130 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-08-20T22:57:05.4312018Z Dynamo produced 3 graphs covering 753 ops with 7 graph breaks (5 unique) 2024-08-20T22:57:11.7984105Z 2024-08-20T22:57:13.1166331Z loading model: 0it [00:00, ?it/s] 2024-08-20T22:57:13.1166924Z loading model: 0it [00:01, ?it/s] 2024-08-20T22:57:13.1167429Z cuda train dla102 2024-08-20T22:58:44.1712084Z W0820 22:58:44.170000 18260 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:00:07.8803957Z pass 2024-08-20T23:00:07.9028824Z TIMING: entire_frame_compile:136.82143 code_gen:61.19397 inductor_compile:97.85679 backend_compile:106.11194 total_wall_time:162.15098 2024-08-20T23:00:07.9031606Z STATS: call_* op count: 1457 | FakeTensorMode.__torch_dispatch__:90022 | FakeTensor.__torch_dispatch__:32600 | ProxyTorchDispatchMode.__torch_dispatch__:36994 2024-08-20T23:00:07.9034529Z Dynamo produced 3 graphs covering 1457 ops with 7 graph breaks (5 unique) 2024-08-20T23:00:19.1658519Z 2024-08-20T23:00:20.9131837Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:00:20.9132319Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:00:20.9135687Z cuda train dm_nfnet_f0 2024-08-20T23:01:17.0320517Z pass 2024-08-20T23:01:17.0836726Z TIMING: entire_frame_compile:30.27092 code_gen:18.02657 inductor_compile:31.0244 backend_compile:21.57221 total_wall_time:49.96502 2024-08-20T23:01:17.0838130Z STATS: call_* op count: 594 | FakeTensorMode.__torch_dispatch__:38376 | ProxyTorchDispatchMode.__torch_dispatch__:16005 | FakeTensor.__torch_dispatch__:11566 2024-08-20T23:01:17.0839253Z Dynamo produced 2 graphs covering 594 ops with 6 graph breaks (5 unique) 2024-08-20T23:01:23.0154889Z 2024-08-20T23:01:24.5823865Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:01:24.5825000Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:01:24.5825722Z cuda train dpn107 2024-08-20T23:03:06.4929107Z pass 2024-08-20T23:03:06.5553495Z TIMING: entire_frame_compile:65.05447 code_gen:33.69508 inductor_compile:54.54266 backend_compile:48.32023 total_wall_time:90.63717 2024-08-20T23:03:06.5554897Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:69021 | FakeTensor.__torch_dispatch__:18035 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-08-20T23:03:06.5556012Z Dynamo produced 2 graphs covering 634 ops with 6 graph breaks (5 unique) 2024-08-20T23:03:14.3923199Z 2024-08-20T23:03:15.3439923Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:03:15.3440500Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:03:15.3442789Z cuda train eca_botnext26ts_256 2024-08-20T23:04:04.0937461Z W0820 23:04:04.092000 21785 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:04:33.1641996Z pass 2024-08-20T23:04:33.1762548Z TIMING: entire_frame_compile:55.19711 code_gen:31.91353 inductor_compile:47.19215 backend_compile:44.62436 total_wall_time:72.71016 2024-08-20T23:04:33.1764849Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:37369 | FakeTensor.__torch_dispatch__:12499 | ProxyTorchDispatchMode.__torch_dispatch__:15134 2024-08-20T23:04:33.1766039Z Dynamo produced 3 graphs covering 634 ops with 7 graph breaks (5 unique) 2024-08-20T23:04:39.6631634Z 2024-08-20T23:04:40.6554820Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:40.6555314Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:04:40.6555760Z cuda train eca_halonext26ts 2024-08-20T23:05:33.5024797Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:05:33.5026075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:05:33.5027257Z pred = mod(*cloned_inputs) 2024-08-20T23:05:33.5030335Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1433, in forward 2024-08-20T23:05:33.5031189Z x = self.forward_features(x) 2024-08-20T23:05:33.5032121Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1425, in forward_features 2024-08-20T23:05:33.5033302Z x = self.stages(x) 2024-08-20T23:05:33.5034131Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 894, in forward 2024-08-20T23:05:33.5035340Z x = self.self_attn(x) 2024-08-20T23:05:33.5036144Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-08-20T23:05:33.5037195Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-08-20T23:05:33.5037757Z 2024-08-20T23:05:33.6963877Z W0820 23:05:33.695000 22685 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:06:03.2883222Z pass 2024-08-20T23:06:03.2951625Z TIMING: entire_frame_compile:57.8791 code_gen:35.25223 inductor_compile:51.14464 backend_compile:45.56514 total_wall_time:78.28177 2024-08-20T23:06:03.2952996Z STATS: call_* op count: 666 | FakeTensorMode.__torch_dispatch__:38610 | FakeTensor.__torch_dispatch__:12313 | ProxyTorchDispatchMode.__torch_dispatch__:15754 2024-08-20T23:06:03.2954574Z Dynamo produced 3 graphs covering 666 ops with 7 graph breaks (5 unique) 2024-08-20T23:06:09.8331720Z 2024-08-20T23:06:10.6726894Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:06:10.6727368Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:06:10.6727802Z cuda train ese_vovnet19b_dw 2024-08-20T23:06:46.6733079Z W0820 23:06:46.672000 23148 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:07:13.7755346Z pass 2024-08-20T23:07:13.7829291Z TIMING: entire_frame_compile:43.60843 code_gen:26.86253 inductor_compile:38.29734 backend_compile:33.8604 total_wall_time:57.89173 2024-08-20T23:07:13.7831814Z STATS: call_* op count: 434 | FakeTensorMode.__torch_dispatch__:26302 | FakeTensor.__torch_dispatch__:9349 | ProxyTorchDispatchMode.__torch_dispatch__:10392 2024-08-20T23:07:13.7834152Z Dynamo produced 4 graphs covering 434 ops with 7 graph breaks (5 unique) 2024-08-20T23:07:19.8629758Z 2024-08-20T23:07:20.7402969Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:20.7403859Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:07:20.7404854Z cuda train fbnetc_100 2024-08-20T23:08:15.2088118Z W0820 23:08:15.207000 23890 torch/_logging/_internal.py:1058] [7/0] Profiler function will be ignored 2024-08-20T23:09:09.1219875Z pass 2024-08-20T23:09:09.1268226Z TIMING: entire_frame_compile:85.69479 code_gen:37.14416 inductor_compile:59.4021 backend_compile:66.80486 total_wall_time:99.52847 2024-08-20T23:09:09.1270973Z STATS: call_* op count: 870 | FakeTensorMode.__torch_dispatch__:55101 | FakeTensor.__torch_dispatch__:20088 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-08-20T23:09:09.1273171Z Dynamo produced 4 graphs covering 870 ops with 7 graph breaks (5 unique) 2024-08-20T23:09:17.3528257Z 2024-08-20T23:09:18.4233484Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:09:18.4234510Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:09:18.4235376Z cuda train fbnetv3_b 2024-08-20T23:10:54.1969593Z pass 2024-08-20T23:10:54.2056314Z TIMING: entire_frame_compile:59.95941 code_gen:35.2122 inductor_compile:58.57695 backend_compile:45.56604 total_wall_time:89.63024 2024-08-20T23:10:54.2057718Z STATS: call_* op count: 459 | FakeTensorMode.__torch_dispatch__:58710 | FakeTensor.__torch_dispatch__:19672 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-08-20T23:10:54.2058826Z Dynamo produced 2 graphs covering 459 ops with 6 graph breaks (5 unique) 2024-08-20T23:11:02.3092224Z 2024-08-20T23:11:03.5176065Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:03.5176638Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:11:03.5177066Z cuda train gernet_l 2024-08-20T23:11:51.5659890Z pass 2024-08-20T23:11:51.5870103Z TIMING: entire_frame_compile:31.91488 code_gen:15.38998 inductor_compile:25.8564 backend_compile:23.25973 total_wall_time:43.26579 2024-08-20T23:11:51.5871732Z STATS: call_* op count: 250 | FakeTensorMode.__torch_dispatch__:30714 | FakeTensor.__torch_dispatch__:8886 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-08-20T23:11:51.5873352Z Dynamo produced 2 graphs covering 250 ops with 6 graph breaks (5 unique) 2024-08-20T23:11:57.5428127Z 2024-08-20T23:11:58.4894246Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:58.4894830Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:11:58.4895326Z cuda train ghostnet_100 2024-08-20T23:13:19.5589400Z pass 2024-08-20T23:13:19.5639228Z TIMING: entire_frame_compile:53.70988 code_gen:31.56723 inductor_compile:49.24653 backend_compile:40.82977 total_wall_time:75.99258 2024-08-20T23:13:19.5641186Z STATS: call_* op count: 405 | FakeTensorMode.__torch_dispatch__:50795 | FakeTensor.__torch_dispatch__:13895 | ProxyTorchDispatchMode.__torch_dispatch__:19864 2024-08-20T23:13:19.5642906Z Dynamo produced 2 graphs covering 405 ops with 6 graph breaks (5 unique) 2024-08-20T23:13:26.7885650Z 2024-08-20T23:13:28.0377194Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:13:28.0378650Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:13:28.0379127Z cuda train gluon_inception_v3 2024-08-20T23:14:06.5126905Z W0820 23:14:06.511000 28755 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:15:17.9269185Z pass 2024-08-20T23:15:17.9504524Z TIMING: entire_frame_compile:97.85074 inductor_compile:44.20854 backend_compile:71.15949 code_gen:21.39108 total_wall_time:101.35078 2024-08-20T23:15:17.9507273Z STATS: call_* op count: 1281 | FakeTensorMode.__torch_dispatch__:68829 | FakeTensor.__torch_dispatch__:14446 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-08-20T23:15:17.9509176Z Dynamo produced 3 graphs covering 1281 ops with 7 graph breaks (5 unique) 2024-08-20T23:15:25.6316392Z 2024-08-20T23:15:26.5668577Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:26.5669695Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:15:26.5670689Z cuda train gmixer_24_224 2024-08-20T23:16:22.3399916Z pass 2024-08-20T23:16:22.3653257Z TIMING: entire_frame_compile:31.86511 code_gen:18.29396 inductor_compile:30.42096 backend_compile:24.12003 total_wall_time:50.80365 2024-08-20T23:16:22.3654652Z STATS: call_* op count: 587 | FakeTensorMode.__torch_dispatch__:49688 | FakeTensor.__torch_dispatch__:11442 | ProxyTorchDispatchMode.__torch_dispatch__:23826 2024-08-20T23:16:22.3655801Z Dynamo produced 2 graphs covering 587 ops with 6 graph breaks (5 unique) 2024-08-20T23:16:28.4452473Z 2024-08-20T23:16:29.3332870Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:29.3333464Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:16:29.3335021Z cuda train gmlp_s16_224 2024-08-20T23:17:22.7485205Z W0820 23:17:22.747000 29274 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:18:33.7861165Z pass 2024-08-20T23:18:33.8068587Z TIMING: entire_frame_compile:95.56779 code_gen:41.57129 inductor_compile:69.32036 backend_compile:76.11398 total_wall_time:115.51577 2024-08-20T23:18:33.8071626Z STATS: call_* op count: 1396 | FakeTensorMode.__torch_dispatch__:77799 | FakeTensor.__torch_dispatch__:26803 | ProxyTorchDispatchMode.__torch_dispatch__:35364 2024-08-20T23:18:33.8073834Z Dynamo produced 3 graphs covering 1396 ops with 7 graph breaks (5 unique) 2024-08-20T23:18:42.7770794Z 2024-08-20T23:18:44.4577151Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:18:44.4577654Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:18:44.4579622Z cuda train hrnet_w18 2024-08-20T23:22:37.8202479Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:22:37.8203741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:22:37.8204694Z pred = mod(*cloned_inputs) 2024-08-20T23:22:37.8205934Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-08-20T23:22:37.8206696Z y = self.forward_features(x) 2024-08-20T23:22:37.8208036Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-08-20T23:22:37.8208816Z yl = self.stages(x) 2024-08-20T23:22:37.8209686Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-08-20T23:22:37.8210578Z yl = self.stage2(xl) 2024-08-20T23:22:37.8211625Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-08-20T23:22:37.8212561Z x = module(x) 2024-08-20T23:22:37.8213305Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-08-20T23:22:37.8214018Z y = y + f(x[j]) 2024-08-20T23:22:37.8214217Z 2024-08-20T23:22:42.1665611Z pass 2024-08-20T23:22:42.1920143Z TIMING: entire_frame_compile:170.3646 code_gen:72.55272 inductor_compile:126.42967 backend_compile:121.61319 total_wall_time:226.68504 2024-08-20T23:22:42.1921633Z STATS: call_* op count: 1475 | FakeTensorMode.__torch_dispatch__:173385 | FakeTensor.__torch_dispatch__:46888 | ProxyTorchDispatchMode.__torch_dispatch__:71305 2024-08-20T23:22:42.1922756Z Dynamo produced 2 graphs covering 1475 ops with 5 graph breaks (4 unique) 2024-08-20T23:22:56.5570756Z 2024-08-20T23:22:57.8096695Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:22:57.8097248Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:22:57.8097689Z cuda train inception_v3 2024-08-20T23:23:37.5469253Z pass 2024-08-20T23:23:37.5687845Z TIMING: entire_frame_compile:31.15671 inductor_compile:6.67295 backend_compile:18.06143 total_wall_time:34.68896 2024-08-20T23:23:37.5689289Z STATS: call_* op count: 412 | FakeTensorMode.__torch_dispatch__:39410 | FakeTensor.__torch_dispatch__:46 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-08-20T23:23:37.5690477Z Dynamo produced 2 graphs covering 412 ops with 6 graph breaks (5 unique) 2024-08-20T23:23:42.4149596Z 2024-08-20T23:23:43.9645209Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:23:43.9645820Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:23:43.9647494Z cuda train jx_nest_base 2024-08-20T23:25:02.1322371Z W0820 23:25:02.131000 30979 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:26:17.4472933Z pass 2024-08-20T23:26:17.4882295Z TIMING: entire_frame_compile:112.51526 code_gen:55.28997 inductor_compile:86.93422 backend_compile:89.70922 total_wall_time:139.85268 2024-08-20T23:26:17.4883677Z STATS: call_* op count: 1671 | FakeTensorMode.__torch_dispatch__:98704 | FakeTensor.__torch_dispatch__:27894 | ProxyTorchDispatchMode.__torch_dispatch__:43281 2024-08-20T23:26:17.4884880Z Dynamo produced 3 graphs covering 1671 ops with 7 graph breaks (5 unique) 2024-08-20T23:26:26.9570709Z 2024-08-20T23:26:27.6961213Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:27.6961842Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:26:27.6962404Z cuda train lcnet_050 2024-08-20T23:26:58.6437369Z pass 2024-08-20T23:26:58.6460861Z TIMING: entire_frame_compile:18.97133 code_gen:11.47504 inductor_compile:19.1664 backend_compile:14.78088 total_wall_time:28.34771 2024-08-20T23:26:58.6462378Z STATS: call_* op count: 129 | FakeTensorMode.__torch_dispatch__:17856 | FakeTensor.__torch_dispatch__:6005 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-08-20T23:26:58.6463474Z Dynamo produced 2 graphs covering 129 ops with 6 graph breaks (5 unique) 2024-08-20T23:27:03.8421100Z 2024-08-20T23:27:04.9393299Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:27:04.9393782Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:27:04.9394207Z cuda train levit_128 2024-08-20T23:28:22.6950186Z skipping cudagraphs due to deterministic index put. Found from : 2024-08-20T23:28:22.6951352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 404, in torch_dynamo_resume_in_forward_and_backward_pass_at_402 2024-08-20T23:28:22.6953387Z pred = mod(*cloned_inputs) 2024-08-20T23:28:22.6954701Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 718, in forward 2024-08-20T23:28:22.6955485Z x = self.forward_features(x) 2024-08-20T23:28:22.6956379Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 709, in forward_features 2024-08-20T23:28:22.6957158Z x = self.stages(x) 2024-08-20T23:28:22.6958006Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 520, in forward 2024-08-20T23:28:22.6958741Z x = self.blocks(x) 2024-08-20T23:28:22.6959500Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 458, in forward 2024-08-20T23:28:22.6960266Z x = x + self.drop_path1(self.attn(x)) 2024-08-20T23:28:22.6961434Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 237, in forward 2024-08-20T23:28:22.6962304Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-08-20T23:28:22.6963352Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 212, in get_attention_biases 2024-08-20T23:28:22.6964267Z return self.attention_biases[:, self.attention_bias_idxs] 2024-08-20T23:28:22.6964753Z 2024-08-20T23:28:22.9861136Z W0820 23:28:22.985000 32355 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:29:13.6723198Z pass 2024-08-20T23:29:13.6760374Z TIMING: entire_frame_compile:97.11534 code_gen:46.2545 inductor_compile:74.10194 backend_compile:78.15266 total_wall_time:122.70524 2024-08-20T23:29:13.6761875Z STATS: call_* op count: 1261 | FakeTensorMode.__torch_dispatch__:76161 | FakeTensor.__torch_dispatch__:22877 | ProxyTorchDispatchMode.__torch_dispatch__:32073 2024-08-20T23:29:13.6763021Z Dynamo produced 3 graphs covering 1261 ops with 7 graph breaks (5 unique) 2024-08-20T23:29:23.0124511Z 2024-08-20T23:29:24.2242822Z loading model: 0it [00:00, ?it/s] 2024-08-20T23:29:24.2243400Z loading model: 0it [00:01, ?it/s] 2024-08-20T23:29:24.2243931Z cuda train mixer_b16_224 2024-08-20T23:29:55.1949624Z W0820 23:29:55.194000 33424 torch/_logging/_internal.py:1058] [6/0] Profiler function will be ignored 2024-08-20T23:30:30.6961901Z pass 2024-08-20T23:30:30.7392811Z TIMING: entire_frame_compile:49.43858 code_gen:21.80486 inductor_compile:35.68085 backend_compile:40.44902 total_wall_time:59.28913 2024-08-20T23:30:30.7394191Z STATS: call_* op count: 670 | FakeTensorMode.__torch_dispatch__:38728 | FakeTensor.__torch_dispatch__:13368 | ProxyTorchDispatchMode.__torch_dispatch__:17868 2024-08-20T23:30:30.7395301Z Dynamo produced 3 graphs covering 670 ops with 7 graph breaks (5 unique) 2024-08-20T23:30:33.9834989Z accuracy pass_rate=96.67% 2024-08-20T23:30:33.9835611Z calls_captured gmean=0.00x mean=779.033x 2024-08-20T23:30:33.9836092Z unique_graphs gmean=0.00x mean=2.533x 2024-08-20T23:30:33.9837940Z graph_breaks gmean=0.00x mean=6.267x 2024-08-20T23:30:33.9841724Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-08-20T23:30:33.9845571Z autograd_captures gmean=0.00x mean=0.000x 2024-08-20T23:30:33.9849319Z autograd_compiles gmean=0.00x mean=0.000x 2024-08-20T23:30:33.9852827Z cudagraph_skips gmean=0.00x mean=0.133x 2024-08-20T23:30:36.3697162Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv 2024-08-20T23:30:36.6334846Z adv_inception_v3 PASS 2024-08-20T23:30:36.6339186Z beit_base_patch16_224 PASS 2024-08-20T23:30:36.6344424Z botnet26t_256 PASS 2024-08-20T23:30:36.6348383Z cait_m36_384 XFAIL 2024-08-20T23:30:36.6352960Z coat_lite_mini PASS 2024-08-20T23:30:36.6357468Z convit_base PASS 2024-08-20T23:30:36.6361887Z convmixer_768_32 PASS 2024-08-20T23:30:36.6366401Z convnext_base PASS 2024-08-20T23:30:36.6371165Z crossvit_9_240 PASS 2024-08-20T23:30:36.6375576Z cspdarknet53 PASS 2024-08-20T23:30:36.6380028Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:30:36.6384554Z dla102 PASS 2024-08-20T23:30:36.6388933Z dm_nfnet_f0 PASS 2024-08-20T23:30:36.6393424Z dpn107 PASS 2024-08-20T23:30:36.6397774Z eca_botnext26ts_256 PASS 2024-08-20T23:30:36.6402417Z eca_halonext26ts PASS 2024-08-20T23:30:36.6406981Z ese_vovnet19b_dw PASS 2024-08-20T23:30:36.6411402Z fbnetc_100 PASS 2024-08-20T23:30:36.6415929Z fbnetv3_b PASS 2024-08-20T23:30:36.6420451Z gernet_l PASS 2024-08-20T23:30:36.6424866Z ghostnet_100 PASS 2024-08-20T23:30:36.6429442Z gluon_inception_v3 PASS 2024-08-20T23:30:36.6434430Z gmixer_24_224 PASS 2024-08-20T23:30:36.6438962Z gmlp_s16_224 PASS 2024-08-20T23:30:36.6443251Z hrnet_w18 PASS 2024-08-20T23:30:36.6448292Z inception_v3 PASS 2024-08-20T23:30:36.6452576Z jx_nest_base PASS 2024-08-20T23:30:36.6456975Z lcnet_050 PASS 2024-08-20T23:30:36.6461399Z levit_128 PASS 2024-08-20T23:30:36.6465794Z mixer_b16_224 PASS 2024-08-20T23:30:36.6885079Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_timm_training.csv 2024-08-20T23:30:36.9517040Z adv_inception_v3 PASS 2024-08-20T23:30:36.9521055Z beit_base_patch16_224 PASS 2024-08-20T23:30:36.9525536Z botnet26t_256 PASS 2024-08-20T23:30:36.9529975Z cait_m36_384 PASS 2024-08-20T23:30:36.9534737Z coat_lite_mini PASS 2024-08-20T23:30:36.9539793Z convit_base PASS 2024-08-20T23:30:36.9544304Z convmixer_768_32 PASS 2024-08-20T23:30:36.9549048Z convnext_base PASS 2024-08-20T23:30:36.9553703Z crossvit_9_240 PASS 2024-08-20T23:30:36.9558190Z cspdarknet53 PASS 2024-08-20T23:30:36.9562757Z deit_base_distilled_patch16_224 PASS 2024-08-20T23:30:36.9567962Z dla102 PASS 2024-08-20T23:30:36.9572147Z dm_nfnet_f0 PASS 2024-08-20T23:30:36.9576617Z dpn107 PASS 2024-08-20T23:30:36.9581250Z eca_botnext26ts_256 PASS 2024-08-20T23:30:36.9585690Z eca_halonext26ts PASS 2024-08-20T23:30:36.9590312Z ese_vovnet19b_dw PASS 2024-08-20T23:30:36.9594937Z fbnetc_100 PASS 2024-08-20T23:30:36.9599663Z fbnetv3_b PASS 2024-08-20T23:30:36.9604054Z gernet_l PASS 2024-08-20T23:30:36.9608806Z ghostnet_100 PASS 2024-08-20T23:30:36.9613279Z gluon_inception_v3 PASS 2024-08-20T23:30:36.9617911Z gmixer_24_224 PASS 2024-08-20T23:30:36.9622407Z gmlp_s16_224 PASS 2024-08-20T23:30:36.9627057Z hrnet_w18 PASS 2024-08-20T23:30:36.9631430Z inception_v3 PASS 2024-08-20T23:30:36.9636563Z jx_nest_base PASS 2024-08-20T23:30:36.9640852Z lcnet_050 PASS 2024-08-20T23:30:36.9645658Z levit_128 PASS 2024-08-20T23:30:36.9650551Z mixer_b16_224 PASS 2024-08-20T23:30:37.0068254Z + cleanup_workspace 2024-08-20T23:30:37.0069523Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-08-20T23:30:37.0070809Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-08-20T23:30:37.0071818Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-08-20T23:30:37.0072455Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-08-20T23:30:37.0073250Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-08-20T23:30:37.0074118Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-08-20T23:30:37.0074811Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-08-20T23:30:37.7593637Z ##[group]Run cat test/**/*_toprint.log || true 2024-08-20T23:30:37.7594165Z cat test/**/*_toprint.log || true 2024-08-20T23:30:37.7608660Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:37.7609159Z env: 2024-08-20T23:30:37.7609445Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:37.7609900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:37.7610629Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:37.7611260Z ##[endgroup] 2024-08-20T23:30:37.7704757Z cat: 'test/**/*_toprint.log': No such file or directory 2024-08-20T23:30:37.7739971Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:30:37.7740449Z kill "$MONITOR_SCRIPT_PID" 2024-08-20T23:30:37.7749697Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:37.7750207Z env: 2024-08-20T23:30:37.7750488Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:37.7750941Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:37.7751677Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:37.7752340Z MONITOR_SCRIPT_PID: 233720 2024-08-20T23:30:37.7752695Z ##[endgroup] 2024-08-20T23:30:37.7947672Z Prepare all required actions 2024-08-20T23:30:37.7948141Z Getting action download info 2024-08-20T23:30:37.9957339Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-08-20T23:30:38.1543318Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-08-20T23:30:38.1543782Z with: 2024-08-20T23:30:38.1544313Z file-suffix: test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906 2024-08-20T23:30:38.1544980Z s3-bucket: gha-artifacts 2024-08-20T23:30:38.1545323Z env: 2024-08-20T23:30:38.1545592Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.1546041Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.1546778Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.1547423Z ##[endgroup] 2024-08-20T23:30:38.1573805Z ##[group]Run # Remove any previous test jsons if they exist 2024-08-20T23:30:38.1574447Z # Remove any previous test jsons if they exist 2024-08-20T23:30:38.1574958Z rm -f test-jsons-*.zip 2024-08-20T23:30:38.1575484Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-08-20T23:30:38.1584607Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:38.1585111Z env: 2024-08-20T23:30:38.1585391Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.1585900Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.1586634Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.1587525Z FILE_SUFFIX: test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906 2024-08-20T23:30:38.1588160Z ##[endgroup] 2024-08-20T23:30:38.1867394Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-08-20T23:30:38.1896736Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-08-20T23:30:38.1897622Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-08-20T23:30:38.1903489Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-08-20T23:30:38.1909059Z adding: test/slow_tests.json (deflated 82%) 2024-08-20T23:30:38.1942068Z ##[group]Run # Remove any previous test reports if they exist 2024-08-20T23:30:38.1942703Z # Remove any previous test reports if they exist 2024-08-20T23:30:38.1943223Z rm -f test-reports-*.zip 2024-08-20T23:30:38.1943803Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-08-20T23:30:38.1952854Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:38.1953340Z env: 2024-08-20T23:30:38.1953621Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.1954071Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.1954800Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.1955680Z FILE_SUFFIX: test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906 2024-08-20T23:30:38.1956324Z ##[endgroup] 2024-08-20T23:30:38.2218756Z adding: test/test-reports/inference_timm_models.csv (deflated 68%) 2024-08-20T23:30:38.2219550Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-08-20T23:30:38.2223602Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-08-20T23:30:38.2224553Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-08-20T23:30:38.2256556Z ##[group]Run # Remove any previous usage logs if they exist 2024-08-20T23:30:38.2257174Z # Remove any previous usage logs if they exist 2024-08-20T23:30:38.2257670Z rm -f logs-*.zip 2024-08-20T23:30:38.2258328Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-08-20T23:30:38.2259100Z # so check to see if the file exists first 2024-08-20T23:30:38.2259594Z if [ -f 'usage_log.txt' ]; then 2024-08-20T23:30:38.2260338Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-08-20T23:30:38.2260836Z fi 2024-08-20T23:30:38.2261198Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-08-20T23:30:38.2261764Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-08-20T23:30:38.2262255Z fi 2024-08-20T23:30:38.2270969Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:38.2271455Z env: 2024-08-20T23:30:38.2271735Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.2272185Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.2272915Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.2273800Z FILE_SUFFIX: test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906 2024-08-20T23:30:38.2274441Z ##[endgroup] 2024-08-20T23:30:38.2360230Z adding: usage_log.txt (deflated 92%) 2024-08-20T23:30:38.2448430Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-08-20T23:30:38.2449144Z # Remove any previous debugging artifacts if they exist 2024-08-20T23:30:38.2449694Z rm -f debug-*.zip 2024-08-20T23:30:38.2450075Z if [ -d 'test/debug' ]; then 2024-08-20T23:30:38.2450568Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-08-20T23:30:38.2451040Z fi 2024-08-20T23:30:38.2459065Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:38.2459561Z env: 2024-08-20T23:30:38.2459834Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.2460287Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.2461019Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.2461910Z FILE_SUFFIX: test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906 2024-08-20T23:30:38.2462541Z ##[endgroup] 2024-08-20T23:30:38.2557601Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:30:38.2558030Z with: 2024-08-20T23:30:38.2558309Z s3-bucket: gha-artifacts 2024-08-20T23:30:38.2558747Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:38.2559382Z retention-days: 14 2024-08-20T23:30:38.2559705Z if-no-files-found: warn 2024-08-20T23:30:38.2560057Z path: test-jsons-*.zip 2024-08-20T23:30:38.2560395Z name: artifact 2024-08-20T23:30:38.2560688Z region: us-east-1 2024-08-20T23:30:38.2560990Z env: 2024-08-20T23:30:38.2561261Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.2561703Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.2562422Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.2563051Z ##[endgroup] 2024-08-20T23:30:38.6237595Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:30:38.6238188Z With the provided path, there will be 1 file uploaded 2024-08-20T23:30:38.6238794Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:38.6295095Z Starting upload of test-jsons-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:38.7440975Z Finished upload of test-jsons-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:38.7710291Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:30:38.7710719Z with: 2024-08-20T23:30:38.7711002Z s3-bucket: gha-artifacts 2024-08-20T23:30:38.7711428Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:38.7711894Z retention-days: 14 2024-08-20T23:30:38.7712218Z if-no-files-found: error 2024-08-20T23:30:38.7712578Z path: test-reports-*.zip 2024-08-20T23:30:38.7712925Z name: artifact 2024-08-20T23:30:38.7713217Z region: us-east-1 2024-08-20T23:30:38.7713515Z env: 2024-08-20T23:30:38.7713791Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:38.7714238Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:38.7714951Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:38.7715580Z ##[endgroup] 2024-08-20T23:30:39.1235007Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:30:39.1235772Z With the provided path, there will be 1 file uploaded 2024-08-20T23:30:39.1236960Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:39.1292050Z Starting upload of test-reports-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:39.2502178Z Finished upload of test-reports-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:39.2775232Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:30:39.2775665Z with: 2024-08-20T23:30:39.2775942Z s3-bucket: gha-artifacts 2024-08-20T23:30:39.2776411Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:39.2776876Z retention-days: 14 2024-08-20T23:30:39.2777206Z if-no-files-found: ignore 2024-08-20T23:30:39.2777559Z path: logs-*.zip 2024-08-20T23:30:39.2777859Z name: artifact 2024-08-20T23:30:39.2778151Z region: us-east-1 2024-08-20T23:30:39.2778460Z env: 2024-08-20T23:30:39.2778746Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:39.2779197Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:39.2779917Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:39.2780552Z ##[endgroup] 2024-08-20T23:30:39.6294221Z NOTE: s3-prefix specified, ignoring name parameter 2024-08-20T23:30:39.6295097Z With the provided path, there will be 1 file uploaded 2024-08-20T23:30:39.6295899Z Uploading to s3 prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:39.6352756Z Starting upload of logs-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:39.7323566Z Finished upload of logs-test-inductor_timm-1-2-amz2023.linux.g5.4xlarge.nvidia.gpu_29026527906.zip 2024-08-20T23:30:39.7598825Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-08-20T23:30:39.7599262Z with: 2024-08-20T23:30:39.7599543Z s3-bucket: gha-artifacts 2024-08-20T23:30:39.7599989Z s3-prefix: pytorch/pytorch/10479310725/1/artifact 2024-08-20T23:30:39.7600629Z retention-days: 14 2024-08-20T23:30:39.7600959Z if-no-files-found: ignore 2024-08-20T23:30:39.7601319Z path: debug-*.zip 2024-08-20T23:30:39.7601627Z name: artifact 2024-08-20T23:30:39.7601920Z region: us-east-1 2024-08-20T23:30:39.7602221Z env: 2024-08-20T23:30:39.7602494Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:39.7602939Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:39.7603660Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:39.7604286Z ##[endgroup] 2024-08-20T23:30:40.1045611Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-08-20T23:30:40.1316218Z ##[group]Run # shellcheck disable=SC2156 2024-08-20T23:30:40.1316757Z # shellcheck disable=SC2156 2024-08-20T23:30:40.1317537Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-08-20T23:30:40.1326871Z shell: /usr/bin/bash -e {0} 2024-08-20T23:30:40.1327239Z env: 2024-08-20T23:30:40.1327529Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:40.1328000Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:40.1328732Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:40.1329392Z ##[endgroup] 2024-08-20T23:30:40.3983097Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-08-20T23:30:40.3983652Z with: 2024-08-20T23:30:40.3983912Z env: 2024-08-20T23:30:40.3984189Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:40.3984630Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:40.3985365Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:40.3985996Z ##[endgroup] 2024-08-20T23:30:40.4005227Z ##[group]Run set -eou pipefail 2024-08-20T23:30:40.4005632Z set -eou pipefail 2024-08-20T23:30:40.4005982Z  2024-08-20T23:30:40.4006546Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-08-20T23:30:40.4007163Z for _ in $(seq 1440); do 2024-08-20T23:30:40.4007627Z  # Break if no ssh session exists anymore 2024-08-20T23:30:40.4008117Z  if [ "$(who)" = "" ]; then 2024-08-20T23:30:40.4008519Z  break 2024-08-20T23:30:40.4008859Z  fi 2024-08-20T23:30:40.4009155Z  echo "." 2024-08-20T23:30:40.4009483Z  sleep 5 2024-08-20T23:30:40.4009801Z done 2024-08-20T23:30:40.4018781Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:40.4019268Z env: 2024-08-20T23:30:40.4019548Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:40.4020002Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:40.4020730Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:40.4021367Z ##[endgroup] 2024-08-20T23:30:40.4051597Z Holding runner for 2 hours until all ssh sessions have logged out 2024-08-20T23:30:40.4108294Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:30:40.4109047Z # ignore expansion of "docker ps -q" since it could be empty 2024-08-20T23:30:40.4109632Z # shellcheck disable=SC2046 2024-08-20T23:30:40.4110103Z docker stop $(docker ps -q) || true 2024-08-20T23:30:40.4110574Z # Prune all of the docker images 2024-08-20T23:30:40.4111026Z docker system prune -af 2024-08-20T23:30:40.4119370Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:40.4119860Z env: 2024-08-20T23:30:40.4120142Z GIT_DEFAULT_BRANCH: main 2024-08-20T23:30:40.4120598Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-08-20T23:30:40.4121334Z DOCKER_CONTAINER_ID: a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:40.4121972Z ##[endgroup] 2024-08-20T23:30:41.0523151Z a553b99c8e72 2024-08-20T23:30:42.5095490Z Deleted Containers: 2024-08-20T23:30:42.5096137Z a553b99c8e728b0fc6c8ce537e64a334f688520cd417fca65b4dfdaeb31f95f5 2024-08-20T23:30:42.5096570Z 2024-08-20T23:30:53.6855434Z Deleted Images: 2024-08-20T23:30:53.6857349Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:f6d216893d65c7b8ae43df4daaf247db808378e9 2024-08-20T23:30:53.6860229Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:3a95d3a177465127ef6cd3b50787d6a050367070fd9063dbb688fad07a96b402 2024-08-20T23:30:53.6861836Z deleted: sha256:8fef7805cec6e829013a3d62f158b75bdc2f02cdfe3428c032ea017b488473bb 2024-08-20T23:30:53.6862716Z deleted: sha256:c2ce10014dd98f6abbfbf2265f9780a8204c2a7a27463e61ee565573a23d141d 2024-08-20T23:30:53.6863784Z deleted: sha256:ae249c0e6e6013bd6ffea2ca0b0160c015c9ebf7c03070bc2f983a9ff2415bf5 2024-08-20T23:30:53.6864704Z deleted: sha256:8a73883c1067c0e29ff128f553518682b47ed9707d2f35e7d5138e2caf029d61 2024-08-20T23:30:53.6865523Z deleted: sha256:9c040db729fa084584aaf5573082408a2196bc64e5952270c79dc610ed4680b6 2024-08-20T23:30:53.6866309Z deleted: sha256:17311eeeb8b71b16699880c0f646e185107496c4d41da5d23370603768ec030c 2024-08-20T23:30:53.6867108Z deleted: sha256:07cc10bece8501b1c8035d4385ad9e9867dd6e92fa32218a4dcb3e42af7e0411 2024-08-20T23:30:53.6868262Z deleted: sha256:f3e32d05099d1a3115a4722bc2e94a48752c3e965446773673808252671bb967 2024-08-20T23:30:53.6869096Z deleted: sha256:be7c3f41475e3e72ffe3ac2f92ab2f4cf49d5496b2aae738942ac3bb687d742c 2024-08-20T23:30:53.6869971Z deleted: sha256:4fcac490be825129b468cf458089f1fcf8fd56afc44ef4ab27c2eff41ab1ac90 2024-08-20T23:30:53.6870788Z deleted: sha256:31c66d80a2a4792897f267d363c0cf84fb821c5d4a275ab1636f778fd0284c08 2024-08-20T23:30:53.6871586Z deleted: sha256:8a8bcb6085e7e438e5533e8a5613e2ae5888543853568d023af3e2623a1593a9 2024-08-20T23:30:53.6872385Z deleted: sha256:2f6cb60029c79d11567b738f5bdbfb9cb1838cc0f194c4a5e6ae685178d2ed34 2024-08-20T23:30:53.6873207Z deleted: sha256:e546391804d3313f558e1ac16e93cb29a417a92e4eccaf0d4bd12af3bbcf39e3 2024-08-20T23:30:53.6874026Z deleted: sha256:1623b2aa16f2bdc6b572f6799498d9a92fa8114d07c440599b17aea72e22f8b1 2024-08-20T23:30:53.6874846Z deleted: sha256:c0386ca58b7cb7afee015b7e58d2f70525eaf73d6da2cc63868b20e14b1f9497 2024-08-20T23:30:53.6875668Z deleted: sha256:dcbf534ff289186e246531925fed90f72516eb3d63663bad234d611cdc025d62 2024-08-20T23:30:53.6876480Z deleted: sha256:0bab537af579f56c9cd51c7073fd53635fffb424b363236e0de38a7dd0e60ae8 2024-08-20T23:30:53.6877296Z deleted: sha256:d85a852ae1ee027e2fc17240ed68b87957b0a6d210c9119752d8979a0dcd55cf 2024-08-20T23:30:53.6878103Z deleted: sha256:f3821d761b555362e44cf9a1447d8fd2d95b37d54aea67c7519cd9e5db0c4d7e 2024-08-20T23:30:53.6878914Z deleted: sha256:c05e84a545db5dfcb0847c73d2b0f4749fa67aabce05705632ff7f76c84d4688 2024-08-20T23:30:53.6879735Z deleted: sha256:27054eae7ca67b2bd5ca0fd3f4488bf5c5f78e95487d8c45616f3d753d48a8b5 2024-08-20T23:30:53.6880566Z deleted: sha256:9804139f6ae97796aaa993fe4b89ea0cf111eb4d30a79f4c87c339c2e1e04fa0 2024-08-20T23:30:53.6881374Z deleted: sha256:ad6ae2fc594aaa359ff4e287323836570454eb24dd162930655e414d1d77c6dd 2024-08-20T23:30:53.6882180Z deleted: sha256:9cfbb36e328af52a248853a46aab88e266c09e2b44a03e0a42194a031b9b9453 2024-08-20T23:30:53.6883013Z deleted: sha256:a8a842bd7f37b8d50b6d12d6f95f46bbba57953db466fcbde9ba522625f024ef 2024-08-20T23:30:53.6883838Z deleted: sha256:da9738654b3c86f89ebeed2560f1bae093f1219f1eff90b0394b7f211ecabd04 2024-08-20T23:30:53.6884661Z deleted: sha256:cc683bf0cb245394e0f568984b2dd4fce1b6a3a7fe7c44734bae1aa1caaa3251 2024-08-20T23:30:53.6885587Z deleted: sha256:965f220e71c7d1ec19aaf41b1529ab287beb9d86eb8482c4c14094485ab34475 2024-08-20T23:30:53.6886392Z deleted: sha256:da32ecec41a73ce7dd29464f90c7951510a18cd378753659a78cd7fabd10dc36 2024-08-20T23:30:53.6887220Z deleted: sha256:d828d187cee9ad48dbc3702fb61d359619dd8a1ed582878680b51686bf117e5a 2024-08-20T23:30:53.6888030Z deleted: sha256:712111f4b12ff8e2787597d5fdcb043bd0824e55ddd97f262ef550506ca195f0 2024-08-20T23:30:53.6889021Z deleted: sha256:a064eea405958ad14dc6c794e5df9fd1d4515424272801ec559d45a59317309a 2024-08-20T23:30:53.6889824Z deleted: sha256:e124382d136bd5e9a8b409ca79c1f1ef76458deaf444b1b2060d3fa99dbeb506 2024-08-20T23:30:53.6890642Z deleted: sha256:6e10fe757cc190a1b31b47010f4f7610e02cb1b4b438f190dcd301aa4fe2278e 2024-08-20T23:30:53.6891505Z deleted: sha256:4cea352e10f46ca4729f96ac6c4600fc53d4c36e7658c2e0f19f4f309cd4a2a7 2024-08-20T23:30:53.6892439Z deleted: sha256:518bba00fe1d0933053066d21cac0235c9fb485abbc41557ec184b293095c924 2024-08-20T23:30:53.6893270Z deleted: sha256:a64f9ca42e9f27ff2690c15a5c84312e6efaccad45668d9b496cb542165cfb5a 2024-08-20T23:30:53.6894090Z deleted: sha256:096ea0fa14a6c420658f5276b8a8f5d264c0de7e18161ceda2cfc2f4a92cd9bf 2024-08-20T23:30:53.6894906Z deleted: sha256:6f91c94ce6e3f8034c91c66ee25c66a3d85dba9d813b749bd6d2000942a99edc 2024-08-20T23:30:53.6895730Z deleted: sha256:8f8d255eeae546cead8f9a94d237fa57a99940c0f5d51bc29a9d41c7df9cffd9 2024-08-20T23:30:53.6896565Z deleted: sha256:fe05276f4a49f91eef56bfbc3da87cfff5eb116a827fa2d30efe42ce1c134b40 2024-08-20T23:30:53.6897388Z deleted: sha256:5f83cd75d0669c813d4f58978c92e14305cd8f6a07eb9a60665a583460fa7b59 2024-08-20T23:30:53.6898189Z deleted: sha256:8f7ab6366169d39d8953129f650f1927d2fb1328b3f11e2fd9076adb5f46b93a 2024-08-20T23:30:53.6899123Z deleted: sha256:4146cfd5dba0f5308954749f98596b14a1d8f6d23bd93f7ad86c6c3d72239328 2024-08-20T23:30:53.6899952Z deleted: sha256:c6d83cb9ccc606ff63bb3c4d5a5e1ff1450c619b5aae494303a94ae25971cca0 2024-08-20T23:30:53.6900765Z deleted: sha256:71e037e0a3e0963c4565534a7387c082afcb04475d3ac629acf7a0ab2f49fadc 2024-08-20T23:30:53.6901558Z deleted: sha256:4b31135c564686bdaf23262b901237880aea635655721cc7e8c95dc42b05e06d 2024-08-20T23:30:53.6902361Z deleted: sha256:d09bd5c2f35cedcb846543941ee64ce1f684b52e4285696c93b3fb2032da8579 2024-08-20T23:30:53.6903186Z deleted: sha256:4de7fb7d4fc22212aae5fc1ceb637526916d0cced240d7c93d5edd6b0ea28013 2024-08-20T23:30:53.6904007Z deleted: sha256:69890e3e82746e10aff2f1084f11afb4b95a4a6c26b30f64b7001bc0a5fe32bf 2024-08-20T23:30:53.6904803Z deleted: sha256:0156df53b3224e645336ea52b4bb816f1e8d8c9b591748cd645e5e5649fa507b 2024-08-20T23:30:53.6905616Z deleted: sha256:0664624cb9cc4a386f31bf49919a8d8a11d9fea428d05585d9e6228f754bc7cd 2024-08-20T23:30:53.6906464Z deleted: sha256:8c2ddd5fa535c768c058a23f5cc4b6c64cdd71bb7458491282aabc7f7bb94d69 2024-08-20T23:30:53.6907278Z deleted: sha256:8dd03dbe0e1c5f47dd333074dff4bd00415f2f63727a65bf17866774e2ebbd0d 2024-08-20T23:30:53.6908093Z deleted: sha256:3c56b63e094bd18fad9da7a541d537210bca3b730c590d3641213259f6908d6f 2024-08-20T23:30:53.6908899Z deleted: sha256:51484fef07bd1266d4d6ff3d767751172660fc4cc6a1ffe883523a63c985e31e 2024-08-20T23:30:53.6909717Z deleted: sha256:03e83c94023982ab7dd8dc65a6df2024e342f1b760c91ec7a680ee45824869c7 2024-08-20T23:30:53.6910503Z deleted: sha256:58440ab3e8175b5de94a76485c8689a6dc186015475ebf4b386ebba309c5da34 2024-08-20T23:30:53.6911303Z deleted: sha256:9a980f3383e8aa511f0ef68982cb6dfc53b38d1ac8c3d673cb3ba3d45158c91e 2024-08-20T23:30:53.6912122Z deleted: sha256:cb0269c9cf1c47b6c4188ca7be4db261b5d3fc4b4a5153629e222dba7b1f9f98 2024-08-20T23:30:53.6912942Z deleted: sha256:41f1ba870cdffc05e0bf8d87b640fb9d4c153d7f4f3b775f37f94e9f3d584aad 2024-08-20T23:30:53.6913750Z deleted: sha256:7b502ab352f48e8b6eb5e6824a29f308952a5ee7c7d987562e1df7e47639d7f1 2024-08-20T23:30:53.6914544Z deleted: sha256:6ba5217e18738660e6ca8b7a7ec88845e70b0fd781301520bf46c71d99df3144 2024-08-20T23:30:53.6915344Z deleted: sha256:dd5a1c233594780163726bdbbf5de00f7b6b8dca1dace8a001340714c54baedb 2024-08-20T23:30:53.6916217Z deleted: sha256:c27f0db27da73e01f67f6cb78d10f355e350d829479ba4351d82055bb29f6597 2024-08-20T23:30:53.6917051Z deleted: sha256:4e3ce3c854f35c08be3dd95e0d6e718512a885e4614291d7d8b7cd23f48b7dd8 2024-08-20T23:30:53.6917938Z deleted: sha256:c2d6fb088bbec16e60cf0ad74b016f7287c92e659ec2db01271bc309b4dc6d83 2024-08-20T23:30:53.6918765Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-08-20T23:30:53.6919685Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-08-20T23:30:53.6920482Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-08-20T23:30:53.6921281Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-08-20T23:30:53.6922100Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-08-20T23:30:53.6922907Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-08-20T23:30:53.6923727Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-08-20T23:30:53.6924544Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-08-20T23:30:53.6925456Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-08-20T23:30:53.6926282Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-08-20T23:30:53.6927098Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-08-20T23:30:53.6927577Z 2024-08-20T23:30:53.6927734Z Total reclaimed space: 44.92GB 2024-08-20T23:30:53.6993079Z Post job cleanup. 2024-08-20T23:30:53.7044360Z Post job cleanup. 2024-08-20T23:30:53.7950390Z [command]/usr/bin/git version 2024-08-20T23:30:53.7980276Z git version 2.40.1 2024-08-20T23:30:53.8021371Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7620fd73-07e7-4ef5-988f-8e7ec28f0caa' before making global git config changes 2024-08-20T23:30:53.8022596Z Adding repository directory to the temporary git global config as a safe directory 2024-08-20T23:30:53.8027012Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-08-20T23:30:53.8075791Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-08-20T23:30:53.8121612Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-08-20T23:30:53.8497820Z Entering 'android/libs/fbjni' 2024-08-20T23:30:53.8566103Z Entering 'third_party/FP16' 2024-08-20T23:30:53.8633559Z Entering 'third_party/FXdiv' 2024-08-20T23:30:53.8704915Z Entering 'third_party/NNPACK' 2024-08-20T23:30:53.8773116Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:30:53.8840636Z Entering 'third_party/XNNPACK' 2024-08-20T23:30:53.8924512Z Entering 'third_party/benchmark' 2024-08-20T23:30:53.8996491Z Entering 'third_party/cpp-httplib' 2024-08-20T23:30:53.9063724Z Entering 'third_party/cpuinfo' 2024-08-20T23:30:53.9131886Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:30:53.9204148Z Entering 'third_party/cutlass' 2024-08-20T23:30:53.9280909Z Entering 'third_party/eigen' 2024-08-20T23:30:53.9350889Z Entering 'third_party/fbgemm' 2024-08-20T23:30:53.9422498Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:30:53.9490190Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:30:53.9562113Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:30:53.9637550Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:30:53.9702885Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:30:53.9773973Z Entering 'third_party/flatbuffers' 2024-08-20T23:30:53.9844843Z Entering 'third_party/fmt' 2024-08-20T23:30:53.9915423Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:30:53.9985085Z Entering 'third_party/gloo' 2024-08-20T23:30:54.0054110Z Entering 'third_party/googletest' 2024-08-20T23:30:54.0125931Z Entering 'third_party/ideep' 2024-08-20T23:30:54.0192574Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:30:54.0267238Z Entering 'third_party/ittapi' 2024-08-20T23:30:54.0335616Z Entering 'third_party/kineto' 2024-08-20T23:30:54.0402172Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:30:54.0466891Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:30:54.0536630Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:30:54.0603971Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:30:54.0672377Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:30:54.0738670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:30:54.0810554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:30:54.0877598Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:30:54.0945120Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:30:54.1014846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:30:54.1085119Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:30:54.1153072Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:30:54.1224060Z Entering 'third_party/mimalloc' 2024-08-20T23:30:54.1291343Z Entering 'third_party/nccl/nccl' 2024-08-20T23:30:54.1360527Z Entering 'third_party/nlohmann' 2024-08-20T23:30:54.1434268Z Entering 'third_party/onnx' 2024-08-20T23:30:54.1521157Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:30:54.1588675Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:30:54.1662184Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:30:54.1730825Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:30:54.1796835Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:30:54.1863511Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:30:54.1929397Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:30:54.1997747Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:30:54.2064660Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:30:54.2131014Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:30:54.2195690Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:30:54.2268718Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:30:54.2338549Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:30:54.2429016Z Entering 'third_party/pocketfft' 2024-08-20T23:30:54.2497071Z Entering 'third_party/protobuf' 2024-08-20T23:30:54.2566619Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:30:54.2633004Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:30:54.2703714Z Entering 'third_party/psimd' 2024-08-20T23:30:54.2772417Z Entering 'third_party/pthreadpool' 2024-08-20T23:30:54.2840726Z Entering 'third_party/pybind11' 2024-08-20T23:30:54.2915078Z Entering 'third_party/python-peachpy' 2024-08-20T23:30:54.2986539Z Entering 'third_party/sleef' 2024-08-20T23:30:54.3057560Z Entering 'third_party/tensorpipe' 2024-08-20T23:30:54.3124475Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:30:54.3194684Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:30:54.3259474Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:30:54.3327577Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:30:54.3392597Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:30:54.3491343Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-08-20T23:30:54.3526713Z http.https://github.com/.extraheader 2024-08-20T23:30:54.3537324Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-08-20T23:30:54.3583658Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-08-20T23:30:54.3951507Z Entering 'android/libs/fbjni' 2024-08-20T23:30:54.3995195Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4039102Z Entering 'third_party/FP16' 2024-08-20T23:30:54.4087597Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4131206Z Entering 'third_party/FXdiv' 2024-08-20T23:30:54.4174805Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4217639Z Entering 'third_party/NNPACK' 2024-08-20T23:30:54.4261579Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4306059Z Entering 'third_party/VulkanMemoryAllocator' 2024-08-20T23:30:54.4350056Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4393938Z Entering 'third_party/XNNPACK' 2024-08-20T23:30:54.4437094Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4495763Z Entering 'third_party/benchmark' 2024-08-20T23:30:54.4539202Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4583728Z Entering 'third_party/cpp-httplib' 2024-08-20T23:30:54.4627249Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4670366Z Entering 'third_party/cpuinfo' 2024-08-20T23:30:54.4713512Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4757753Z Entering 'third_party/cudnn_frontend' 2024-08-20T23:30:54.4800536Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4844651Z Entering 'third_party/cutlass' 2024-08-20T23:30:54.4888469Z http.https://github.com/.extraheader 2024-08-20T23:30:54.4941874Z Entering 'third_party/eigen' 2024-08-20T23:30:54.4984771Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5029651Z Entering 'third_party/fbgemm' 2024-08-20T23:30:54.5078303Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5120743Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-08-20T23:30:54.5164307Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5207780Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-08-20T23:30:54.5250650Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5294136Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-08-20T23:30:54.5337045Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5391530Z Entering 'third_party/fbgemm/third_party/googletest' 2024-08-20T23:30:54.5434410Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5477399Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-08-20T23:30:54.5519500Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5569316Z Entering 'third_party/flatbuffers' 2024-08-20T23:30:54.5619476Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5665237Z Entering 'third_party/fmt' 2024-08-20T23:30:54.5708876Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5751700Z Entering 'third_party/gemmlowp/gemmlowp' 2024-08-20T23:30:54.5795178Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5839576Z Entering 'third_party/gloo' 2024-08-20T23:30:54.5883226Z http.https://github.com/.extraheader 2024-08-20T23:30:54.5926897Z Entering 'third_party/googletest' 2024-08-20T23:30:54.5971396Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6016806Z Entering 'third_party/ideep' 2024-08-20T23:30:54.6061543Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6105047Z Entering 'third_party/ideep/mkl-dnn' 2024-08-20T23:30:54.6148155Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6200612Z Entering 'third_party/ittapi' 2024-08-20T23:30:54.6248714Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6293510Z Entering 'third_party/kineto' 2024-08-20T23:30:54.6337070Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6378990Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-08-20T23:30:54.6421794Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6465103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-08-20T23:30:54.6507952Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6559301Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-08-20T23:30:54.6602120Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6647537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-08-20T23:30:54.6691229Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6735849Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-08-20T23:30:54.6778514Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6822395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-08-20T23:30:54.6866803Z http.https://github.com/.extraheader 2024-08-20T23:30:54.6916452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-08-20T23:30:54.6962459Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7006295Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-08-20T23:30:54.7049834Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7095734Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-08-20T23:30:54.7138751Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7183310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-08-20T23:30:54.7226466Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7273317Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-08-20T23:30:54.7316070Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7359701Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-08-20T23:30:54.7402699Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7450941Z Entering 'third_party/mimalloc' 2024-08-20T23:30:54.7494555Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7538069Z Entering 'third_party/nccl/nccl' 2024-08-20T23:30:54.7581943Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7625240Z Entering 'third_party/nlohmann' 2024-08-20T23:30:54.7668737Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7712879Z Entering 'third_party/onnx' 2024-08-20T23:30:54.7757227Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7814982Z Entering 'third_party/onnx/third_party/benchmark' 2024-08-20T23:30:54.7858515Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7902329Z Entering 'third_party/onnx/third_party/pybind11' 2024-08-20T23:30:54.7945434Z http.https://github.com/.extraheader 2024-08-20T23:30:54.7994735Z Entering 'third_party/opentelemetry-cpp' 2024-08-20T23:30:54.8041728Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8086043Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-08-20T23:30:54.8127978Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8171362Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-08-20T23:30:54.8213368Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8256550Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-08-20T23:30:54.8298884Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8342351Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-08-20T23:30:54.8384437Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8428686Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-08-20T23:30:54.8472281Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8515492Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-08-20T23:30:54.8558108Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8601736Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-08-20T23:30:54.8648042Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8689592Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-08-20T23:30:54.8732253Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8778732Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-08-20T23:30:54.8821315Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8867933Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-08-20T23:30:54.8911387Z http.https://github.com/.extraheader 2024-08-20T23:30:54.8980375Z Entering 'third_party/pocketfft' 2024-08-20T23:30:54.9023758Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9067417Z Entering 'third_party/protobuf' 2024-08-20T23:30:54.9111301Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9156907Z Entering 'third_party/protobuf/third_party/benchmark' 2024-08-20T23:30:54.9199708Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9246704Z Entering 'third_party/protobuf/third_party/googletest' 2024-08-20T23:30:54.9289092Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9335396Z Entering 'third_party/psimd' 2024-08-20T23:30:54.9381817Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9425277Z Entering 'third_party/pthreadpool' 2024-08-20T23:30:54.9469812Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9513647Z Entering 'third_party/pybind11' 2024-08-20T23:30:54.9559652Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9603778Z Entering 'third_party/python-peachpy' 2024-08-20T23:30:54.9646657Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9691023Z Entering 'third_party/sleef' 2024-08-20T23:30:54.9737079Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9782039Z Entering 'third_party/tensorpipe' 2024-08-20T23:30:54.9824819Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9869070Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-08-20T23:30:54.9912283Z http.https://github.com/.extraheader 2024-08-20T23:30:54.9956267Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-08-20T23:30:54.9998685Z http.https://github.com/.extraheader 2024-08-20T23:30:55.0042712Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-08-20T23:30:55.0085023Z http.https://github.com/.extraheader 2024-08-20T23:30:55.0128132Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-08-20T23:30:55.0171695Z http.https://github.com/.extraheader 2024-08-20T23:30:55.0214211Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-08-20T23:30:55.0257629Z http.https://github.com/.extraheader 2024-08-20T23:30:55.0416156Z A job completed hook has been configured by the self-hosted runner administrator 2024-08-20T23:30:55.0437791Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2024-08-20T23:30:55.0446431Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-08-20T23:30:55.0446953Z ##[endgroup] 2024-08-20T23:31:01.9003315Z Cleaning up orphan processes