2024-06-26T06:59:45.7945746Z Current runner version: '2.317.0' 2024-06-26T06:59:45.7952414Z Runner name: 'i-01e2eadbbc5643a4a' 2024-06-26T06:59:45.7953255Z Runner group name: 'Default' 2024-06-26T06:59:45.7954228Z Machine name: 'ip-10-0-34-11' 2024-06-26T06:59:45.7959140Z ##[group]GITHUB_TOKEN Permissions 2024-06-26T06:59:45.7961122Z Actions: read 2024-06-26T06:59:45.7961712Z Attestations: read 2024-06-26T06:59:45.7962278Z Checks: read 2024-06-26T06:59:45.7962772Z Contents: read 2024-06-26T06:59:45.7963338Z Deployments: read 2024-06-26T06:59:45.7963937Z Discussions: read 2024-06-26T06:59:45.7964485Z Issues: read 2024-06-26T06:59:45.7965008Z Metadata: read 2024-06-26T06:59:45.7965537Z Packages: read 2024-06-26T06:59:45.7966211Z Pages: read 2024-06-26T06:59:45.7966807Z PullRequests: read 2024-06-26T06:59:45.7967380Z RepositoryProjects: read 2024-06-26T06:59:45.7968020Z SecurityEvents: read 2024-06-26T06:59:45.7968608Z Statuses: read 2024-06-26T06:59:45.7969116Z ##[endgroup] 2024-06-26T06:59:45.7972566Z Secret source: Actions 2024-06-26T06:59:45.7973350Z Prepare workflow directory 2024-06-26T06:59:46.1464396Z Prepare all required actions 2024-06-26T06:59:46.1655207Z Getting action download info 2024-06-26T06:59:46.3861751Z Download action repository 'pytorch/test-infra@main' (SHA:43a2ce341cc31288e9a38b65ce600a7f43021bd5) 2024-06-26T06:59:46.7257731Z Download action repository 'pytorch/pytorch@main' (SHA:c422a9549d2b320b47a9391f6cbcf592b2876c88) 2024-06-26T06:59:49.3965963Z Download action repository 'aws-actions/configure-aws-credentials@v3' (SHA:50ac8dd1e1b10d09dac7b8727528b91bed831ac0) 2024-06-26T06:59:49.5303785Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2024-06-26T06:59:49.8119630Z Getting action download info 2024-06-26T06:59:49.9181291Z Download action repository 'malfet/checkout@silent-checkout' (SHA:e07af140b3ccefc05679e3755b9db68f4ee4589c) 2024-06-26T06:59:50.0896469Z Getting action download info 2024-06-26T06:59:50.1831828Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2024-06-26T06:59:50.3053198Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/inductor/129470 (b8c4c54d347aa776934c60784e35936878ef18dc) 2024-06-26T06:59:50.3055407Z ##[group] Inputs 2024-06-26T06:59:50.3055883Z build-environment: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T06:59:50.3064437Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T06:59:50.3073815Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T06:59:50.3074983Z sync-tag: 2024-06-26T06:59:50.3075838Z timeout-minutes: 240 2024-06-26T06:59:50.3076180Z use-gha: 2024-06-26T06:59:50.3076465Z dashboard-tag: 2024-06-26T06:59:50.3076787Z s3-bucket: gha-artifacts 2024-06-26T06:59:50.3077142Z aws-role-to-assume: 2024-06-26T06:59:50.3077475Z ##[endgroup] 2024-06-26T06:59:50.3078358Z Complete job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T06:59:50.3649628Z A job started hook has been configured by the self-hosted runner administrator 2024-06-26T06:59:50.3789762Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T06:59:50.3800089Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:59:50.3800611Z ##[endgroup] 2024-06-26T06:59:50.9696999Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2024-06-26T06:59:50.9697553Z with: 2024-06-26T06:59:50.9698164Z github-secret: *** 2024-06-26T06:59:50.9699110Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2024-06-26T06:59:50.9700099Z activate-with-label: false 2024-06-26T06:59:50.9700465Z label: with-ssh 2024-06-26T06:59:50.9700785Z remove-existing-keys: true 2024-06-26T06:59:50.9701139Z fail-silently: true 2024-06-26T06:59:50.9701450Z env: 2024-06-26T06:59:50.9701739Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:50.9702079Z ##[endgroup] 2024-06-26T06:59:51.0592764Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2024-06-26T06:59:51.0593776Z ciflow reference detected, attempting to extract PR number 2024-06-26T06:59:51.3139299Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys 2024-06-26T06:59:51.3777706Z No SSH keys found for user pytorch-bot[bot] 2024-06-26T06:59:51.3778855Z Grabbing public ssh keys from https://github.com/leslie-fang-intel.keys 2024-06-26T06:59:51.4457579Z No SSH keys found for user leslie-fang-intel 2024-06-26T06:59:51.4565714Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2024-06-26T06:59:51.4566384Z with: 2024-06-26T06:59:51.4566673Z submodules: recursive 2024-06-26T06:59:51.4567003Z fetch-depth: 0 2024-06-26T06:59:51.4567297Z env: 2024-06-26T06:59:51.4567570Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:51.4567907Z ##[endgroup] 2024-06-26T06:59:51.4756083Z ##[group]Run retry () { 2024-06-26T06:59:51.4756504Z retry () { 2024-06-26T06:59:51.4757096Z  $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*) 2024-06-26T06:59:51.4757723Z } 2024-06-26T06:59:51.4758110Z echo "${GITHUB_WORKSPACE}" 2024-06-26T06:59:51.4758564Z if [ -z "${NO_SUDO}" ]; then 2024-06-26T06:59:51.4759102Z  retry sudo rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:59:51.4759627Z else 2024-06-26T06:59:51.4760046Z  retry rm -rf "${GITHUB_WORKSPACE}" 2024-06-26T06:59:51.4760506Z fi 2024-06-26T06:59:51.4760953Z mkdir "${GITHUB_WORKSPACE}" 2024-06-26T06:59:51.4769163Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T06:59:51.4769707Z env: 2024-06-26T06:59:51.4770035Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:51.4770457Z NO_SUDO: 2024-06-26T06:59:51.4770808Z ##[endgroup] 2024-06-26T06:59:51.4793952Z /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:53.6642799Z ##[group]Run malfet/checkout@silent-checkout 2024-06-26T06:59:53.6643245Z with: 2024-06-26T06:59:53.6643581Z ref: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T06:59:53.6644024Z fetch-depth: 0 2024-06-26T06:59:53.6644348Z submodules: recursive 2024-06-26T06:59:53.6644703Z quiet-checkout: true 2024-06-26T06:59:53.6645056Z repository: pytorch/pytorch 2024-06-26T06:59:53.6645534Z token: *** 2024-06-26T06:59:53.6645832Z ssh-strict: true 2024-06-26T06:59:53.6646390Z persist-credentials: true 2024-06-26T06:59:53.6646750Z clean: true 2024-06-26T06:59:53.6647082Z sparse-checkout-cone-mode: true 2024-06-26T06:59:53.6647470Z lfs: false 2024-06-26T06:59:53.6647775Z set-safe-directory: true 2024-06-26T06:59:53.6648110Z env: 2024-06-26T06:59:53.6648385Z GIT_DEFAULT_BRANCH: main 2024-06-26T06:59:53.6648729Z ##[endgroup] 2024-06-26T06:59:53.7576965Z Syncing repository: pytorch/pytorch 2024-06-26T06:59:53.7578511Z ##[group]Getting Git version info 2024-06-26T06:59:53.7579305Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:59:53.7580145Z [command]/usr/bin/git version 2024-06-26T06:59:53.7580517Z git version 2.40.1 2024-06-26T06:59:53.7589031Z ##[endgroup] 2024-06-26T06:59:53.7603823Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/c3b8fd66-50ec-49cf-82df-fc4bb444ed2a' before making global git config changes 2024-06-26T06:59:53.7605068Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T06:59:53.7609906Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:53.7641616Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2024-06-26T06:59:53.7645636Z ##[group]Initializing the repository 2024-06-26T06:59:53.7648957Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T06:59:53.7679771Z hint: Using 'master' as the name for the initial branch. This default branch name 2024-06-26T06:59:53.7680900Z hint: is subject to change. To configure the initial branch name to use in all 2024-06-26T06:59:53.7681746Z hint: of your new repositories, which will suppress this warning, call: 2024-06-26T06:59:53.7682500Z hint: 2024-06-26T06:59:53.7682977Z hint: git config --global init.defaultBranch 2024-06-26T06:59:53.7683465Z hint: 2024-06-26T06:59:53.7683993Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2024-06-26T06:59:53.7684840Z hint: 'development'. The just-created branch can be renamed via this command: 2024-06-26T06:59:53.7685792Z hint: 2024-06-26T06:59:53.7686358Z hint: git branch -m 2024-06-26T06:59:53.7687110Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2024-06-26T06:59:53.7689754Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2024-06-26T06:59:53.7716639Z ##[endgroup] 2024-06-26T06:59:53.7717416Z ##[group]Disabling automatic garbage collection 2024-06-26T06:59:53.7720201Z [command]/usr/bin/git config --local gc.auto 0 2024-06-26T06:59:53.7740343Z ##[endgroup] 2024-06-26T06:59:53.7740912Z ##[group]Setting up auth 2024-06-26T06:59:53.7746228Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T06:59:53.7766616Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T06:59:53.7962215Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T06:59:53.7990756Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T06:59:53.8191710Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T06:59:53.8228674Z ##[endgroup] 2024-06-26T06:59:53.8229451Z ##[group]Fetching the repository 2024-06-26T06:59:53.8236475Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --progress --no-recurse-submodules --quiet origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2024-06-26T06:59:56.0386112Z remote: Enumerating objects: 987619 2024-06-26T06:59:56.0386920Z remote: Enumerating objects: 990209, done. 2024-06-26T06:59:56.0388376Z remote: Counting objects: 0% (1/2590) 2024-06-26T06:59:56.0393981Z remote: Counting objects: 1% (26/2590) 2024-06-26T06:59:56.0394726Z remote: Counting objects: 2% (52/2590) 2024-06-26T06:59:56.0395384Z remote: Counting objects: 3% (78/2590) 2024-06-26T06:59:56.0396149Z remote: Counting objects: 4% (104/2590) 2024-06-26T06:59:56.0396908Z remote: Counting objects: 5% (130/2590) 2024-06-26T06:59:56.0397694Z remote: Counting objects: 6% (156/2590) 2024-06-26T06:59:56.0398479Z remote: Counting objects: 7% (182/2590) 2024-06-26T06:59:56.0399269Z remote: Counting objects: 8% (208/2590) 2024-06-26T06:59:56.0400044Z remote: Counting objects: 9% (234/2590) 2024-06-26T06:59:56.0400834Z remote: Counting objects: 10% (259/2590) 2024-06-26T06:59:56.0401471Z remote: Counting objects: 11% (285/2590) 2024-06-26T06:59:56.0402272Z remote: Counting objects: 12% (311/2590) 2024-06-26T06:59:56.0403071Z remote: Counting objects: 13% (337/2590) 2024-06-26T06:59:56.0403837Z remote: Counting objects: 14% (363/2590) 2024-06-26T06:59:56.0404510Z remote: Counting objects: 15% (389/2590) 2024-06-26T06:59:56.0405304Z remote: Counting objects: 16% (415/2590) 2024-06-26T06:59:56.0406199Z remote: Counting objects: 17% (441/2590) 2024-06-26T06:59:56.0406930Z remote: Counting objects: 18% (467/2590) 2024-06-26T06:59:56.0407698Z remote: Counting objects: 19% (493/2590) 2024-06-26T06:59:56.0408391Z remote: Counting objects: 20% (518/2590) 2024-06-26T06:59:56.0409140Z remote: Counting objects: 21% (544/2590) 2024-06-26T06:59:56.0409913Z remote: Counting objects: 22% (570/2590) 2024-06-26T06:59:56.0410532Z remote: Counting objects: 23% (596/2590) 2024-06-26T06:59:56.0411304Z remote: Counting objects: 24% (622/2590) 2024-06-26T06:59:56.0411963Z remote: Counting objects: 25% (648/2590) 2024-06-26T06:59:56.0412746Z remote: Counting objects: 26% (674/2590) 2024-06-26T06:59:56.0413379Z remote: Counting objects: 27% (700/2590) 2024-06-26T06:59:56.0414162Z remote: Counting objects: 28% (726/2590) 2024-06-26T06:59:56.0414673Z remote: Counting objects: 29% (752/2590) 2024-06-26T06:59:56.0415170Z remote: Counting objects: 30% (777/2590) 2024-06-26T06:59:56.0415660Z remote: Counting objects: 31% (803/2590) 2024-06-26T06:59:56.0416155Z remote: Counting objects: 32% (829/2590) 2024-06-26T06:59:56.0416665Z remote: Counting objects: 33% (855/2590) 2024-06-26T06:59:56.0417160Z remote: Counting objects: 34% (881/2590) 2024-06-26T06:59:56.0417654Z remote: Counting objects: 35% (907/2590) 2024-06-26T06:59:56.0418163Z remote: Counting objects: 36% (933/2590) 2024-06-26T06:59:56.0421139Z remote: Counting objects: 37% (959/2590) 2024-06-26T06:59:56.0421879Z remote: Counting objects: 38% (985/2590) 2024-06-26T06:59:56.0422609Z remote: Counting objects: 39% (1011/2590) 2024-06-26T06:59:56.0423348Z remote: Counting objects: 40% (1036/2590) 2024-06-26T06:59:56.0424046Z remote: Counting objects: 41% (1062/2590) 2024-06-26T06:59:56.0424842Z remote: Counting objects: 42% (1088/2590) 2024-06-26T06:59:56.0425517Z remote: Counting objects: 43% (1114/2590) 2024-06-26T06:59:56.0426132Z remote: Counting objects: 44% (1140/2590) 2024-06-26T06:59:56.0426875Z remote: Counting objects: 45% (1166/2590) 2024-06-26T06:59:56.0427852Z remote: Counting objects: 46% (1192/2590) 2024-06-26T06:59:56.0428756Z remote: Counting objects: 47% (1218/2590) 2024-06-26T06:59:56.0429515Z remote: Counting objects: 48% (1244/2590) 2024-06-26T06:59:56.0430233Z remote: Counting objects: 49% (1270/2590) 2024-06-26T06:59:56.0430807Z remote: Counting objects: 50% (1295/2590) 2024-06-26T06:59:56.0431329Z remote: Counting objects: 51% (1321/2590) 2024-06-26T06:59:56.0431945Z remote: Counting objects: 52% (1347/2590) 2024-06-26T06:59:56.0432570Z remote: Counting objects: 53% (1373/2590) 2024-06-26T06:59:56.0433156Z remote: Counting objects: 54% (1399/2590) 2024-06-26T06:59:56.0433677Z remote: Counting objects: 55% (1425/2590) 2024-06-26T06:59:56.0434180Z remote: Counting objects: 56% (1451/2590) 2024-06-26T06:59:56.0434712Z remote: Counting objects: 57% (1477/2590) 2024-06-26T06:59:56.0435389Z remote: Counting objects: 58% (1503/2590) 2024-06-26T06:59:56.0435921Z remote: Counting objects: 59% (1529/2590) 2024-06-26T06:59:56.0436424Z remote: Counting objects: 60% (1554/2590) 2024-06-26T06:59:56.0436932Z remote: Counting objects: 61% (1580/2590) 2024-06-26T06:59:56.0437440Z remote: Counting objects: 62% (1606/2590) 2024-06-26T06:59:56.0437973Z remote: Counting objects: 63% (1632/2590) 2024-06-26T06:59:56.0438600Z remote: Counting objects: 64% (1658/2590) 2024-06-26T06:59:56.0439282Z remote: Counting objects: 65% (1684/2590) 2024-06-26T06:59:56.0439845Z remote: Counting objects: 66% (1710/2590) 2024-06-26T06:59:56.0440363Z remote: Counting objects: 67% (1736/2590) 2024-06-26T06:59:56.0440877Z remote: Counting objects: 68% (1762/2590) 2024-06-26T06:59:56.0441382Z remote: Counting objects: 69% (1788/2590) 2024-06-26T06:59:56.0441896Z remote: Counting objects: 70% (1813/2590) 2024-06-26T06:59:56.0442462Z remote: Counting objects: 71% (1839/2590) 2024-06-26T06:59:56.0442989Z remote: Counting objects: 72% (1865/2590) 2024-06-26T06:59:56.0443609Z remote: Counting objects: 73% (1891/2590) 2024-06-26T06:59:56.0444271Z remote: Counting objects: 74% (1917/2590) 2024-06-26T06:59:56.0444790Z remote: Counting objects: 75% (1943/2590) 2024-06-26T06:59:56.0445301Z remote: Counting objects: 76% (1969/2590) 2024-06-26T06:59:56.0445810Z remote: Counting objects: 77% (1995/2590) 2024-06-26T06:59:56.0446498Z remote: Counting objects: 78% (2021/2590) 2024-06-26T06:59:56.0447131Z remote: Counting objects: 79% (2047/2590) 2024-06-26T06:59:56.0447643Z remote: Counting objects: 80% (2072/2590) 2024-06-26T06:59:56.0448141Z remote: Counting objects: 81% (2098/2590) 2024-06-26T06:59:56.0448646Z remote: Counting objects: 82% (2124/2590) 2024-06-26T06:59:56.0449156Z remote: Counting objects: 83% (2150/2590) 2024-06-26T06:59:56.0449669Z remote: Counting objects: 84% (2176/2590) 2024-06-26T06:59:56.0450170Z remote: Counting objects: 85% (2202/2590) 2024-06-26T06:59:56.0450677Z remote: Counting objects: 86% (2228/2590) 2024-06-26T06:59:56.0451187Z remote: Counting objects: 87% (2254/2590) 2024-06-26T06:59:56.0451692Z remote: Counting objects: 88% (2280/2590) 2024-06-26T06:59:56.0452194Z remote: Counting objects: 89% (2306/2590) 2024-06-26T06:59:56.0452701Z remote: Counting objects: 90% (2331/2590) 2024-06-26T06:59:56.0453208Z remote: Counting objects: 91% (2357/2590) 2024-06-26T06:59:56.0453720Z remote: Counting objects: 92% (2383/2590) 2024-06-26T06:59:56.0454222Z remote: Counting objects: 93% (2409/2590) 2024-06-26T06:59:56.0454728Z remote: Counting objects: 94% (2435/2590) 2024-06-26T06:59:56.0455237Z remote: Counting objects: 95% (2461/2590) 2024-06-26T06:59:56.0455741Z remote: Counting objects: 96% (2487/2590) 2024-06-26T06:59:56.0456315Z remote: Counting objects: 97% (2513/2590) 2024-06-26T06:59:56.0456822Z remote: Counting objects: 98% (2539/2590) 2024-06-26T06:59:56.0457331Z remote: Counting objects: 99% (2565/2590) 2024-06-26T06:59:56.0457841Z remote: Counting objects: 100% (2590/2590) 2024-06-26T06:59:56.0458377Z remote: Counting objects: 100% (2590/2590), done. 2024-06-26T06:59:56.0572618Z remote: Compressing objects: 0% (1/1068) 2024-06-26T06:59:56.0981192Z remote: Compressing objects: 1% (11/1068) 2024-06-26T06:59:56.1500585Z remote: Compressing objects: 2% (22/1068) 2024-06-26T06:59:56.1901104Z remote: Compressing objects: 3% (33/1068) 2024-06-26T06:59:56.2200966Z remote: Compressing objects: 4% (43/1068) 2024-06-26T06:59:56.2468884Z remote: Compressing objects: 5% (54/1068) 2024-06-26T06:59:56.2687778Z remote: Compressing objects: 6% (65/1068) 2024-06-26T06:59:56.3113911Z remote: Compressing objects: 7% (75/1068) 2024-06-26T06:59:56.4043660Z remote: Compressing objects: 8% (86/1068) 2024-06-26T06:59:56.4848053Z remote: Compressing objects: 9% (97/1068) 2024-06-26T06:59:56.5096878Z remote: Compressing objects: 10% (107/1068) 2024-06-26T06:59:56.5282437Z remote: Compressing objects: 11% (118/1068) 2024-06-26T06:59:56.5389449Z remote: Compressing objects: 12% (129/1068) 2024-06-26T06:59:56.5390017Z remote: Compressing objects: 13% (139/1068) 2024-06-26T06:59:56.5517331Z remote: Compressing objects: 14% (150/1068) 2024-06-26T06:59:56.5518485Z remote: Compressing objects: 15% (161/1068) 2024-06-26T06:59:56.5586079Z remote: Compressing objects: 16% (171/1068) 2024-06-26T06:59:56.5586697Z remote: Compressing objects: 17% (182/1068) 2024-06-26T06:59:56.5587228Z remote: Compressing objects: 18% (193/1068) 2024-06-26T06:59:56.5604014Z remote: Compressing objects: 19% (203/1068) 2024-06-26T06:59:56.5626211Z remote: Compressing objects: 20% (214/1068) 2024-06-26T06:59:56.5643218Z remote: Compressing objects: 21% (225/1068) 2024-06-26T06:59:56.5663105Z remote: Compressing objects: 22% (235/1068) 2024-06-26T06:59:56.5689319Z remote: Compressing objects: 23% (246/1068) 2024-06-26T06:59:56.5704185Z remote: Compressing objects: 24% (257/1068) 2024-06-26T06:59:56.5711747Z remote: Compressing objects: 25% (267/1068) 2024-06-26T06:59:56.5721904Z remote: Compressing objects: 26% (278/1068) 2024-06-26T06:59:56.5729002Z remote: Compressing objects: 27% (289/1068) 2024-06-26T06:59:56.5734613Z remote: Compressing objects: 28% (300/1068) 2024-06-26T06:59:56.5745730Z remote: Compressing objects: 29% (310/1068) 2024-06-26T06:59:56.5753389Z remote: Compressing objects: 30% (321/1068) 2024-06-26T06:59:56.5761896Z remote: Compressing objects: 31% (332/1068) 2024-06-26T06:59:56.5768626Z remote: Compressing objects: 32% (342/1068) 2024-06-26T06:59:56.5782378Z remote: Compressing objects: 33% (353/1068) 2024-06-26T06:59:56.5797833Z remote: Compressing objects: 34% (364/1068) 2024-06-26T06:59:56.5816775Z remote: Compressing objects: 35% (374/1068) 2024-06-26T06:59:56.5832887Z remote: Compressing objects: 36% (385/1068) 2024-06-26T06:59:56.5849930Z remote: Compressing objects: 37% (396/1068) 2024-06-26T06:59:56.5868862Z remote: Compressing objects: 38% (406/1068) 2024-06-26T06:59:56.5884628Z remote: Compressing objects: 39% (417/1068) 2024-06-26T06:59:56.5902448Z remote: Compressing objects: 40% (428/1068) 2024-06-26T06:59:56.5921047Z remote: Compressing objects: 41% (438/1068) 2024-06-26T06:59:56.5938230Z remote: Compressing objects: 42% (449/1068) 2024-06-26T06:59:56.5953010Z remote: Compressing objects: 43% (460/1068) 2024-06-26T06:59:56.5967660Z remote: Compressing objects: 44% (470/1068) 2024-06-26T06:59:56.5968768Z remote: Compressing objects: 45% (481/1068) 2024-06-26T06:59:56.5969785Z remote: Compressing objects: 46% (492/1068) 2024-06-26T06:59:56.5991043Z remote: Compressing objects: 47% (502/1068) 2024-06-26T06:59:56.6007402Z remote: Compressing objects: 48% (513/1068) 2024-06-26T06:59:56.6008930Z remote: Compressing objects: 49% (524/1068) 2024-06-26T06:59:56.6016960Z remote: Compressing objects: 50% (534/1068) 2024-06-26T06:59:56.6027053Z remote: Compressing objects: 51% (545/1068) 2024-06-26T06:59:56.6041169Z remote: Compressing objects: 52% (556/1068) 2024-06-26T06:59:56.6063217Z remote: Compressing objects: 53% (567/1068) 2024-06-26T06:59:56.6075464Z remote: Compressing objects: 54% (577/1068) 2024-06-26T06:59:56.6086285Z remote: Compressing objects: 55% (588/1068) 2024-06-26T06:59:56.6097319Z remote: Compressing objects: 56% (599/1068) 2024-06-26T06:59:56.6114497Z remote: Compressing objects: 57% (609/1068) 2024-06-26T06:59:56.6126793Z remote: Compressing objects: 58% (620/1068) 2024-06-26T06:59:56.6137045Z remote: Compressing objects: 59% (631/1068) 2024-06-26T06:59:56.6148493Z remote: Compressing objects: 60% (641/1068) 2024-06-26T06:59:56.6163112Z remote: Compressing objects: 61% (652/1068) 2024-06-26T06:59:56.6169189Z remote: Compressing objects: 62% (663/1068) 2024-06-26T06:59:56.6177554Z remote: Compressing objects: 63% (673/1068) 2024-06-26T06:59:56.6186707Z remote: Compressing objects: 64% (684/1068) 2024-06-26T06:59:56.6194229Z remote: Compressing objects: 65% (695/1068) 2024-06-26T06:59:56.6196645Z remote: Compressing objects: 66% (705/1068) 2024-06-26T06:59:56.6207587Z remote: Compressing objects: 67% (716/1068) 2024-06-26T06:59:56.6212847Z remote: Compressing objects: 68% (727/1068) 2024-06-26T06:59:56.6216667Z remote: Compressing objects: 69% (737/1068) 2024-06-26T06:59:56.6219856Z remote: Compressing objects: 70% (748/1068) 2024-06-26T06:59:56.6223044Z remote: Compressing objects: 71% (759/1068) 2024-06-26T06:59:56.6225856Z remote: Compressing objects: 72% (769/1068) 2024-06-26T06:59:56.6226976Z remote: Compressing objects: 73% (780/1068) 2024-06-26T06:59:56.6229906Z remote: Compressing objects: 74% (791/1068) 2024-06-26T06:59:56.6230618Z remote: Compressing objects: 75% (801/1068) 2024-06-26T06:59:56.6231241Z remote: Compressing objects: 76% (812/1068) 2024-06-26T06:59:56.6231894Z remote: Compressing objects: 77% (823/1068) 2024-06-26T06:59:56.6232435Z remote: Compressing objects: 78% (834/1068) 2024-06-26T06:59:56.6235123Z remote: Compressing objects: 79% (844/1068) 2024-06-26T06:59:56.6235861Z remote: Compressing objects: 80% (855/1068) 2024-06-26T06:59:56.6238193Z remote: Compressing objects: 81% (866/1068) 2024-06-26T06:59:56.6241245Z remote: Compressing objects: 82% (876/1068) 2024-06-26T06:59:56.6242234Z remote: Compressing objects: 83% (887/1068) 2024-06-26T06:59:56.6244288Z remote: Compressing objects: 84% (898/1068) 2024-06-26T06:59:56.6247869Z remote: Compressing objects: 85% (908/1068) 2024-06-26T06:59:56.6249424Z remote: Compressing objects: 86% (919/1068) 2024-06-26T06:59:56.6251767Z remote: Compressing objects: 87% (930/1068) 2024-06-26T06:59:56.6253621Z remote: Compressing objects: 88% (940/1068) 2024-06-26T06:59:56.6257029Z remote: Compressing objects: 89% (951/1068) 2024-06-26T06:59:56.6259177Z remote: Compressing objects: 90% (962/1068) 2024-06-26T06:59:56.6260762Z remote: Compressing objects: 91% (972/1068) 2024-06-26T06:59:56.6262721Z remote: Compressing objects: 92% (983/1068) 2024-06-26T06:59:56.6263432Z remote: Compressing objects: 93% (994/1068) 2024-06-26T06:59:56.6264354Z remote: Compressing objects: 94% (1004/1068) 2024-06-26T06:59:56.6267193Z remote: Compressing objects: 95% (1015/1068) 2024-06-26T06:59:56.6267924Z remote: Compressing objects: 96% (1026/1068) 2024-06-26T06:59:56.6268956Z remote: Compressing objects: 97% (1036/1068) 2024-06-26T06:59:56.6270412Z remote: Compressing objects: 98% (1047/1068) 2024-06-26T06:59:56.6271436Z remote: Compressing objects: 99% (1058/1068) 2024-06-26T06:59:56.6272502Z remote: Compressing objects: 100% (1068/1068) 2024-06-26T06:59:56.6273214Z remote: Compressing objects: 100% (1068/1068), done. 2024-06-26T07:00:17.1382165Z remote: Total 990209 (delta 2055), reused 1861 (delta 1521), pack-reused 987619 2024-06-26T07:00:41.9493786Z [command]/usr/bin/git rev-parse --verify --quiet b8c4c54d347aa776934c60784e35936878ef18dc^{object} 2024-06-26T07:00:41.9514629Z b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:41.9518953Z ##[endgroup] 2024-06-26T07:00:41.9519516Z ##[group]Determining the checkout info 2024-06-26T07:00:41.9522033Z ##[endgroup] 2024-06-26T07:00:41.9522608Z ##[group]Checking out the ref 2024-06-26T07:00:41.9523788Z [command]/usr/bin/git checkout --quiet --force b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:00:43.1740213Z ##[endgroup] 2024-06-26T07:00:43.1740820Z ##[group]Setting up auth for fetching submodules 2024-06-26T07:00:43.1746328Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2024-06-26T07:00:43.1790282Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2024-06-26T07:00:43.1811511Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2024-06-26T07:00:43.1833119Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2024-06-26T07:00:43.1850309Z ##[endgroup] 2024-06-26T07:00:43.1850833Z ##[group]Fetching submodules 2024-06-26T07:00:43.1854100Z [command]/usr/bin/git submodule sync --recursive 2024-06-26T07:00:43.2055864Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2024-06-26T07:00:43.2255933Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2024-06-26T07:00:43.2257321Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2024-06-26T07:00:43.2258698Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2024-06-26T07:00:43.2260261Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2024-06-26T07:00:43.2262067Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2024-06-26T07:00:43.2264166Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2024-06-26T07:00:43.2265667Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2024-06-26T07:00:43.2268358Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2024-06-26T07:00:43.2271756Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2024-06-26T07:00:43.2274065Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2024-06-26T07:00:43.2275907Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2024-06-26T07:00:43.2278581Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen' 2024-06-26T07:00:43.2281192Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2024-06-26T07:00:43.2283693Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2024-06-26T07:00:43.2286166Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2024-06-26T07:00:43.2288933Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi' 2024-06-26T07:00:43.2291896Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:00:43.2294270Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo' 2024-06-26T07:00:43.2297177Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2024-06-26T07:00:43.2299950Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2024-06-26T07:00:43.2303147Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2024-06-26T07:00:43.2306091Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2024-06-26T07:00:43.2309700Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2024-06-26T07:00:43.2312846Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl' 2024-06-26T07:00:43.2316110Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2024-06-26T07:00:43.2319293Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2024-06-26T07:00:43.2322897Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2024-06-26T07:00:43.2326289Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2024-06-26T07:00:43.2329851Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2024-06-26T07:00:43.2333471Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2024-06-26T07:00:43.2337441Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2024-06-26T07:00:43.2340948Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2024-06-26T07:00:43.2344668Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2024-06-26T07:00:43.2349528Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2024-06-26T07:00:43.2353643Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2024-06-26T07:00:43.2373004Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2024-06-26T07:00:43.4999304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2024-06-26T07:00:43.6579366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2024-06-26T07:00:43.8249891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2024-06-26T07:00:44.0849017Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2024-06-26T07:00:46.1621311Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2024-06-26T07:00:55.9962407Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2024-06-26T07:00:56.3840178Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2024-06-26T07:00:56.8687559Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2024-06-26T07:00:57.4448087Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2024-06-26T07:00:58.6224381Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2024-06-26T07:01:00.3384828Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'... 2024-06-26T07:01:06.0408646Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2024-06-26T07:01:07.3402476Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2024-06-26T07:01:08.9374697Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2024-06-26T07:01:10.1704955Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'... 2024-06-26T07:01:10.3709393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2024-06-26T07:01:10.7208507Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2024-06-26T07:01:10.9957304Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2024-06-26T07:01:11.9266868Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2024-06-26T07:01:12.2792900Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2024-06-26T07:01:12.5254561Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2024-06-26T07:01:14.0149961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2024-06-26T07:01:14.7090289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'... 2024-06-26T07:01:15.6281061Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2024-06-26T07:01:21.1903879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2024-06-26T07:01:23.4755932Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2024-06-26T07:01:28.9893199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2024-06-26T07:01:29.2034199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2024-06-26T07:01:37.3013382Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2024-06-26T07:01:37.4598096Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2024-06-26T07:01:37.6371394Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2024-06-26T07:01:38.6182759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2024-06-26T07:01:38.8762989Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2024-06-26T07:01:39.4902921Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2024-06-26T07:01:39.8739508Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2024-06-26T07:01:39.8825059Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2024-06-26T07:01:39.8885167Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2024-06-26T07:01:39.9069824Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2024-06-26T07:01:39.9372078Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191' 2024-06-26T07:01:40.7048658Z Submodule path 'third_party/XNNPACK': checked out 'fcbf55af6cf28a4627bcd1f703ab7ad843f0f3a2' 2024-06-26T07:01:40.7217576Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415' 2024-06-26T07:01:40.7571435Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006' 2024-06-26T07:01:40.8417920Z Submodule path 'third_party/cpuinfo': checked out '3c8b1533ac03dd6531ab6e7b9245d488f13a82a5' 2024-06-26T07:01:40.8684805Z Submodule path 'third_party/cudnn_frontend': checked out 'aa3abd4bc689d6412979c7f55f9cd132848c9c6a' 2024-06-26T07:01:41.2962276Z Submodule path 'third_party/cutlass': checked out 'bbe579a9e3beb6ea6626d9227ec32d0dae119a49' 2024-06-26T07:01:41.5101386Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c' 2024-06-26T07:01:41.5737535Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e' 2024-06-26T07:01:41.5748904Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:01:41.5750867Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:01:41.5752579Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:01:41.5754216Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:01:41.5756527Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:01:41.5774750Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'... 2024-06-26T07:01:42.5353499Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'... 2024-06-26T07:01:43.0943067Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cutlass'... 2024-06-26T07:01:44.8105443Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'... 2024-06-26T07:01:45.7573942Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'... 2024-06-26T07:01:46.0765631Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81' 2024-06-26T07:01:46.1603537Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3' 2024-06-26T07:01:46.5008877Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6' 2024-06-26T07:01:46.5549433Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796' 2024-06-26T07:01:46.5636924Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1' 2024-06-26T07:01:46.6565048Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b' 2024-06-26T07:01:46.6846934Z Submodule path 'third_party/fmt': checked out 'e69e5f977d458f2650bb346dadf2ad30c5320281' 2024-06-26T07:01:46.6909081Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad' 2024-06-26T07:01:46.7212726Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2024-06-26T07:01:46.7413835Z Submodule path 'third_party/gloo': checked out '5354032ea08eadd7fc4456477f7f7c6308818509' 2024-06-26T07:01:46.7779920Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:01:46.7878237Z Submodule path 'third_party/ideep': checked out '55ca0191687aaf19aca5cdb7881c791e3bea442b' 2024-06-26T07:01:46.7888743Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2024-06-26T07:01:46.7904114Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2024-06-26T07:01:59.0114648Z Submodule path 'third_party/ideep/mkl-dnn': checked out '1137e04ec0b5251ca2b4400a4fd3c667ce843d67' 2024-06-26T07:01:59.0252229Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42' 2024-06-26T07:01:59.1029651Z Submodule path 'third_party/kineto': checked out '8681ff11e1fa54da39023076c5c43eddd87b7a8a' 2024-06-26T07:01:59.1040450Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:01:59.1042116Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:01:59.1043954Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:01:59.1062250Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2024-06-26T07:01:59.6077316Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2024-06-26T07:02:00.8442516Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2024-06-26T07:02:01.9096287Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2024-06-26T07:02:01.9108350Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:01.9110398Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:01.9112663Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:01.9114767Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:01.9116665Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:01.9118964Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:01.9120879Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:01.9123106Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:01.9141594Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2024-06-26T07:02:02.9956757Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2024-06-26T07:02:03.3862027Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2024-06-26T07:02:04.5995873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2024-06-26T07:02:04.9162130Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2024-06-26T07:02:05.4367708Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2024-06-26T07:02:06.3782939Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2024-06-26T07:02:11.9304745Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2024-06-26T07:02:12.2575182Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2024-06-26T07:02:12.2712706Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2024-06-26T07:02:12.3003635Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2024-06-26T07:02:12.3100280Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2024-06-26T07:02:12.3110859Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:12.3127497Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2024-06-26T07:02:12.6010900Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2024-06-26T07:02:12.6147260Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2024-06-26T07:02:12.6482776Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2024-06-26T07:02:12.7269478Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2024-06-26T07:02:12.7392735Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2024-06-26T07:02:12.7690433Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out 'a33701196adfad74917046096bf5a2aa0ab0bb50' 2024-06-26T07:02:12.8180955Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2024-06-26T07:02:12.8484573Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78' 2024-06-26T07:02:12.8680217Z Submodule path 'third_party/nccl/nccl': checked out '48bb7fec7953112ff37499a272317f6663f8f600' 2024-06-26T07:02:12.9539110Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806' 2024-06-26T07:02:13.2259303Z Submodule path 'third_party/onnx': checked out '990217f043af7222348ca8f0301e17fa7b841781' 2024-06-26T07:02:13.2283745Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:13.2285864Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:13.2301472Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'... 2024-06-26T07:02:13.5918047Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2024-06-26T07:02:14.7915300Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '2dd015dfef425c866d9a43f2c67d8b52d709acb6' 2024-06-26T07:02:14.8187152Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '5b0a6fc2017fcc176545afe3e09c9f9885283242' 2024-06-26T07:02:14.8711860Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2024-06-26T07:02:14.8724750Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:14.8759538Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:14.8761231Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:14.8762831Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:14.8764684Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:14.8766874Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:14.8768734Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:14.8770297Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:14.8771657Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2024-06-26T07:02:15.3052817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2024-06-26T07:02:16.4078964Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2024-06-26T07:02:16.7575589Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2024-06-26T07:02:22.5047069Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2024-06-26T07:02:22.7770262Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2024-06-26T07:02:23.0245231Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2024-06-26T07:02:23.3048029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2024-06-26T07:02:30.8311753Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2024-06-26T07:02:30.8642121Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2024-06-26T07:02:30.8767506Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2024-06-26T07:02:30.9603813Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2024-06-26T07:02:30.9695210Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2024-06-26T07:02:30.9800511Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2024-06-26T07:02:30.9914915Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2024-06-26T07:02:30.9925555Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:30.9927501Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:30.9944734Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2024-06-26T07:02:32.7343108Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2024-06-26T07:02:33.9451350Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2024-06-26T07:02:33.9812672Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2024-06-26T07:02:34.3427029Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2024-06-26T07:02:34.3514465Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe' 2024-06-26T07:02:34.5672093Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2024-06-26T07:02:34.5689452Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:34.5691547Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:34.5708976Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2024-06-26T07:02:34.9545439Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2024-06-26T07:02:35.9425390Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2024-06-26T07:02:36.0046010Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2024-06-26T07:02:36.0107189Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2024-06-26T07:02:36.0196824Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2024-06-26T07:02:36.0475854Z Submodule path 'third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b' 2024-06-26T07:02:36.0695317Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2024-06-26T07:02:36.1020314Z Submodule path 'third_party/sleef': checked out '60e76d2bce17d278b439d9da17177c8f957a9e9b' 2024-06-26T07:02:36.1220285Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e' 2024-06-26T07:02:36.1231957Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:36.1233782Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:36.1236392Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:36.1238849Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:36.1256747Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2024-06-26T07:02:37.0896790Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2024-06-26T07:02:37.3465039Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2024-06-26T07:02:38.3984735Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2024-06-26T07:02:39.4735463Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2024-06-26T07:02:39.4846497Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2024-06-26T07:02:39.5399377Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242' 2024-06-26T07:02:39.5621445Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2024-06-26T07:02:39.5631549Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:39.5646634Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2024-06-26T07:02:39.7919195Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2024-06-26T07:02:39.7941040Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2024-06-26T07:02:39.8142466Z Entering 'android/libs/fbjni' 2024-06-26T07:02:39.8168291Z Entering 'third_party/FP16' 2024-06-26T07:02:39.8197077Z Entering 'third_party/FXdiv' 2024-06-26T07:02:39.8221995Z Entering 'third_party/NNPACK' 2024-06-26T07:02:39.8248964Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:39.8276135Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:39.8313417Z Entering 'third_party/benchmark' 2024-06-26T07:02:39.8342396Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:39.8368624Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:39.8398306Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:39.8424980Z Entering 'third_party/cutlass' 2024-06-26T07:02:39.8458391Z Entering 'third_party/eigen' 2024-06-26T07:02:39.8487525Z Entering 'third_party/fbgemm' 2024-06-26T07:02:39.8514826Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:39.8541807Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:39.8568827Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:39.8600797Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:39.8628057Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:39.8655779Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:39.8687622Z Entering 'third_party/fmt' 2024-06-26T07:02:39.8716449Z Entering 'third_party/foxi' 2024-06-26T07:02:39.8742575Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:39.8771542Z Entering 'third_party/gloo' 2024-06-26T07:02:39.8797202Z Entering 'third_party/googletest' 2024-06-26T07:02:39.8824642Z Entering 'third_party/ideep' 2024-06-26T07:02:39.8850867Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:39.8883541Z Entering 'third_party/ittapi' 2024-06-26T07:02:39.8912974Z Entering 'third_party/kineto' 2024-06-26T07:02:39.8940932Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:39.8966719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:39.8995938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:39.9022229Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:39.9048571Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:39.9075175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:39.9102727Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:39.9131502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:39.9157370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:39.9183379Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:39.9212102Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:39.9238117Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:39.9265025Z Entering 'third_party/mimalloc' 2024-06-26T07:02:39.9294845Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:39.9324086Z Entering 'third_party/nlohmann' 2024-06-26T07:02:39.9353470Z Entering 'third_party/onnx' 2024-06-26T07:02:39.9394392Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:39.9420926Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:39.9451981Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:39.9478879Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:39.9505528Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:39.9532809Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:39.9559703Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:39.9587829Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:39.9615654Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:39.9642441Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:39.9669510Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:39.9697506Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:39.9726526Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:39.9769151Z Entering 'third_party/pocketfft' 2024-06-26T07:02:39.9795507Z Entering 'third_party/protobuf' 2024-06-26T07:02:39.9826459Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:39.9851476Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:39.9877419Z Entering 'third_party/psimd' 2024-06-26T07:02:39.9906311Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:39.9935866Z Entering 'third_party/pybind11' 2024-06-26T07:02:39.9962511Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:39.9990365Z Entering 'third_party/sleef' 2024-06-26T07:02:40.0018317Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:40.0045417Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:40.0072112Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:40.0098197Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:40.0123547Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:40.0147551Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:40.0187657Z ##[endgroup] 2024-06-26T07:02:40.0190290Z ##[group]Persisting credentials for submodules 2024-06-26T07:02:40.0192320Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2024-06-26T07:02:40.0402425Z Entering 'android/libs/fbjni' 2024-06-26T07:02:40.0437172Z Entering 'third_party/FP16' 2024-06-26T07:02:40.0471095Z Entering 'third_party/FXdiv' 2024-06-26T07:02:40.0504800Z Entering 'third_party/NNPACK' 2024-06-26T07:02:40.0542036Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:40.0578197Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:40.0625629Z Entering 'third_party/benchmark' 2024-06-26T07:02:40.0660341Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:40.0696151Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:40.0731323Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:40.0767315Z Entering 'third_party/cutlass' 2024-06-26T07:02:40.0808683Z Entering 'third_party/eigen' 2024-06-26T07:02:40.0845754Z Entering 'third_party/fbgemm' 2024-06-26T07:02:40.0878859Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:40.0913011Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:40.0947092Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:40.0985500Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:40.1020900Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:40.1053396Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:40.1091395Z Entering 'third_party/fmt' 2024-06-26T07:02:40.1128436Z Entering 'third_party/foxi' 2024-06-26T07:02:40.1165010Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:40.1197256Z Entering 'third_party/gloo' 2024-06-26T07:02:40.1233426Z Entering 'third_party/googletest' 2024-06-26T07:02:40.1267880Z Entering 'third_party/ideep' 2024-06-26T07:02:40.1301564Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:40.1340633Z Entering 'third_party/ittapi' 2024-06-26T07:02:40.1375742Z Entering 'third_party/kineto' 2024-06-26T07:02:40.1409438Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:40.1442874Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:40.1479107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:40.1513035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:40.1545931Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:40.1577111Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:40.1613183Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:40.1645903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:40.1680431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:40.1715593Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:40.1751186Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:40.1784491Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:40.1820319Z Entering 'third_party/mimalloc' 2024-06-26T07:02:40.1856980Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:40.1893325Z Entering 'third_party/nlohmann' 2024-06-26T07:02:40.1928245Z Entering 'third_party/onnx' 2024-06-26T07:02:40.1976038Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:40.2011668Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:40.2046191Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:40.2082277Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:40.2116175Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:40.2147951Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:40.2181969Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:40.2217762Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:40.2252585Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:40.2287319Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:40.2320854Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:40.2356047Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:40.2391317Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:40.2440062Z Entering 'third_party/pocketfft' 2024-06-26T07:02:40.2473953Z Entering 'third_party/protobuf' 2024-06-26T07:02:40.2512397Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:40.2546515Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:40.2582728Z Entering 'third_party/psimd' 2024-06-26T07:02:40.2618995Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:40.2655154Z Entering 'third_party/pybind11' 2024-06-26T07:02:40.2688907Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:40.2724354Z Entering 'third_party/sleef' 2024-06-26T07:02:40.2760694Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:40.2794972Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:40.2828686Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:40.2862807Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:40.2897500Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:40.2928410Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:40.2976387Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2024-06-26T07:02:40.3180840Z Entering 'android/libs/fbjni' 2024-06-26T07:02:40.3211415Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2024-06-26T07:02:40.3225045Z Entering 'third_party/FP16' 2024-06-26T07:02:40.3255821Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2024-06-26T07:02:40.3268818Z Entering 'third_party/FXdiv' 2024-06-26T07:02:40.3299222Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2024-06-26T07:02:40.3313414Z Entering 'third_party/NNPACK' 2024-06-26T07:02:40.3343901Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2024-06-26T07:02:40.3356466Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:40.3389303Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2024-06-26T07:02:40.3402586Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:40.3434959Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2024-06-26T07:02:40.3459274Z Entering 'third_party/benchmark' 2024-06-26T07:02:40.3492318Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:40.3504449Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:40.3537290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2024-06-26T07:02:40.3549430Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:40.3581149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:40.3593932Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:40.3626327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2024-06-26T07:02:40.3640343Z Entering 'third_party/cutlass' 2024-06-26T07:02:40.3671793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:40.3691089Z Entering 'third_party/eigen' 2024-06-26T07:02:40.3722039Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config remote.origin.url 2024-06-26T07:02:40.3737506Z Entering 'third_party/fbgemm' 2024-06-26T07:02:40.3770914Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2024-06-26T07:02:40.3783779Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:40.3815282Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config remote.origin.url 2024-06-26T07:02:40.3827049Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:40.3858985Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config remote.origin.url 2024-06-26T07:02:40.3872188Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:40.3903302Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config remote.origin.url 2024-06-26T07:02:40.3920675Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:40.3953536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.3965575Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:40.3997146Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config remote.origin.url 2024-06-26T07:02:40.4011470Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:40.4042883Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2024-06-26T07:02:40.4059151Z Entering 'third_party/fmt' 2024-06-26T07:02:40.4090582Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:40.4103145Z Entering 'third_party/foxi' 2024-06-26T07:02:40.4136104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config remote.origin.url 2024-06-26T07:02:40.4147979Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:40.4180057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2024-06-26T07:02:40.4192186Z Entering 'third_party/gloo' 2024-06-26T07:02:40.4223595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2024-06-26T07:02:40.4236564Z Entering 'third_party/googletest' 2024-06-26T07:02:40.4267134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.4281919Z Entering 'third_party/ideep' 2024-06-26T07:02:40.4312510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2024-06-26T07:02:40.4324078Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:40.4355682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2024-06-26T07:02:40.4373870Z Entering 'third_party/ittapi' 2024-06-26T07:02:40.4405701Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2024-06-26T07:02:40.4419446Z Entering 'third_party/kineto' 2024-06-26T07:02:40.4450703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2024-06-26T07:02:40.4462213Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:40.4494866Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2024-06-26T07:02:40.4507386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:40.4539153Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2024-06-26T07:02:40.4553204Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:40.4583170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2024-06-26T07:02:40.4594957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:40.4626838Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2024-06-26T07:02:40.4638663Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:40.4671668Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2024-06-26T07:02:40.4683073Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:40.4715489Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2024-06-26T07:02:40.4729951Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:40.4761492Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2024-06-26T07:02:40.4774718Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:40.4806871Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.4819801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:40.4852538Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2024-06-26T07:02:40.4865105Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:40.4897123Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2024-06-26T07:02:40.4911323Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:40.4942614Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2024-06-26T07:02:40.4955128Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:40.4986075Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.4999709Z Entering 'third_party/mimalloc' 2024-06-26T07:02:40.5030991Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2024-06-26T07:02:40.5044561Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:40.5076466Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config remote.origin.url 2024-06-26T07:02:40.5089174Z Entering 'third_party/nlohmann' 2024-06-26T07:02:40.5120659Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2024-06-26T07:02:40.5135100Z Entering 'third_party/onnx' 2024-06-26T07:02:40.5166483Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2024-06-26T07:02:40.5192508Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:40.5224022Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:40.5237366Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:40.5268678Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:40.5283394Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:40.5315666Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2024-06-26T07:02:40.5329386Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:40.5361440Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:40.5374918Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:40.5407152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.5419113Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:40.5451752Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2024-06-26T07:02:40.5464141Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:40.5494285Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2024-06-26T07:02:40.5507410Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:40.5538580Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2024-06-26T07:02:40.5551400Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:40.5582254Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2024-06-26T07:02:40.5593952Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:40.5624950Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2024-06-26T07:02:40.5637494Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:40.5669437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2024-06-26T07:02:40.5683751Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:40.5714536Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2024-06-26T07:02:40.5728826Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:40.5759348Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2024-06-26T07:02:40.5787412Z Entering 'third_party/pocketfft' 2024-06-26T07:02:40.5819187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2024-06-26T07:02:40.5832029Z Entering 'third_party/protobuf' 2024-06-26T07:02:40.5864567Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2024-06-26T07:02:40.5878705Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:40.5910720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2024-06-26T07:02:40.5922840Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:40.5954262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.5968320Z Entering 'third_party/psimd' 2024-06-26T07:02:40.5999924Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2024-06-26T07:02:40.6013729Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:40.6044969Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2024-06-26T07:02:40.6058291Z Entering 'third_party/pybind11' 2024-06-26T07:02:40.6090972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:40.6104010Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:40.6137922Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2024-06-26T07:02:40.6151363Z Entering 'third_party/sleef' 2024-06-26T07:02:40.6181723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2024-06-26T07:02:40.6195872Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:40.6227204Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2024-06-26T07:02:40.6239142Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:40.6269875Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2024-06-26T07:02:40.6281877Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:40.6312171Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2024-06-26T07:02:40.6324775Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:40.6355999Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2024-06-26T07:02:40.6368635Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:40.6399808Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2024-06-26T07:02:40.6411921Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:40.6444041Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2024-06-26T07:02:40.7284123Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2024-06-26T07:02:40.7495941Z Entering 'android/libs/fbjni' 2024-06-26T07:02:40.7524931Z Entering 'third_party/FP16' 2024-06-26T07:02:40.7552561Z Entering 'third_party/FXdiv' 2024-06-26T07:02:40.7580373Z Entering 'third_party/NNPACK' 2024-06-26T07:02:40.7607605Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:40.7636148Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:40.7674891Z Entering 'third_party/benchmark' 2024-06-26T07:02:40.7701521Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:40.7729453Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:40.7757482Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:40.7784807Z Entering 'third_party/cutlass' 2024-06-26T07:02:40.7820260Z Entering 'third_party/eigen' 2024-06-26T07:02:40.7849773Z Entering 'third_party/fbgemm' 2024-06-26T07:02:40.7876421Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:40.7904733Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:40.7932902Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:40.7964626Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:40.7990881Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:40.8019458Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:40.8048404Z Entering 'third_party/fmt' 2024-06-26T07:02:40.8074571Z Entering 'third_party/foxi' 2024-06-26T07:02:40.8103638Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:40.8129548Z Entering 'third_party/gloo' 2024-06-26T07:02:40.8155759Z Entering 'third_party/googletest' 2024-06-26T07:02:40.8184367Z Entering 'third_party/ideep' 2024-06-26T07:02:40.8210567Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:40.8243732Z Entering 'third_party/ittapi' 2024-06-26T07:02:40.8273701Z Entering 'third_party/kineto' 2024-06-26T07:02:40.8299354Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:40.8326383Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:40.8354813Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:40.8379982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:40.8407107Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:40.8434418Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:40.8461096Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:40.8490326Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:40.8517458Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:40.8546263Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:40.8575325Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:40.8603932Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:40.8632931Z Entering 'third_party/mimalloc' 2024-06-26T07:02:40.8660665Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:40.8687654Z Entering 'third_party/nlohmann' 2024-06-26T07:02:40.8717328Z Entering 'third_party/onnx' 2024-06-26T07:02:40.8757303Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:40.8785446Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:40.8815569Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:40.8844818Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:40.8870118Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:40.8896768Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:40.8922568Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:40.8949167Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:40.8976776Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:40.9002127Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:40.9029951Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:40.9057404Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:40.9086730Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:40.9129377Z Entering 'third_party/pocketfft' 2024-06-26T07:02:40.9157672Z Entering 'third_party/protobuf' 2024-06-26T07:02:40.9186589Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:40.9214226Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:40.9243040Z Entering 'third_party/psimd' 2024-06-26T07:02:40.9271505Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:40.9300629Z Entering 'third_party/pybind11' 2024-06-26T07:02:40.9329286Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:40.9355698Z Entering 'third_party/sleef' 2024-06-26T07:02:40.9384065Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:40.9410437Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:40.9438748Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:40.9465267Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:40.9491922Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:40.9517143Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:40.9558921Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2024-06-26T07:02:40.9764829Z Entering 'android/libs/fbjni' 2024-06-26T07:02:40.9791881Z Entering 'third_party/FP16' 2024-06-26T07:02:40.9820507Z Entering 'third_party/FXdiv' 2024-06-26T07:02:40.9849952Z Entering 'third_party/NNPACK' 2024-06-26T07:02:40.9878831Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T07:02:40.9906453Z Entering 'third_party/XNNPACK' 2024-06-26T07:02:40.9947252Z Entering 'third_party/benchmark' 2024-06-26T07:02:40.9975107Z Entering 'third_party/cpp-httplib' 2024-06-26T07:02:41.0004167Z Entering 'third_party/cpuinfo' 2024-06-26T07:02:41.0032680Z Entering 'third_party/cudnn_frontend' 2024-06-26T07:02:41.0058565Z Entering 'third_party/cutlass' 2024-06-26T07:02:41.0093350Z Entering 'third_party/eigen' 2024-06-26T07:02:41.0124754Z Entering 'third_party/fbgemm' 2024-06-26T07:02:41.0153720Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T07:02:41.0181467Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T07:02:41.0209221Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T07:02:41.0241950Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T07:02:41.0268762Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T07:02:41.0296781Z Entering 'third_party/flatbuffers' 2024-06-26T07:02:41.0327553Z Entering 'third_party/fmt' 2024-06-26T07:02:41.0357584Z Entering 'third_party/foxi' 2024-06-26T07:02:41.0386057Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T07:02:41.0413522Z Entering 'third_party/gloo' 2024-06-26T07:02:41.0441646Z Entering 'third_party/googletest' 2024-06-26T07:02:41.0469290Z Entering 'third_party/ideep' 2024-06-26T07:02:41.0496752Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T07:02:41.0529675Z Entering 'third_party/ittapi' 2024-06-26T07:02:41.0556866Z Entering 'third_party/kineto' 2024-06-26T07:02:41.0583269Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T07:02:41.0611831Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T07:02:41.0640291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T07:02:41.0665602Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T07:02:41.0694209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T07:02:41.0720230Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T07:02:41.0749223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T07:02:41.0776118Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T07:02:41.0802757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T07:02:41.0830967Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T07:02:41.0858497Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T07:02:41.0884987Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T07:02:41.0915010Z Entering 'third_party/mimalloc' 2024-06-26T07:02:41.0944727Z Entering 'third_party/nccl/nccl' 2024-06-26T07:02:41.0973470Z Entering 'third_party/nlohmann' 2024-06-26T07:02:41.1000803Z Entering 'third_party/onnx' 2024-06-26T07:02:41.1040951Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T07:02:41.1069321Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T07:02:41.1098786Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T07:02:41.1129773Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T07:02:41.1157078Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T07:02:41.1184057Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T07:02:41.1211184Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T07:02:41.1237119Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T07:02:41.1262729Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T07:02:41.1290798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T07:02:41.1316812Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T07:02:41.1346428Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T07:02:41.1374788Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T07:02:41.1416525Z Entering 'third_party/pocketfft' 2024-06-26T07:02:41.1446354Z Entering 'third_party/protobuf' 2024-06-26T07:02:41.1476335Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T07:02:41.1504178Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T07:02:41.1534787Z Entering 'third_party/psimd' 2024-06-26T07:02:41.1561946Z Entering 'third_party/pthreadpool' 2024-06-26T07:02:41.1590392Z Entering 'third_party/pybind11' 2024-06-26T07:02:41.1618940Z Entering 'third_party/python-peachpy' 2024-06-26T07:02:41.1645921Z Entering 'third_party/sleef' 2024-06-26T07:02:41.1675100Z Entering 'third_party/tensorpipe' 2024-06-26T07:02:41.1700884Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T07:02:41.1728462Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T07:02:41.1754461Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T07:02:41.1780954Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T07:02:41.1807002Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T07:02:41.1847060Z ##[endgroup] 2024-06-26T07:02:41.1875716Z [command]/usr/bin/git log -1 --format='%H' 2024-06-26T07:02:41.1896314Z 'b8c4c54d347aa776934c60784e35936878ef18dc' 2024-06-26T07:02:41.2070611Z Prepare all required actions 2024-06-26T07:02:41.2071136Z Getting action download info 2024-06-26T07:02:41.3381216Z ##[group]Run ./.github/actions/setup-linux 2024-06-26T07:02:41.3381643Z env: 2024-06-26T07:02:41.3381927Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:41.3382277Z ##[endgroup] 2024-06-26T07:02:41.3441098Z ##[group]Run set -euo pipefail 2024-06-26T07:02:41.3441540Z set -euo pipefail 2024-06-26T07:02:41.3441929Z function get_ec2_metadata() { 2024-06-26T07:02:41.3442517Z  # Pulled from instance metadata endpoint for EC2 2024-06-26T07:02:41.3443405Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2024-06-26T07:02:41.3444173Z  category=$1 2024-06-26T07:02:41.3444714Z  # If it is GCP runner (runner name contains gcp), do not run this 2024-06-26T07:02:41.3445357Z  runner_name_str=i-01e2eadbbc5643a4a 2024-06-26T07:02:41.3445857Z  if [[ -f /.inarc ]]; then 2024-06-26T07:02:41.3446450Z  echo "ARC Runner, no info on ec2 metadata" 2024-06-26T07:02:41.3447025Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2024-06-26T07:02:41.3447713Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2024-06-26T07:02:41.3448316Z  else 2024-06-26T07:02:41.3448802Z  curl -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2024-06-26T07:02:41.3449377Z  fi 2024-06-26T07:02:41.3449667Z } 2024-06-26T07:02:41.3450030Z echo "ami-id: $(get_ec2_metadata ami-id)" 2024-06-26T07:02:41.3450611Z echo "instance-id: $(get_ec2_metadata instance-id)" 2024-06-26T07:02:41.3451264Z echo "instance-type: $(get_ec2_metadata instance-type)" 2024-06-26T07:02:41.3451835Z echo "system info $(uname -a)" 2024-06-26T07:02:41.3459637Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:41.3460130Z env: 2024-06-26T07:02:41.3460417Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:41.3460773Z ##[endgroup] 2024-06-26T07:02:41.3530081Z ami-id: ami-0ce0c36d7a00b20e2 2024-06-26T07:02:41.3571933Z instance-id: i-01e2eadbbc5643a4a 2024-06-26T07:02:41.3611031Z instance-type: g5.4xlarge 2024-06-26T07:02:41.3617087Z system info Linux ip-10-0-34-11.ec2.internal 4.14.336-257.562.amzn2.x86_64 #1 SMP Sat Feb 24 09:50:35 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux 2024-06-26T07:02:41.3638599Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:41.3639590Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> $GITHUB_OUTPUT 2024-06-26T07:02:41.3647985Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:41.3648667Z env: 2024-06-26T07:02:41.3649024Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:41.3649454Z ##[endgroup] 2024-06-26T07:02:41.3709579Z ##[group]Run if systemctl is-active --quiet docker; then 2024-06-26T07:02:41.3710195Z if systemctl is-active --quiet docker; then 2024-06-26T07:02:41.3710736Z  echo "Docker daemon is running..."; 2024-06-26T07:02:41.3711193Z else 2024-06-26T07:02:41.3711685Z  echo "Starting docker deamon..." && sudo systemctl start docker; 2024-06-26T07:02:41.3712446Z fi 2024-06-26T07:02:41.3719275Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:41.3719760Z env: 2024-06-26T07:02:41.3720039Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:41.3720388Z ##[endgroup] 2024-06-26T07:02:41.3755215Z Docker daemon is running... 2024-06-26T07:02:41.3797668Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:02:41.3798228Z with: 2024-06-26T07:02:41.3798499Z shell: bash 2024-06-26T07:02:41.3798815Z timeout_minutes: 5 2024-06-26T07:02:41.3799146Z max_attempts: 3 2024-06-26T07:02:41.3799475Z retry_wait_seconds: 30 2024-06-26T07:02:41.3800963Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" 2024-06-26T07:02:41.3802442Z polling_interval_seconds: 1 2024-06-26T07:02:41.3802818Z warning_on_retry: true 2024-06-26T07:02:41.3803177Z continue_on_error: false 2024-06-26T07:02:41.3803521Z env: 2024-06-26T07:02:41.3803792Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:41.3804153Z AWS_RETRY_MODE: standard 2024-06-26T07:02:41.3804512Z AWS_MAX_ATTEMPTS: 5 2024-06-26T07:02:41.3804859Z AWS_DEFAULT_REGION: us-east-1 2024-06-26T07:02:41.3805230Z ##[endgroup] 2024-06-26T07:02:42.2312532Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:42.2313463Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:42.2314304Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:42.2314823Z 2024-06-26T07:02:42.2314965Z Login Succeeded 2024-06-26T07:02:42.4297872Z Command completed after 1 attempt(s). 2024-06-26T07:02:42.4342047Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:42.4342771Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:42.4343431Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2024-06-26T07:02:42.4351480Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:42.4351977Z env: 2024-06-26T07:02:42.4352255Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.4352610Z ##[endgroup] 2024-06-26T07:02:42.4411986Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:42.4412745Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T07:02:42.4413345Z # shellcheck disable=SC2046 2024-06-26T07:02:42.4413803Z docker stop $(docker ps -q) || true 2024-06-26T07:02:42.4414282Z # Prune all of the docker images 2024-06-26T07:02:42.4414726Z docker system prune -af 2024-06-26T07:02:42.4421594Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:42.4422087Z env: 2024-06-26T07:02:42.4422355Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.4422700Z ##[endgroup] 2024-06-26T07:02:42.4667929Z "docker stop" requires at least 1 argument. 2024-06-26T07:02:42.4668965Z See 'docker stop --help'. 2024-06-26T07:02:42.4669209Z 2024-06-26T07:02:42.4669439Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2024-06-26T07:02:42.4669823Z 2024-06-26T07:02:42.4669976Z Stop one or more running containers 2024-06-26T07:02:42.4811839Z Total reclaimed space: 0B 2024-06-26T07:02:42.4840882Z ##[group]Run set +e 2024-06-26T07:02:42.4841242Z set +e 2024-06-26T07:02:42.4841692Z set -x 2024-06-26T07:02:42.4841990Z  2024-06-26T07:02:42.4842309Z PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:42.4843157Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2024-06-26T07:02:42.4844222Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2024-06-26T07:02:42.4844966Z # one is returned at random 2024-06-26T07:02:42.4845510Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2024-06-26T07:02:42.4846015Z  2024-06-26T07:02:42.4846487Z if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:42.4847111Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2024-06-26T07:02:42.4847875Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2024-06-26T07:02:42.4848418Z  2024-06-26T07:02:42.4848751Z  if [ -z "${RESOLVED_IP}" ]; then 2024-06-26T07:02:42.4849318Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2024-06-26T07:02:42.4849827Z  exit 1 2024-06-26T07:02:42.4850137Z  fi 2024-06-26T07:02:42.4850424Z fi 2024-06-26T07:02:42.4850823Z  2024-06-26T07:02:42.4851189Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2024-06-26T07:02:42.4851697Z  # Clean up any old records first 2024-06-26T07:02:42.4852207Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2024-06-26T07:02:42.4852665Z fi 2024-06-26T07:02:42.4852937Z  2024-06-26T07:02:42.4853376Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2024-06-26T07:02:42.4853923Z cat /etc/hosts 2024-06-26T07:02:42.4861639Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:42.4862139Z env: 2024-06-26T07:02:42.4862436Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.4862794Z ##[endgroup] 2024-06-26T07:02:42.4882645Z + PT_DOMAIN=download.pytorch.org 2024-06-26T07:02:42.4886540Z ++ dig -4 +short download.pytorch.org 2024-06-26T07:02:42.4886984Z ++ tail -n1 2024-06-26T07:02:42.5019266Z + RESOLVED_IP=18.160.10.28 2024-06-26T07:02:42.5019698Z + '[' -z 18.160.10.28 ']' 2024-06-26T07:02:42.5020130Z + grep -r download.pytorch.org /etc/hosts 2024-06-26T07:02:42.5025297Z 18.160.10.28 download.pytorch.org 2024-06-26T07:02:42.5026393Z + sudo sed -i /download.pytorch.org/d /etc/hosts 2024-06-26T07:02:42.5123058Z + echo '18.160.10.28 download.pytorch.org' 2024-06-26T07:02:42.5123700Z + sudo tee -a /etc/hosts 2024-06-26T07:02:42.5380569Z 18.160.10.28 download.pytorch.org 2024-06-26T07:02:42.5393484Z + cat /etc/hosts 2024-06-26T07:02:42.5397638Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2024-06-26T07:02:42.5409401Z ::1 localhost6 localhost6.localdomain6 2024-06-26T07:02:42.5409906Z 18.160.10.28 download.pytorch.org 2024-06-26T07:02:42.5558737Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2024-06-26T07:02:42.5559416Z with: 2024-06-26T07:02:42.5560631Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5561901Z docker-build-dir: .ci/docker 2024-06-26T07:02:42.5562786Z working-directory: . 2024-06-26T07:02:42.5597930Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.5598524Z force-push: false 2024-06-26T07:02:42.5598840Z env: 2024-06-26T07:02:42.5599118Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.5599474Z ##[endgroup] 2024-06-26T07:02:42.5617329Z ##[group]Run set -ex 2024-06-26T07:02:42.5617679Z set -ex 2024-06-26T07:02:42.5617973Z  2024-06-26T07:02:42.5618522Z # If the docker build directory or the build script doesn't exist, the action will 2024-06-26T07:02:42.5619478Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2024-06-26T07:02:42.5620397Z # job could then download the pre-built image as usual 2024-06-26T07:02:42.5621132Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/build.sh" ]]; then 2024-06-26T07:02:42.5621794Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5622418Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5622960Z  2024-06-26T07:02:42.5623456Z  echo "There is no Docker build script in ${REPO_NAME} repo, skipping..." 2024-06-26T07:02:42.5624066Z  exit 0 2024-06-26T07:02:42.5624360Z else 2024-06-26T07:02:42.5624722Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5625161Z fi 2024-06-26T07:02:42.5625428Z  2024-06-26T07:02:42.5625886Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2024-06-26T07:02:42.5626722Z  # The docker image name already includes the ECR prefix and tag, so we can just 2024-06-26T07:02:42.5627491Z  # use it as it is, but first let's extract the tag 2024-06-26T07:02:42.5628390Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2024-06-26T07:02:42.5629110Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5629872Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5630420Z else 2024-06-26T07:02:42.5630846Z  DOCKER_TAG=$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2024-06-26T07:02:42.5631498Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5632375Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5633115Z fi 2024-06-26T07:02:42.5640693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:42.5641180Z env: 2024-06-26T07:02:42.5641469Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.5641819Z REPO_NAME: pytorch 2024-06-26T07:02:42.5642947Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5644127Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:42.5644613Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.5645114Z ##[endgroup] 2024-06-26T07:02:42.5664549Z + [[ ! -d .ci/docker ]] 2024-06-26T07:02:42.5664977Z + [[ ! -f .ci/docker/build.sh ]] 2024-06-26T07:02:42.5665360Z + echo skip=false 2024-06-26T07:02:42.5667320Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2024-06-26T07:02:42.5670115Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5671318Z ++ awk -F '[:,]' '{print $2}' 2024-06-26T07:02:42.5677175Z + DOCKER_TAG=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5678034Z + echo docker-tag=91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5679461Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5705691Z ##[group]Run set +e 2024-06-26T07:02:42.5706063Z set +e 2024-06-26T07:02:42.5706362Z set -x 2024-06-26T07:02:42.5706656Z  2024-06-26T07:02:42.5706945Z login() { 2024-06-26T07:02:42.5707619Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:42.5709264Z } 2024-06-26T07:02:42.5709547Z  2024-06-26T07:02:42.5709963Z retry () { 2024-06-26T07:02:42.5710360Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:42.5710817Z } 2024-06-26T07:02:42.5711093Z  2024-06-26T07:02:42.5711421Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:42.5711844Z  2024-06-26T07:02:42.5712315Z # Check if image already exists, if it does then skip building it 2024-06-26T07:02:42.5713020Z if docker manifest inspect "${DOCKER_IMAGE}"; then 2024-06-26T07:02:42.5713535Z  exit 0 2024-06-26T07:02:42.5713846Z fi 2024-06-26T07:02:42.5714126Z  2024-06-26T07:02:42.5714639Z # NB: This part requires a full checkout. Otherwise, the merge base will 2024-06-26T07:02:42.5715469Z # be empty. The default action would be to continue rebuild the image 2024-06-26T07:02:42.5716217Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2024-06-26T07:02:42.5716885Z  # if we're on the base branch then use the parent commit 2024-06-26T07:02:42.5717492Z  MERGE_BASE=$(git rev-parse HEAD~) 2024-06-26T07:02:42.5717929Z else 2024-06-26T07:02:42.5718413Z  # otherwise we're on a PR, so use the most recent base commit 2024-06-26T07:02:42.5719108Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2024-06-26T07:02:42.5719621Z fi 2024-06-26T07:02:42.5719911Z  2024-06-26T07:02:42.5720237Z if [[ -z "${MERGE_BASE}" ]]; then 2024-06-26T07:02:42.5720746Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5721213Z  2024-06-26T07:02:42.5721876Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2024-06-26T07:02:42.5722632Z  exit 0 2024-06-26T07:02:42.5722941Z fi 2024-06-26T07:02:42.5723226Z  2024-06-26T07:02:42.5723674Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2024-06-26T07:02:42.5724646Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2024-06-26T07:02:42.5725436Z  exit 1 2024-06-26T07:02:42.5725745Z fi 2024-06-26T07:02:42.5726028Z  2024-06-26T07:02:42.5726650Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2024-06-26T07:02:42.5727590Z # If no image exists but the hash is the same as the previous hash then we should error out here 2024-06-26T07:02:42.5728456Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2024-06-26T07:02:42.5729412Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2024-06-26T07:02:42.5730507Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2024-06-26T07:02:42.5731242Z fi 2024-06-26T07:02:42.5731520Z  2024-06-26T07:02:42.5731890Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2024-06-26T07:02:42.5739578Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:42.5740078Z env: 2024-06-26T07:02:42.5740352Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:42.5740720Z DOCKER_BUILD_DIR: .ci/docker 2024-06-26T07:02:42.5741185Z BASE_REVISION: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:02:42.5742437Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5743666Z DOCKER_TAG: 91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:42.5744256Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.5744772Z ##[endgroup] 2024-06-26T07:02:42.5764996Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.5765660Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.5766624Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:42.5767802Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:42.9868498Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:42.9869348Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:42.9870167Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:42.9870683Z 2024-06-26T07:02:42.9870820Z Login Succeeded 2024-06-26T07:02:42.9879875Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.1802645Z { 2024-06-26T07:02:43.1803089Z "schemaVersion": 2, 2024-06-26T07:02:43.1803842Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2024-06-26T07:02:43.1804619Z "config": { 2024-06-26T07:02:43.1805214Z "mediaType": "application/vnd.docker.container.image.v1+json", 2024-06-26T07:02:43.1806006Z "size": 48681, 2024-06-26T07:02:43.1806956Z "digest": "sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd" 2024-06-26T07:02:43.1807797Z }, 2024-06-26T07:02:43.1808156Z "layers": [ 2024-06-26T07:02:43.1808514Z { 2024-06-26T07:02:43.1809095Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1809865Z "size": 28580681, 2024-06-26T07:02:43.1810626Z "digest": "sha256:7a2c559011895d255fce249c00396abff5ae7e0c0a92931d0ed493e71de78e3a" 2024-06-26T07:02:43.1811493Z }, 2024-06-26T07:02:43.1811828Z { 2024-06-26T07:02:43.1812437Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1813232Z "size": 7943451, 2024-06-26T07:02:43.1814026Z "digest": "sha256:224fe954d7252f10539d243d6c9688806f7d13ad775ed02e7f7c79077844510d" 2024-06-26T07:02:43.1814915Z }, 2024-06-26T07:02:43.1815238Z { 2024-06-26T07:02:43.1815786Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1816502Z "size": 55728572, 2024-06-26T07:02:43.1817265Z "digest": "sha256:75722010b82e31715876aeeed0b2cee414296f0124fdfa061ab845ba2a158450" 2024-06-26T07:02:43.1818052Z }, 2024-06-26T07:02:43.1818370Z { 2024-06-26T07:02:43.1818940Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1819598Z "size": 186, 2024-06-26T07:02:43.1820143Z "digest": "sha256:d527cbbb87e3016fd72a18a9b468c945ad0ca27c5770b39debd6ed704db3a195" 2024-06-26T07:02:43.1820757Z }, 2024-06-26T07:02:43.1821008Z { 2024-06-26T07:02:43.1821447Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1821998Z "size": 6886, 2024-06-26T07:02:43.1822531Z "digest": "sha256:b57676e46aee1a8c82e528d78e5a13e31142524eea31c8b213d69ddcb6f1fe80" 2024-06-26T07:02:43.1823139Z }, 2024-06-26T07:02:43.1823585Z { 2024-06-26T07:02:43.1824022Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1824572Z "size": 1329001756, 2024-06-26T07:02:43.1825153Z "digest": "sha256:a8c1e85b5e14cec7af70bf304cb4d4cee6a1d25eb8215b2cf4fdc33e5af5e108" 2024-06-26T07:02:43.1825774Z }, 2024-06-26T07:02:43.1826017Z { 2024-06-26T07:02:43.1826451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1827000Z "size": 62501, 2024-06-26T07:02:43.1827543Z "digest": "sha256:a41a8d1c11c8d80fe4e82b0d05478f8d51176ff20b8350905fc1b25c93a51198" 2024-06-26T07:02:43.1828467Z }, 2024-06-26T07:02:43.1828885Z + exit 0 2024-06-26T07:02:43.1829154Z { 2024-06-26T07:02:43.1829588Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1830128Z "size": 1684, 2024-06-26T07:02:43.1830663Z "digest": "sha256:0c12278907551c2962927d27c115f6f7bf0df894318b8aea6ece3ef01ccd0a8a" 2024-06-26T07:02:43.1831264Z }, 2024-06-26T07:02:43.1831523Z { 2024-06-26T07:02:43.1831945Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1832617Z "size": 1523, 2024-06-26T07:02:43.1833221Z "digest": "sha256:d8d1234baab3ec9ccb8bb710fc6b8ff6c10896ba2e8d27a347583eca770f9ff1" 2024-06-26T07:02:43.1833839Z }, 2024-06-26T07:02:43.1834091Z { 2024-06-26T07:02:43.1834521Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1835077Z "size": 2528295403, 2024-06-26T07:02:43.1835628Z "digest": "sha256:7ed32bc8e4696fcdb2feef850781160597b2275ad756819c4add88236b0577d5" 2024-06-26T07:02:43.1836244Z }, 2024-06-26T07:02:43.1836492Z { 2024-06-26T07:02:43.1836920Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1837462Z "size": 86016, 2024-06-26T07:02:43.1838012Z "digest": "sha256:ec1e7978c1fe161ced1d98092a51e7c5953ca5fda5577f54df9dbda4afff1b2b" 2024-06-26T07:02:43.1838627Z }, 2024-06-26T07:02:43.1838870Z { 2024-06-26T07:02:43.1839307Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1839865Z "size": 1823, 2024-06-26T07:02:43.1840396Z "digest": "sha256:309b872da805fcdf74137a887464876fa0615dc8b24199307119eb65fc2bfb49" 2024-06-26T07:02:43.1840996Z }, 2024-06-26T07:02:43.1841244Z { 2024-06-26T07:02:43.1841672Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1842224Z "size": 246737542, 2024-06-26T07:02:43.1842786Z "digest": "sha256:e82932cb002c06a782077b8e5965f8733eb8563406a23cab260d9d8c4a994f7d" 2024-06-26T07:02:43.1843417Z }, 2024-06-26T07:02:43.1843667Z { 2024-06-26T07:02:43.1844098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1844647Z "size": 545, 2024-06-26T07:02:43.1845194Z "digest": "sha256:ca9dd73e3e9378225ebdcfd12199ef44649e34fe55003cfbc8afe2e8a4527c5b" 2024-06-26T07:02:43.1845813Z }, 2024-06-26T07:02:43.1846053Z { 2024-06-26T07:02:43.1846636Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1847180Z "size": 1283, 2024-06-26T07:02:43.1847706Z "digest": "sha256:16d76a79d5d6616e228537785ec83a869f68a402520bbc6e826201621ca6cb98" 2024-06-26T07:02:43.1848288Z }, 2024-06-26T07:02:43.1848534Z { 2024-06-26T07:02:43.1848959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1849498Z "size": 485, 2024-06-26T07:02:43.1850030Z "digest": "sha256:4bdf41c9ecbe1651e0986ccbea4663a0556af27b7145a679cfa45c35dc96b891" 2024-06-26T07:02:43.1850645Z }, 2024-06-26T07:02:43.1850892Z { 2024-06-26T07:02:43.1851308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1851854Z "size": 91710937, 2024-06-26T07:02:43.1852419Z "digest": "sha256:f4863801e30cdbcd1d4e6cb1cbbcde7dbcaf6ae317ca80429295741ce60ac31d" 2024-06-26T07:02:43.1853039Z }, 2024-06-26T07:02:43.1853279Z { 2024-06-26T07:02:43.1853709Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1854331Z "size": 3233, 2024-06-26T07:02:43.1854880Z "digest": "sha256:d37dd9ed8948a0d02e1198bbedadbca36477a0980dd553196385aab0c025581d" 2024-06-26T07:02:43.1855500Z }, 2024-06-26T07:02:43.1855749Z { 2024-06-26T07:02:43.1856180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1856730Z "size": 1905, 2024-06-26T07:02:43.1857272Z "digest": "sha256:cf2f34f225dbed17c1a9a8b9ea4ad030aba797774a716dc0a82fe212324aa751" 2024-06-26T07:02:43.1857891Z }, 2024-06-26T07:02:43.1858143Z { 2024-06-26T07:02:43.1858561Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1859106Z "size": 702, 2024-06-26T07:02:43.1859640Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:43.1860255Z }, 2024-06-26T07:02:43.1860502Z { 2024-06-26T07:02:43.1860931Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1861484Z "size": 2784789002, 2024-06-26T07:02:43.1862047Z "digest": "sha256:3d914950f8addfc4d657a2365c76e1143eb80b710cd3f6e9ae05640217fc6347" 2024-06-26T07:02:43.1862700Z }, 2024-06-26T07:02:43.1862954Z { 2024-06-26T07:02:43.1863431Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1863974Z "size": 381, 2024-06-26T07:02:43.1864515Z "digest": "sha256:96d0e2032184c57e62a652aaffcf0241f0d0c7a23e894ca0ad192c4a64a36583" 2024-06-26T07:02:43.1865129Z }, 2024-06-26T07:02:43.1865379Z { 2024-06-26T07:02:43.1865799Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1866350Z "size": 11879, 2024-06-26T07:02:43.1866884Z "digest": "sha256:06e12ce59790cb8b091e5d619c2a62a28329008ee184a8b9a562e3db12471f30" 2024-06-26T07:02:43.1867485Z }, 2024-06-26T07:02:43.1867731Z { 2024-06-26T07:02:43.1868359Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1868912Z "size": 803, 2024-06-26T07:02:43.1869456Z "digest": "sha256:ed712f63cde7070bdec2a6179c4c35e752c6991ddbc69fe9f2724a9903250ceb" 2024-06-26T07:02:43.1870066Z }, 2024-06-26T07:02:43.1870311Z { 2024-06-26T07:02:43.1870732Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1871265Z "size": 106, 2024-06-26T07:02:43.1871790Z "digest": "sha256:a97a71ef8a4e66234dfc2098834f8395ac5630240e60164d1a251fdb0115ba85" 2024-06-26T07:02:43.1872385Z }, 2024-06-26T07:02:43.1872635Z { 2024-06-26T07:02:43.1873098Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1873640Z "size": 503, 2024-06-26T07:02:43.1874168Z "digest": "sha256:25cd0c482ad612f936b35b4f220c580e16be982226aac8801a6a849a6bfdc221" 2024-06-26T07:02:43.1874770Z }, 2024-06-26T07:02:43.1875010Z { 2024-06-26T07:02:43.1875432Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1875981Z "size": 121477177, 2024-06-26T07:02:43.1876539Z "digest": "sha256:9bf94bc456584365fda21d0ed83b7cc690b08a71727b71c4c1e667e15028a386" 2024-06-26T07:02:43.1877138Z }, 2024-06-26T07:02:43.1877392Z { 2024-06-26T07:02:43.1877823Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1878358Z "size": 109, 2024-06-26T07:02:43.1878885Z "digest": "sha256:de45a14c9b227e4b0c9d45d42223b50a71c03522ddd9e2ca0d9517835206c823" 2024-06-26T07:02:43.1879487Z }, 2024-06-26T07:02:43.1879732Z { 2024-06-26T07:02:43.1880146Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1880689Z "size": 491, 2024-06-26T07:02:43.1881215Z "digest": "sha256:e99241e77209b0b35a0fb59a61c0821fc161190178af7e4af45e6eeaa043375e" 2024-06-26T07:02:43.1881812Z }, 2024-06-26T07:02:43.1882052Z { 2024-06-26T07:02:43.1882476Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1883018Z "size": 291, 2024-06-26T07:02:43.1883547Z "digest": "sha256:35e8f4e2a42f9df0c969dedd2199cf71719cf5ecfa676dfa014bbdeb4fdb295f" 2024-06-26T07:02:43.1884256Z }, 2024-06-26T07:02:43.1884510Z { 2024-06-26T07:02:43.1884941Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1885488Z "size": 101, 2024-06-26T07:02:43.1886019Z "digest": "sha256:a823ffa4c541394b52953391dc0a6867e26c37fe4a71fa022a2cc3f07f0ba70d" 2024-06-26T07:02:43.1886786Z }, 2024-06-26T07:02:43.1887035Z { 2024-06-26T07:02:43.1887451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1887992Z "size": 1473, 2024-06-26T07:02:43.1888509Z "digest": "sha256:44ba272eae2064603605a6238b3f35cc6578859d660068e702a047a9cd8ab2fd" 2024-06-26T07:02:43.1889101Z }, 2024-06-26T07:02:43.1889341Z { 2024-06-26T07:02:43.1889762Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1890307Z "size": 446710781, 2024-06-26T07:02:43.1890850Z "digest": "sha256:da02d278e5240c285bc855ce4e42cd6121d5b903787f5c14f03e575ad1268650" 2024-06-26T07:02:43.1891450Z }, 2024-06-26T07:02:43.1891703Z { 2024-06-26T07:02:43.1892132Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1892759Z "size": 161, 2024-06-26T07:02:43.1893330Z "digest": "sha256:73a0ce681c295135456e190578733273276b38b4dd5babbb39298f384d43b1c6" 2024-06-26T07:02:43.1893925Z }, 2024-06-26T07:02:43.1894174Z { 2024-06-26T07:02:43.1894597Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1895145Z "size": 564, 2024-06-26T07:02:43.1895679Z "digest": "sha256:49737bd5ae709dbf7956e21fc8b05ba009e9e6c9b130b0126a73bd3d3f5efc78" 2024-06-26T07:02:43.1896285Z }, 2024-06-26T07:02:43.1896535Z { 2024-06-26T07:02:43.1896959Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1897505Z "size": 35874600, 2024-06-26T07:02:43.1898053Z "digest": "sha256:9cd4c745f8e4f9d5c9bb8050923575afad21f5b7852975a4a1981bc9eff282c5" 2024-06-26T07:02:43.1898662Z }, 2024-06-26T07:02:43.1898911Z { 2024-06-26T07:02:43.1899348Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1899899Z "size": 104, 2024-06-26T07:02:43.1900422Z "digest": "sha256:b2950a0365345d4c17a48627c905fd8930b3b3154c727f0526eb9ca547102bb7" 2024-06-26T07:02:43.1901018Z }, 2024-06-26T07:02:43.1901270Z { 2024-06-26T07:02:43.1901691Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1902233Z "size": 425, 2024-06-26T07:02:43.1902762Z "digest": "sha256:25e19d81d9fc081962ff4cbc09e098f4035419822712c19a410f5282300dfb9e" 2024-06-26T07:02:43.1903403Z }, 2024-06-26T07:02:43.1903653Z { 2024-06-26T07:02:43.1904080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1904630Z "size": 20262084, 2024-06-26T07:02:43.1905180Z "digest": "sha256:60156b40fcbec2c25f0170b0a82fa8cd9303e0daae8bb1811fd0fc6562e88e8e" 2024-06-26T07:02:43.1905797Z }, 2024-06-26T07:02:43.1906046Z { 2024-06-26T07:02:43.1906479Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1907019Z "size": 439, 2024-06-26T07:02:43.1907547Z "digest": "sha256:7a868239227fd19b30a19927bbc8f058d247403fe5b343c55f4b4b8987918911" 2024-06-26T07:02:43.1908332Z }, 2024-06-26T07:02:43.1908604Z { 2024-06-26T07:02:43.1909032Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1909573Z "size": 702, 2024-06-26T07:02:43.1910102Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:43.1910702Z }, 2024-06-26T07:02:43.1910946Z { 2024-06-26T07:02:43.1911368Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1911907Z "size": 143, 2024-06-26T07:02:43.1912423Z "digest": "sha256:83911d7c7ecf11b5932bb7449975c52e7f737a5b867691935e1ccb85ceb25b52" 2024-06-26T07:02:43.1913020Z }, 2024-06-26T07:02:43.1913264Z { 2024-06-26T07:02:43.1913681Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1914316Z "size": 135, 2024-06-26T07:02:43.1914860Z "digest": "sha256:b3c5d66c679f58c71febdd502889bd8cd3d36e5001754cdf1e30eabe21135cce" 2024-06-26T07:02:43.1915481Z }, 2024-06-26T07:02:43.1915724Z { 2024-06-26T07:02:43.1916152Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1916705Z "size": 4911495247, 2024-06-26T07:02:43.1917258Z "digest": "sha256:84b2c7a899820f6acdb22148c5e8346d8b26c33269fd01842c9573515251e1e1" 2024-06-26T07:02:43.1917845Z }, 2024-06-26T07:02:43.1918098Z { 2024-06-26T07:02:43.1918525Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1919072Z "size": 190, 2024-06-26T07:02:43.1919613Z "digest": "sha256:c1068d03a811d8dead5da9e82fdc8869205fc4ee9cb92d4f7a4acfbb5ba83950" 2024-06-26T07:02:43.1920232Z }, 2024-06-26T07:02:43.1920481Z { 2024-06-26T07:02:43.1920903Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1921453Z "size": 565, 2024-06-26T07:02:43.1921983Z "digest": "sha256:671b4a301d81074fca17d36851f81801552c4ecb2f90ff3412aa9ab6a1a91371" 2024-06-26T07:02:43.1922671Z }, 2024-06-26T07:02:43.1922951Z { 2024-06-26T07:02:43.1923397Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1923947Z "size": 43163095, 2024-06-26T07:02:43.1924500Z "digest": "sha256:67760cf5ec46ab58d3ca98637fd9b26cb47a67446bfd0db176fe5e023606aa0d" 2024-06-26T07:02:43.1925104Z }, 2024-06-26T07:02:43.1925361Z { 2024-06-26T07:02:43.1925787Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1926420Z "size": 106, 2024-06-26T07:02:43.1926954Z "digest": "sha256:ddc9740bcea2b301e99b50525059aea8d9c7121a1c8eab566561a8968438a8c2" 2024-06-26T07:02:43.1927559Z }, 2024-06-26T07:02:43.1927805Z { 2024-06-26T07:02:43.1928220Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1928781Z "size": 1213, 2024-06-26T07:02:43.1929313Z "digest": "sha256:fb2e977685609fcc01de170ba7f50a573e73e6520e47253717e92a28952c5ee8" 2024-06-26T07:02:43.1929908Z }, 2024-06-26T07:02:43.1930153Z { 2024-06-26T07:02:43.1930572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1931112Z "size": 702, 2024-06-26T07:02:43.1931646Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:43.1932247Z }, 2024-06-26T07:02:43.1932495Z { 2024-06-26T07:02:43.1932957Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1933511Z "size": 138, 2024-06-26T07:02:43.1934049Z "digest": "sha256:b599c5a8c7cb461cbd7b3c98b99960b8eb04506e05e79db1cc0cfadb14a47e91" 2024-06-26T07:02:43.1934663Z }, 2024-06-26T07:02:43.1934907Z { 2024-06-26T07:02:43.1935323Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1935867Z "size": 120, 2024-06-26T07:02:43.1936409Z "digest": "sha256:e08e88c1366827455ee7ddbf92b6ae615ceb3cbd3fb11cd03ddd31bd3fa94a9c" 2024-06-26T07:02:43.1937021Z }, 2024-06-26T07:02:43.1937259Z { 2024-06-26T07:02:43.1937690Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1938234Z "size": 1904157495, 2024-06-26T07:02:43.1938798Z "digest": "sha256:12ba7431ff38de8cf6af3052a2a52e79f4e712b8c99f5bafde3c20e63b1f570f" 2024-06-26T07:02:43.1939400Z }, 2024-06-26T07:02:43.1939648Z { 2024-06-26T07:02:43.1940076Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1940610Z "size": 172, 2024-06-26T07:02:43.1941148Z "digest": "sha256:cf7c2cb896cba6a594028cc0fd246e127a2f4481d48905a3aa12c5f5872332d4" 2024-06-26T07:02:43.1941755Z }, 2024-06-26T07:02:43.1942004Z { 2024-06-26T07:02:43.1942417Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1942966Z "size": 908, 2024-06-26T07:02:43.1943509Z "digest": "sha256:3a778ddfac2af5b3a11f49ddf2f3f1ad1731f74ba691f434ab69966d1e2a189f" 2024-06-26T07:02:43.1944189Z }, 2024-06-26T07:02:43.1944429Z { 2024-06-26T07:02:43.1944852Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1945404Z "size": 702, 2024-06-26T07:02:43.1945937Z "digest": "sha256:df546d7b3e34396a6f2b51d8c88be976eca017bc99168d838c2c7bacabab9ec4" 2024-06-26T07:02:43.1946533Z }, 2024-06-26T07:02:43.1946779Z { 2024-06-26T07:02:43.1947201Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1947735Z "size": 134, 2024-06-26T07:02:43.1948440Z "digest": "sha256:236c8f0245f4b40616a4322eaf1e1b3e178d5990db1fcaba39622ea4e79c06a1" 2024-06-26T07:02:43.1949153Z }, 2024-06-26T07:02:43.1949407Z { 2024-06-26T07:02:43.1949828Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1950376Z "size": 32, 2024-06-26T07:02:43.1950914Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:43.1951529Z }, 2024-06-26T07:02:43.1951783Z { 2024-06-26T07:02:43.1952213Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1952848Z "size": 156, 2024-06-26T07:02:43.1953383Z "digest": "sha256:dda1ce302b9a9df0c8f1145d2e3eaa03c96909bb45a792451f1fcb555b06738d" 2024-06-26T07:02:43.1954006Z }, 2024-06-26T07:02:43.1954258Z { 2024-06-26T07:02:43.1954687Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1955226Z "size": 1841, 2024-06-26T07:02:43.1955769Z "digest": "sha256:e797fe86caef19319e03f8dca92cf96838ab3bb3644f4447c6dfc04230118afa" 2024-06-26T07:02:43.1956386Z }, 2024-06-26T07:02:43.1956640Z { 2024-06-26T07:02:43.1957063Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1957614Z "size": 7529784, 2024-06-26T07:02:43.1958163Z "digest": "sha256:967190076fc0ab2bb2fc0d74e58edd3d662a537d702ff3f8b6e876230154a16f" 2024-06-26T07:02:43.1958774Z }, 2024-06-26T07:02:43.1959017Z { 2024-06-26T07:02:43.1959451Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1959996Z "size": 165, 2024-06-26T07:02:43.1960532Z "digest": "sha256:4fb0a83f06855534ef82b80dcf7c685303edb2bcbd446196066dc2ad2dd71281" 2024-06-26T07:02:43.1961145Z }, 2024-06-26T07:02:43.1961395Z { 2024-06-26T07:02:43.1961822Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1962361Z "size": 7944, 2024-06-26T07:02:43.1962901Z "digest": "sha256:68af2ec4e30a33f26947ece9f3a31b7d3b04c2bca5865d4b48618457832a1da2" 2024-06-26T07:02:43.1963510Z }, 2024-06-26T07:02:43.1963761Z { 2024-06-26T07:02:43.1964180Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1964730Z "size": 8070, 2024-06-26T07:02:43.1965263Z "digest": "sha256:cc91933e985a044c6df53caa15488d2285896f5e991f38ec068c253ceec53f14" 2024-06-26T07:02:43.1965860Z }, 2024-06-26T07:02:43.1966188Z { 2024-06-26T07:02:43.1966646Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1967191Z "size": 303, 2024-06-26T07:02:43.1967718Z "digest": "sha256:9f3187916e7dcad33cc572f8f0e949d2f507b887f99cfd6acefd14826b44960d" 2024-06-26T07:02:43.1968337Z }, 2024-06-26T07:02:43.1968581Z { 2024-06-26T07:02:43.1969006Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1969545Z "size": 7629071, 2024-06-26T07:02:43.1970065Z "digest": "sha256:52608399485738d10d779254946c9e1d536010dd45593c69bf1eb9f62f85dc24" 2024-06-26T07:02:43.1970645Z }, 2024-06-26T07:02:43.1970883Z { 2024-06-26T07:02:43.1971308Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1971850Z "size": 108, 2024-06-26T07:02:43.1972380Z "digest": "sha256:05a6f6dcc1c00bac6660923a22adc7d7c34416718f8c89daf9b3351d9365f694" 2024-06-26T07:02:43.1972982Z }, 2024-06-26T07:02:43.1973228Z { 2024-06-26T07:02:43.1973652Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1974274Z "size": 54145775, 2024-06-26T07:02:43.1974828Z "digest": "sha256:6bc9dc8315b09b27c976bfba9700767ba412d72fd0528b18bbe8f8e62005ec22" 2024-06-26T07:02:43.1975440Z }, 2024-06-26T07:02:43.1975694Z { 2024-06-26T07:02:43.1976128Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1976666Z "size": 474, 2024-06-26T07:02:43.1977199Z "digest": "sha256:1b0e27c35f07b3fbf0aeb548d1949bf1d60a420136da21789d94d196c891d981" 2024-06-26T07:02:43.1977808Z }, 2024-06-26T07:02:43.1978056Z { 2024-06-26T07:02:43.1978484Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1979038Z "size": 1374858747, 2024-06-26T07:02:43.1979587Z "digest": "sha256:95237e7311a1c5ce6937db7196c866632f44e8788514a5c0eb4a3f04da4017e6" 2024-06-26T07:02:43.1980172Z }, 2024-06-26T07:02:43.1980425Z { 2024-06-26T07:02:43.1980856Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1981409Z "size": 106, 2024-06-26T07:02:43.1981956Z "digest": "sha256:789eff6c51aada70bf75eb1acafed63c0d839b768638001500616ff6e53b854a" 2024-06-26T07:02:43.1982634Z }, 2024-06-26T07:02:43.1982883Z { 2024-06-26T07:02:43.1983303Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1983854Z "size": 558, 2024-06-26T07:02:43.1984399Z "digest": "sha256:de0f8a8c45c6077b3c5ba3b84d7f07de98353fd618dad9a1e2f368ffe00493c7" 2024-06-26T07:02:43.1985021Z }, 2024-06-26T07:02:43.1985267Z { 2024-06-26T07:02:43.1985696Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1986245Z "size": 46248513, 2024-06-26T07:02:43.1986791Z "digest": "sha256:2060028f74888d218bebdd10d723f66c33ee1c708935c4ffd4654d34912ac4dc" 2024-06-26T07:02:43.1987385Z }, 2024-06-26T07:02:43.1987639Z { 2024-06-26T07:02:43.1988267Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1988864Z "size": 111, 2024-06-26T07:02:43.1989398Z "digest": "sha256:e412c7f3b2ded440992955bd92512345cbece8a8894cd3592f714ebc758ad1a0" 2024-06-26T07:02:43.1990001Z }, 2024-06-26T07:02:43.1990257Z { 2024-06-26T07:02:43.1990673Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1991217Z "size": 32, 2024-06-26T07:02:43.1991749Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:43.1992360Z }, 2024-06-26T07:02:43.1992602Z { 2024-06-26T07:02:43.1993028Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1993572Z "size": 32, 2024-06-26T07:02:43.1994099Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:43.1994703Z }, 2024-06-26T07:02:43.1994950Z { 2024-06-26T07:02:43.1995376Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1995918Z "size": 32, 2024-06-26T07:02:43.1996447Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:43.1997054Z }, 2024-06-26T07:02:43.1997299Z { 2024-06-26T07:02:43.1997725Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2024-06-26T07:02:43.1998270Z "size": 32, 2024-06-26T07:02:43.1998803Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2024-06-26T07:02:43.1999409Z } 2024-06-26T07:02:43.1999649Z ] 2024-06-26T07:02:43.1999895Z } 2024-06-26T07:02:43.2120635Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:43.2121121Z tag=${ECR_DOCKER_IMAGE##*/} 2024-06-26T07:02:43.2121676Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2024-06-26T07:02:43.2129692Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:43.2130191Z env: 2024-06-26T07:02:43.2130469Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:43.2131609Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.2132763Z ##[endgroup] 2024-06-26T07:02:43.2156260Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks-91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.2203621Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2024-06-26T07:02:43.2204193Z with: 2024-06-26T07:02:43.2205256Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.2206723Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:43.2207225Z env: 2024-06-26T07:02:43.2207508Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:43.2207854Z ##[endgroup] 2024-06-26T07:02:43.2222776Z ##[group]Run set -x 2024-06-26T07:02:43.2223113Z set -x 2024-06-26T07:02:43.2223411Z set +e 2024-06-26T07:02:43.2223704Z  2024-06-26T07:02:43.2223973Z login() { 2024-06-26T07:02:43.2224627Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2024-06-26T07:02:43.2225435Z } 2024-06-26T07:02:43.2225715Z  2024-06-26T07:02:43.2226014Z retry () { 2024-06-26T07:02:43.2226405Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2024-06-26T07:02:43.2226840Z } 2024-06-26T07:02:43.2227112Z  2024-06-26T07:02:43.2227426Z retry login "${DOCKER_REGISTRY}" 2024-06-26T07:02:43.2227832Z  2024-06-26T07:02:43.2228297Z set -e 2024-06-26T07:02:43.2228814Z # ignore output since only exit code is used for conditional 2024-06-26T07:02:43.2229512Z # only pull docker image if it's not available locally 2024-06-26T07:02:43.2230265Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2024-06-26T07:02:43.2230953Z  retry docker pull "${DOCKER_IMAGE}" 2024-06-26T07:02:43.2231388Z fi 2024-06-26T07:02:43.2238055Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:02:43.2238549Z env: 2024-06-26T07:02:43.2238823Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:02:43.2239948Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.2241226Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:43.2241730Z ##[endgroup] 2024-06-26T07:02:43.2261237Z + set +e 2024-06-26T07:02:43.2261837Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:43.2262458Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:43.2263490Z + aws ecr get-login-password --region us-east-1 2024-06-26T07:02:43.2264617Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2024-06-26T07:02:43.6254315Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2024-06-26T07:02:43.6255230Z Configure a credential helper to remove this warning. See 2024-06-26T07:02:43.6256088Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2024-06-26T07:02:43.6256616Z 2024-06-26T07:02:43.6256746Z Login Succeeded 2024-06-26T07:02:43.6265077Z + set -e 2024-06-26T07:02:43.6266354Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.6392310Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.6394397Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:02:43.8259806Z 91382da70d5719cd7007b6b80b71d2f48398f6b7: Pulling from pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks 2024-06-26T07:02:43.8261125Z 7a2c55901189: Pulling fs layer 2024-06-26T07:02:43.8261803Z 224fe954d725: Pulling fs layer 2024-06-26T07:02:43.8262590Z 75722010b82e: Pulling fs layer 2024-06-26T07:02:43.8263170Z d527cbbb87e3: Pulling fs layer 2024-06-26T07:02:43.8263673Z b57676e46aee: Pulling fs layer 2024-06-26T07:02:43.8264171Z a8c1e85b5e14: Pulling fs layer 2024-06-26T07:02:43.8264648Z a41a8d1c11c8: Pulling fs layer 2024-06-26T07:02:43.8265163Z 0c1227890755: Pulling fs layer 2024-06-26T07:02:43.8265660Z d8d1234baab3: Pulling fs layer 2024-06-26T07:02:43.8266107Z 7ed32bc8e469: Pulling fs layer 2024-06-26T07:02:43.8266616Z ec1e7978c1fe: Pulling fs layer 2024-06-26T07:02:43.8267115Z 309b872da805: Pulling fs layer 2024-06-26T07:02:43.8267551Z e82932cb002c: Pulling fs layer 2024-06-26T07:02:43.8267918Z ca9dd73e3e93: Pulling fs layer 2024-06-26T07:02:43.8268628Z 16d76a79d5d6: Pulling fs layer 2024-06-26T07:02:43.8269121Z d527cbbb87e3: Waiting 2024-06-26T07:02:43.8269546Z 0c1227890755: Waiting 2024-06-26T07:02:43.8270118Z 4bdf41c9ecbe: Pulling fs layer 2024-06-26T07:02:43.8270583Z b57676e46aee: Waiting 2024-06-26T07:02:43.8270928Z f4863801e30c: Pulling fs layer 2024-06-26T07:02:43.8271278Z d8d1234baab3: Waiting 2024-06-26T07:02:43.8271620Z d37dd9ed8948: Pulling fs layer 2024-06-26T07:02:43.8271988Z a8c1e85b5e14: Waiting 2024-06-26T07:02:43.8272306Z a41a8d1c11c8: Waiting 2024-06-26T07:02:43.8272638Z cf2f34f225db: Pulling fs layer 2024-06-26T07:02:43.8273044Z df546d7b3e34: Pulling fs layer 2024-06-26T07:02:43.8273443Z 3d914950f8ad: Pulling fs layer 2024-06-26T07:02:43.8273824Z 96d0e2032184: Pulling fs layer 2024-06-26T07:02:43.8274175Z ec1e7978c1fe: Waiting 2024-06-26T07:02:43.8274505Z 06e12ce59790: Pulling fs layer 2024-06-26T07:02:43.8274859Z 309b872da805: Waiting 2024-06-26T07:02:43.8275184Z ed712f63cde7: Pulling fs layer 2024-06-26T07:02:43.8275651Z df546d7b3e34: Waiting 2024-06-26T07:02:43.8275973Z 16d76a79d5d6: Waiting 2024-06-26T07:02:43.8276290Z f4863801e30c: Waiting 2024-06-26T07:02:43.8276724Z a97a71ef8a4e: Pulling fs layer 2024-06-26T07:02:43.8277098Z 4bdf41c9ecbe: Waiting 2024-06-26T07:02:43.8277421Z e82932cb002c: Waiting 2024-06-26T07:02:43.8277821Z d37dd9ed8948: Waiting 2024-06-26T07:02:43.8278146Z cf2f34f225db: Waiting 2024-06-26T07:02:43.8278473Z 25cd0c482ad6: Pulling fs layer 2024-06-26T07:02:43.8278839Z ca9dd73e3e93: Waiting 2024-06-26T07:02:43.8279157Z 3d914950f8ad: Waiting 2024-06-26T07:02:43.8279491Z 9bf94bc45658: Pulling fs layer 2024-06-26T07:02:43.8279854Z 96d0e2032184: Waiting 2024-06-26T07:02:43.8280188Z de45a14c9b22: Pulling fs layer 2024-06-26T07:02:43.8280550Z ed712f63cde7: Waiting 2024-06-26T07:02:43.8280862Z a97a71ef8a4e: Waiting 2024-06-26T07:02:43.8281191Z e99241e77209: Pulling fs layer 2024-06-26T07:02:43.8281552Z 06e12ce59790: Waiting 2024-06-26T07:02:43.8281884Z 35e8f4e2a42f: Pulling fs layer 2024-06-26T07:02:43.8282308Z a823ffa4c541: Pulling fs layer 2024-06-26T07:02:43.8282721Z 9bf94bc45658: Waiting 2024-06-26T07:02:43.8283079Z 44ba272eae20: Pulling fs layer 2024-06-26T07:02:43.8283499Z de45a14c9b22: Waiting 2024-06-26T07:02:43.8283844Z da02d278e524: Pulling fs layer 2024-06-26T07:02:43.8284219Z 73a0ce681c29: Pulling fs layer 2024-06-26T07:02:43.8284628Z 44ba272eae20: Waiting 2024-06-26T07:02:43.8284989Z 49737bd5ae70: Pulling fs layer 2024-06-26T07:02:43.8285357Z 35e8f4e2a42f: Waiting 2024-06-26T07:02:43.8285728Z 9cd4c745f8e4: Pulling fs layer 2024-06-26T07:02:43.8286086Z 73a0ce681c29: Waiting 2024-06-26T07:02:43.8286534Z b2950a036534: Pulling fs layer 2024-06-26T07:02:43.8286891Z e99241e77209: Waiting 2024-06-26T07:02:43.8287219Z 25e19d81d9fc: Pulling fs layer 2024-06-26T07:02:43.8287586Z 60156b40fcbe: Pulling fs layer 2024-06-26T07:02:43.8287956Z 7a868239227f: Pulling fs layer 2024-06-26T07:02:43.8288328Z 83911d7c7ecf: Pulling fs layer 2024-06-26T07:02:43.8288700Z b3c5d66c679f: Pulling fs layer 2024-06-26T07:02:43.8289066Z 84b2c7a89982: Pulling fs layer 2024-06-26T07:02:43.8289436Z c1068d03a811: Pulling fs layer 2024-06-26T07:02:43.8289804Z 671b4a301d81: Pulling fs layer 2024-06-26T07:02:43.8290157Z 9cd4c745f8e4: Waiting 2024-06-26T07:02:43.8290493Z 67760cf5ec46: Pulling fs layer 2024-06-26T07:02:43.8291081Z ddc9740bcea2: Pulling fs layer 2024-06-26T07:02:43.8291455Z 60156b40fcbe: Waiting 2024-06-26T07:02:43.8291820Z b3c5d66c679f: Waiting 2024-06-26T07:02:43.8292142Z 83911d7c7ecf: Waiting 2024-06-26T07:02:43.8292479Z fb2e97768560: Pulling fs layer 2024-06-26T07:02:43.8292840Z b2950a036534: Waiting 2024-06-26T07:02:43.8293147Z 84b2c7a89982: Waiting 2024-06-26T07:02:43.8293461Z 7a868239227f: Waiting 2024-06-26T07:02:43.8293775Z c1068d03a811: Waiting 2024-06-26T07:02:43.8294084Z 25e19d81d9fc: Waiting 2024-06-26T07:02:43.8294468Z b599c5a8c7cb: Pulling fs layer 2024-06-26T07:02:43.8294870Z fb2e97768560: Waiting 2024-06-26T07:02:43.8295200Z e08e88c13668: Pulling fs layer 2024-06-26T07:02:43.8295568Z 12ba7431ff38: Pulling fs layer 2024-06-26T07:02:43.8295929Z e08e88c13668: Waiting 2024-06-26T07:02:43.8296264Z cf7c2cb896cb: Pulling fs layer 2024-06-26T07:02:43.8296635Z 3a778ddfac2a: Pulling fs layer 2024-06-26T07:02:43.8297084Z 236c8f0245f4: Pulling fs layer 2024-06-26T07:02:43.8297466Z 4f4fb700ef54: Pulling fs layer 2024-06-26T07:02:43.8297855Z dda1ce302b9a: Pulling fs layer 2024-06-26T07:02:43.8298229Z e797fe86caef: Pulling fs layer 2024-06-26T07:02:43.8298604Z 967190076fc0: Pulling fs layer 2024-06-26T07:02:43.8298977Z 4fb0a83f0685: Pulling fs layer 2024-06-26T07:02:43.8299351Z 68af2ec4e30a: Pulling fs layer 2024-06-26T07:02:43.8299723Z cc91933e985a: Pulling fs layer 2024-06-26T07:02:43.8300103Z 9f3187916e7d: Pulling fs layer 2024-06-26T07:02:43.8300465Z 12ba7431ff38: Waiting 2024-06-26T07:02:43.8300790Z 526083994857: Pulling fs layer 2024-06-26T07:02:43.8301153Z dda1ce302b9a: Waiting 2024-06-26T07:02:43.8301494Z 05a6f6dcc1c0: Pulling fs layer 2024-06-26T07:02:43.8301856Z 4fb0a83f0685: Waiting 2024-06-26T07:02:43.8302189Z 6bc9dc8315b0: Pulling fs layer 2024-06-26T07:02:43.8302555Z e797fe86caef: Waiting 2024-06-26T07:02:43.8302891Z 1b0e27c35f07: Pulling fs layer 2024-06-26T07:02:43.8303254Z 68af2ec4e30a: Waiting 2024-06-26T07:02:43.8303580Z cc91933e985a: Waiting 2024-06-26T07:02:43.8303912Z 95237e7311a1: Pulling fs layer 2024-06-26T07:02:43.8304278Z 236c8f0245f4: Waiting 2024-06-26T07:02:43.8304605Z 789eff6c51aa: Pulling fs layer 2024-06-26T07:02:43.8304986Z de0f8a8c45c6: Pulling fs layer 2024-06-26T07:02:43.8305346Z 4f4fb700ef54: Waiting 2024-06-26T07:02:43.8305680Z 2060028f7488: Pulling fs layer 2024-06-26T07:02:43.8306028Z 9f3187916e7d: Waiting 2024-06-26T07:02:43.8306374Z e412c7f3b2de: Pulling fs layer 2024-06-26T07:02:43.8306737Z 3a778ddfac2a: Waiting 2024-06-26T07:02:43.8307056Z 6bc9dc8315b0: Waiting 2024-06-26T07:02:43.8307364Z 526083994857: Waiting 2024-06-26T07:02:43.8307677Z 95237e7311a1: Waiting 2024-06-26T07:02:43.8307992Z 1b0e27c35f07: Waiting 2024-06-26T07:02:43.8308430Z 2060028f7488: Waiting 2024-06-26T07:02:43.8308745Z 789eff6c51aa: Waiting 2024-06-26T07:02:43.8309117Z de0f8a8c45c6: Waiting 2024-06-26T07:02:43.9651446Z 224fe954d725: Verifying Checksum 2024-06-26T07:02:43.9651867Z 224fe954d725: Download complete 2024-06-26T07:02:44.0453137Z d527cbbb87e3: Verifying Checksum 2024-06-26T07:02:44.0453640Z d527cbbb87e3: Download complete 2024-06-26T07:02:44.1290177Z b57676e46aee: Verifying Checksum 2024-06-26T07:02:44.1290612Z b57676e46aee: Download complete 2024-06-26T07:02:44.1762195Z 7a2c55901189: Verifying Checksum 2024-06-26T07:02:44.1762649Z 7a2c55901189: Download complete 2024-06-26T07:02:44.2537196Z a41a8d1c11c8: Verifying Checksum 2024-06-26T07:02:44.2537685Z a41a8d1c11c8: Download complete 2024-06-26T07:02:44.3277561Z 0c1227890755: Verifying Checksum 2024-06-26T07:02:44.3278182Z 0c1227890755: Download complete 2024-06-26T07:02:44.3972574Z d8d1234baab3: Download complete 2024-06-26T07:02:44.4373863Z 75722010b82e: Verifying Checksum 2024-06-26T07:02:44.4374432Z 75722010b82e: Download complete 2024-06-26T07:02:44.5159156Z ec1e7978c1fe: Download complete 2024-06-26T07:02:44.5901154Z 309b872da805: Verifying Checksum 2024-06-26T07:02:44.5901746Z 309b872da805: Download complete 2024-06-26T07:02:45.2231662Z 7a2c55901189: Pull complete 2024-06-26T07:02:45.5138917Z 224fe954d725: Pull complete 2024-06-26T07:02:46.4521973Z 75722010b82e: Pull complete 2024-06-26T07:02:46.4999959Z d527cbbb87e3: Pull complete 2024-06-26T07:02:46.5433461Z b57676e46aee: Pull complete 2024-06-26T07:02:47.1131385Z e82932cb002c: Download complete 2024-06-26T07:02:47.1961356Z ca9dd73e3e93: Verifying Checksum 2024-06-26T07:02:47.1961825Z ca9dd73e3e93: Download complete 2024-06-26T07:02:47.2705751Z 16d76a79d5d6: Download complete 2024-06-26T07:02:47.4205938Z 4bdf41c9ecbe: Verifying Checksum 2024-06-26T07:02:47.4206597Z 4bdf41c9ecbe: Download complete 2024-06-26T07:02:48.3843040Z f4863801e30c: Verifying Checksum 2024-06-26T07:02:48.3843569Z f4863801e30c: Download complete 2024-06-26T07:02:48.4656979Z d37dd9ed8948: Download complete 2024-06-26T07:02:48.5481734Z cf2f34f225db: Download complete 2024-06-26T07:02:48.6379036Z df546d7b3e34: Verifying Checksum 2024-06-26T07:02:48.6379687Z df546d7b3e34: Download complete 2024-06-26T07:02:57.4633458Z a8c1e85b5e14: Verifying Checksum 2024-06-26T07:02:57.4634665Z a8c1e85b5e14: Download complete 2024-06-26T07:02:57.5410745Z 96d0e2032184: Verifying Checksum 2024-06-26T07:02:57.5411155Z 96d0e2032184: Download complete 2024-06-26T07:02:57.6049026Z 06e12ce59790: Verifying Checksum 2024-06-26T07:02:57.6049449Z 06e12ce59790: Download complete 2024-06-26T07:02:57.6773809Z ed712f63cde7: Download complete 2024-06-26T07:02:57.7429529Z a97a71ef8a4e: Verifying Checksum 2024-06-26T07:02:57.7429965Z a97a71ef8a4e: Download complete 2024-06-26T07:02:57.8291830Z 25cd0c482ad6: Verifying Checksum 2024-06-26T07:02:57.8292346Z 25cd0c482ad6: Download complete 2024-06-26T07:02:59.1554685Z 9bf94bc45658: Verifying Checksum 2024-06-26T07:02:59.1555293Z 9bf94bc45658: Download complete 2024-06-26T07:02:59.2293575Z de45a14c9b22: Verifying Checksum 2024-06-26T07:02:59.2294054Z de45a14c9b22: Download complete 2024-06-26T07:02:59.3065155Z e99241e77209: Download complete 2024-06-26T07:02:59.3916005Z 35e8f4e2a42f: Verifying Checksum 2024-06-26T07:02:59.3916618Z 35e8f4e2a42f: Download complete 2024-06-26T07:02:59.4845984Z a823ffa4c541: Verifying Checksum 2024-06-26T07:02:59.4846674Z a823ffa4c541: Download complete 2024-06-26T07:02:59.5621524Z 44ba272eae20: Verifying Checksum 2024-06-26T07:02:59.5622068Z 44ba272eae20: Download complete 2024-06-26T07:03:04.6276297Z da02d278e524: Verifying Checksum 2024-06-26T07:03:04.6276813Z da02d278e524: Download complete 2024-06-26T07:03:04.7110721Z 73a0ce681c29: Verifying Checksum 2024-06-26T07:03:04.7111336Z 73a0ce681c29: Download complete 2024-06-26T07:03:04.7819936Z 49737bd5ae70: Verifying Checksum 2024-06-26T07:03:04.7820473Z 49737bd5ae70: Download complete 2024-06-26T07:03:05.3379965Z 9cd4c745f8e4: Verifying Checksum 2024-06-26T07:03:05.4100524Z b2950a036534: Verifying Checksum 2024-06-26T07:03:05.4101138Z b2950a036534: Download complete 2024-06-26T07:03:05.4844775Z 25e19d81d9fc: Download complete 2024-06-26T07:03:05.7691695Z 60156b40fcbe: Verifying Checksum 2024-06-26T07:03:05.7692257Z 60156b40fcbe: Download complete 2024-06-26T07:03:05.8778059Z 7a868239227f: Verifying Checksum 2024-06-26T07:03:05.8778649Z 7a868239227f: Download complete 2024-06-26T07:03:05.9487958Z 83911d7c7ecf: Verifying Checksum 2024-06-26T07:03:05.9488523Z 83911d7c7ecf: Download complete 2024-06-26T07:03:06.0289495Z b3c5d66c679f: Verifying Checksum 2024-06-26T07:03:06.0290414Z b3c5d66c679f: Download complete 2024-06-26T07:03:09.7399040Z 7ed32bc8e469: Verifying Checksum 2024-06-26T07:03:09.7399772Z 7ed32bc8e469: Download complete 2024-06-26T07:03:09.8165891Z c1068d03a811: Verifying Checksum 2024-06-26T07:03:09.8167170Z c1068d03a811: Download complete 2024-06-26T07:03:09.8984535Z 671b4a301d81: Download complete 2024-06-26T07:03:10.3587482Z a8c1e85b5e14: Pull complete 2024-06-26T07:03:10.4442052Z 67760cf5ec46: Verifying Checksum 2024-06-26T07:03:10.4442759Z 67760cf5ec46: Download complete 2024-06-26T07:03:10.5173742Z a41a8d1c11c8: Pull complete 2024-06-26T07:03:10.5245592Z ddc9740bcea2: Verifying Checksum 2024-06-26T07:03:10.5246279Z ddc9740bcea2: Download complete 2024-06-26T07:03:10.5931067Z fb2e97768560: Verifying Checksum 2024-06-26T07:03:10.5931856Z fb2e97768560: Download complete 2024-06-26T07:03:10.6724945Z b599c5a8c7cb: Verifying Checksum 2024-06-26T07:03:10.6725411Z b599c5a8c7cb: Download complete 2024-06-26T07:03:10.7395341Z e08e88c13668: Download complete 2024-06-26T07:03:10.7464178Z 0c1227890755: Pull complete 2024-06-26T07:03:10.9604376Z d8d1234baab3: Pull complete 2024-06-26T07:03:41.6295863Z 7ed32bc8e469: Pull complete 2024-06-26T07:03:41.9612960Z ec1e7978c1fe: Pull complete 2024-06-26T07:03:42.1981650Z 309b872da805: Pull complete 2024-06-26T07:03:45.0176360Z 3d914950f8ad: Verifying Checksum 2024-06-26T07:03:45.0176838Z 3d914950f8ad: Download complete 2024-06-26T07:03:45.1056459Z cf7c2cb896cb: Download complete 2024-06-26T07:03:45.1741409Z 3a778ddfac2a: Verifying Checksum 2024-06-26T07:03:45.1741879Z 3a778ddfac2a: Download complete 2024-06-26T07:03:45.2479859Z 236c8f0245f4: Verifying Checksum 2024-06-26T07:03:45.2480611Z 236c8f0245f4: Download complete 2024-06-26T07:03:45.2598397Z 4f4fb700ef54: Download complete 2024-06-26T07:03:45.3636223Z dda1ce302b9a: Verifying Checksum 2024-06-26T07:03:45.4341686Z e797fe86caef: Download complete 2024-06-26T07:03:45.7681862Z 967190076fc0: Verifying Checksum 2024-06-26T07:03:45.7682353Z 967190076fc0: Download complete 2024-06-26T07:03:45.8335448Z 4fb0a83f0685: Verifying Checksum 2024-06-26T07:03:45.8336084Z 4fb0a83f0685: Download complete 2024-06-26T07:03:45.9348822Z 68af2ec4e30a: Download complete 2024-06-26T07:03:46.0196822Z cc91933e985a: Download complete 2024-06-26T07:03:46.1388710Z 9f3187916e7d: Download complete 2024-06-26T07:03:46.4239940Z 526083994857: Verifying Checksum 2024-06-26T07:03:46.4240495Z 526083994857: Download complete 2024-06-26T07:03:46.5107786Z 05a6f6dcc1c0: Download complete 2024-06-26T07:03:48.5371834Z 6bc9dc8315b0: Verifying Checksum 2024-06-26T07:03:48.5372318Z 6bc9dc8315b0: Download complete 2024-06-26T07:03:48.6355589Z 1b0e27c35f07: Verifying Checksum 2024-06-26T07:03:48.6357609Z 1b0e27c35f07: Download complete 2024-06-26T07:03:49.7070532Z e82932cb002c: Pull complete 2024-06-26T07:03:49.9164275Z ca9dd73e3e93: Pull complete 2024-06-26T07:03:50.1357876Z 16d76a79d5d6: Pull complete 2024-06-26T07:03:50.3583266Z 4bdf41c9ecbe: Pull complete 2024-06-26T07:03:52.0360764Z 12ba7431ff38: Verifying Checksum 2024-06-26T07:03:52.0361229Z 12ba7431ff38: Download complete 2024-06-26T07:03:52.1072321Z 789eff6c51aa: Verifying Checksum 2024-06-26T07:03:52.1072788Z 789eff6c51aa: Download complete 2024-06-26T07:03:52.2214939Z de0f8a8c45c6: Verifying Checksum 2024-06-26T07:03:52.2215727Z de0f8a8c45c6: Download complete 2024-06-26T07:03:52.7560698Z f4863801e30c: Pull complete 2024-06-26T07:03:52.9488106Z d37dd9ed8948: Pull complete 2024-06-26T07:03:53.1589804Z cf2f34f225db: Pull complete 2024-06-26T07:03:53.3639330Z df546d7b3e34: Pull complete 2024-06-26T07:03:53.4478813Z 2060028f7488: Verifying Checksum 2024-06-26T07:03:53.4479303Z 2060028f7488: Download complete 2024-06-26T07:03:53.5334992Z e412c7f3b2de: Download complete 2024-06-26T07:04:15.7652740Z 95237e7311a1: Download complete 2024-06-26T07:04:49.3813983Z 84b2c7a89982: Verifying Checksum 2024-06-26T07:04:49.3816602Z 84b2c7a89982: Download complete 2024-06-26T07:05:07.6568403Z 3d914950f8ad: Pull complete 2024-06-26T07:05:07.8430941Z 96d0e2032184: Pull complete 2024-06-26T07:05:08.0918612Z 06e12ce59790: Pull complete 2024-06-26T07:05:08.3143525Z ed712f63cde7: Pull complete 2024-06-26T07:05:08.5157177Z a97a71ef8a4e: Pull complete 2024-06-26T07:05:08.7221719Z 25cd0c482ad6: Pull complete 2024-06-26T07:05:12.3326966Z 9bf94bc45658: Pull complete 2024-06-26T07:05:12.4869654Z de45a14c9b22: Pull complete 2024-06-26T07:05:12.5873730Z e99241e77209: Pull complete 2024-06-26T07:05:12.7559662Z 35e8f4e2a42f: Pull complete 2024-06-26T07:05:12.9587158Z a823ffa4c541: Pull complete 2024-06-26T07:05:13.1560099Z 44ba272eae20: Pull complete 2024-06-26T07:05:21.4617972Z da02d278e524: Pull complete 2024-06-26T07:05:21.6661571Z 73a0ce681c29: Pull complete 2024-06-26T07:05:21.8099926Z 49737bd5ae70: Pull complete 2024-06-26T07:05:22.8003937Z 9cd4c745f8e4: Pull complete 2024-06-26T07:05:22.9868586Z b2950a036534: Pull complete 2024-06-26T07:05:23.1852082Z 25e19d81d9fc: Pull complete 2024-06-26T07:05:23.6766657Z 60156b40fcbe: Pull complete 2024-06-26T07:05:23.8889733Z 7a868239227f: Pull complete 2024-06-26T07:05:24.3402475Z 83911d7c7ecf: Pull complete 2024-06-26T07:05:24.4691224Z b3c5d66c679f: Pull complete 2024-06-26T07:06:03.5290450Z 84b2c7a89982: Pull complete 2024-06-26T07:06:03.7155248Z c1068d03a811: Pull complete 2024-06-26T07:06:03.9138569Z 671b4a301d81: Pull complete 2024-06-26T07:06:05.9294796Z 67760cf5ec46: Pull complete 2024-06-26T07:06:05.9707476Z ddc9740bcea2: Pull complete 2024-06-26T07:06:06.0034036Z fb2e97768560: Pull complete 2024-06-26T07:06:06.1074797Z b599c5a8c7cb: Pull complete 2024-06-26T07:06:06.1465743Z e08e88c13668: Pull complete 2024-06-26T07:06:40.0212628Z 12ba7431ff38: Pull complete 2024-06-26T07:06:40.2534096Z cf7c2cb896cb: Pull complete 2024-06-26T07:06:40.4778263Z 3a778ddfac2a: Pull complete 2024-06-26T07:06:40.9747457Z 236c8f0245f4: Pull complete 2024-06-26T07:06:41.1981804Z 4f4fb700ef54: Pull complete 2024-06-26T07:06:41.4192069Z dda1ce302b9a: Pull complete 2024-06-26T07:06:41.6455743Z e797fe86caef: Pull complete 2024-06-26T07:06:42.0307893Z 967190076fc0: Pull complete 2024-06-26T07:06:42.2065026Z 4fb0a83f0685: Pull complete 2024-06-26T07:06:42.4196953Z 68af2ec4e30a: Pull complete 2024-06-26T07:06:42.6521293Z cc91933e985a: Pull complete 2024-06-26T07:06:42.8766705Z 9f3187916e7d: Pull complete 2024-06-26T07:06:43.9860609Z 526083994857: Pull complete 2024-06-26T07:06:44.2207475Z 05a6f6dcc1c0: Pull complete 2024-06-26T07:06:46.3052676Z 6bc9dc8315b0: Pull complete 2024-06-26T07:06:46.5260396Z 1b0e27c35f07: Pull complete 2024-06-26T07:06:59.9760943Z 95237e7311a1: Pull complete 2024-06-26T07:07:00.1914858Z 789eff6c51aa: Pull complete 2024-06-26T07:07:00.4160807Z de0f8a8c45c6: Pull complete 2024-06-26T07:07:01.3078597Z 2060028f7488: Pull complete 2024-06-26T07:07:01.5192517Z e412c7f3b2de: Pull complete 2024-06-26T07:07:02.6081762Z Digest: sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T07:07:02.6501274Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:02.6721979Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:07:02.6777718Z ##[group]Run echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:02.6778652Z echo "IN_ARC_RUNNER=$([ -f /.inarc ] && echo true || echo false)" >> "$GITHUB_OUTPUT" 2024-06-26T07:07:02.6786780Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:02.6787297Z env: 2024-06-26T07:07:02.6787594Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:02.6787972Z ##[endgroup] 2024-06-26T07:07:02.6945064Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main 2024-06-26T07:07:02.6945624Z with: 2024-06-26T07:07:02.6945914Z driver-version: 550.54.15 2024-06-26T07:07:02.6946266Z env: 2024-06-26T07:07:02.6946540Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:02.6946895Z ##[endgroup] 2024-06-26T07:07:02.7093583Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:07:02.7094134Z with: 2024-06-26T07:07:02.7094412Z timeout_minutes: 10 2024-06-26T07:07:02.7094731Z max_attempts: 3 2024-06-26T07:07:02.7125892Z command: # Is it disgusting to have a full shell script here in this github action? Sure # But is it the best way to make it so that this action relies on nothing else? Absolutely set -eou pipefail DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID) DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run" install_nvidia_docker2_amzn2() { ( set -x # Needed for yum-config-manager sudo yum install -y yum-utils if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo" else # Amazon Linux 2 YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo" fi sudo yum-config-manager --add-repo "${YUM_REPO_URL}" sudo yum install -y nvidia-docker2 sudo systemctl restart docker ) } install_nvidia_docker2_ubuntu20() { ( set -x # Install nvidia-driver package if not installed status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)" if [ ! $? = 0 ] || [ ! "$status" = installed ]; then sudo apt-get install -y nvidia-docker2 sudo systemctl restart docker fi ) } pre_install_nvidia_driver_amzn2() { ( # Purge any nvidia driver installed from RHEL repo sudo yum remove -y nvidia-driver-latest-dkms ) } install_nvidia_driver_common() { ( # Try to gather more information about the runner and its existing NVIDIA driver if any echo "Before installing NVIDIA driver" lspci lsmod modinfo nvidia || true HAS_NVIDIA_DRIVER=0 # Check if NVIDIA driver has already been installed if [ -x "$(command -v nvidia-smi)" ]; then set +e # The driver exists, check its version next. Also check only the first GPU if there are more than one of them # so that the same driver version is not print over multiple lines INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing" elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing" else HAS_NVIDIA_DRIVER=1 echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation" fi set -e fi if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then # CAUTION: this may need to be updated in future if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then sudo yum groupinstall -y "Development Tools" # ensure our kernel install is the same as our underlying kernel, # groupinstall "Development Tools" has a habit of mismatching kernel headers sudo yum install -y "kernel-devel-uname-r == $(uname -r)" sudo modprobe backlight fi sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN" set +e sudo /bin/bash /tmp/nvidia_driver -s --no-drm NVIDIA_INSTALLATION_STATUS=$? RESET_GPU=0 if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then sudo cat /var/log/nvidia-installer.log # Fail to install NVIDIA driver, try to reset the GPU RESET_GPU=1 elif [ -x "$(command -v nvidia-smi)" ]; then # Check again if nvidia-smi works even if the driver installation completes successfully INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0) NVIDIA_SMI_STATUS=$? if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then RESET_GPU=1 fi fi if [ "$RESET_GPU" -eq 1 ]; then NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1) # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388 for PCI_ID in $NVIDIA_DEVICES; do DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable) echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)" # This requires sudo permission of course echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset sleep 1 done fi sudo rm -fv /tmp/nvidia_driver set -e fi ) } post_install_nvidia_driver_common() { ( sudo modprobe nvidia || true echo "After installing NVIDIA driver" lspci lsmod modinfo nvidia || true ( set +e nvidia-smi # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in # the case where the driver has already crashed as it still can get the driver version # and some basic information like the bus ID. However, the rest of the information # would be missing (ERR!), for example: # # +-----------------------------------------------------------------------------+ # | NVIDIA-SMI 525.89.02 Driver Version: 525.89.02 CUDA Version: 12.0 | # |-------------------------------+----------------------+----------------------+ # | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | # | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | # | | | MIG M. | # |===============================+======================+======================| # | 0 ERR! Off | 00000000:00:1E.0 Off | ERR! | # |ERR! ERR! ERR! ERR! / ERR! | 4184MiB / 23028MiB | ERR! Default | # | | | ERR! | # +-------------------------------+----------------------+----------------------+ # # +-----------------------------------------------------------------------------+ # | Processes: | # | GPU GI CI PID Type Process name GPU Memory | # | ID ID Usage | # |=============================================================================| # +-----------------------------------------------------------------------------+ # # This should be reported as a failure instead as it will guarantee to fail when # Docker tries to run with --gpus all # # So, the correct check here is to query one of the missing piece of info like # GPU name, so that the command can fail accordingly nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 NVIDIA_SMI_STATUS=$? # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285 if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}" else echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}" exit ${NVIDIA_SMI_STATUS} fi set -e ) ) } install_nvidia_driver_amzn2() { ( set -x pre_install_nvidia_driver_amzn2 install_nvidia_driver_common post_install_nvidia_driver_common ) } install_nvidia_driver_ubuntu20() { ( set -x install_nvidia_driver_common post_install_nvidia_driver_common ) } echo "== Installing nvidia driver ${DRIVER_FN} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_driver_amzn2 ;; ubuntu20.04) install_nvidia_driver_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac # Install container toolkit based on distribution echo "== Installing nvidia container toolkit for ${DISTRIBUTION} ==" case "${DISTRIBUTION}" in amzn*) install_nvidia_docker2_amzn2 ;; ubuntu20.04) install_nvidia_docker2_ubuntu20 ;; *) echo "ERROR: Unknown distribution ${DISTRIBUTION}" exit 1 ;; esac echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}" 2024-06-26T07:07:02.7157265Z retry_wait_seconds: 10 2024-06-26T07:07:02.7157625Z polling_interval_seconds: 1 2024-06-26T07:07:02.7157991Z warning_on_retry: true 2024-06-26T07:07:02.7158338Z continue_on_error: false 2024-06-26T07:07:02.7158672Z env: 2024-06-26T07:07:02.7158937Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:02.7159390Z DRIVER_VERSION: 550.54.15 2024-06-26T07:07:02.7159733Z ##[endgroup] 2024-06-26T07:07:02.7686707Z == Installing nvidia driver NVIDIA-Linux-x86_64-550.54.15.run == 2024-06-26T07:07:02.7687496Z + pre_install_nvidia_driver_amzn2 2024-06-26T07:07:03.0731969Z + sudo yum remove -y nvidia-driver-latest-dkms 2024-06-26T07:07:03.0732678Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:03.1119689Z No Match for argument: nvidia-driver-latest-dkms 2024-06-26T07:07:03.1361693Z No Packages marked for removal 2024-06-26T07:07:03.1486223Z + install_nvidia_driver_common 2024-06-26T07:07:03.1489453Z + echo 'Before installing NVIDIA driver' 2024-06-26T07:07:03.1490243Z Before installing NVIDIA driver 2024-06-26T07:07:03.1491599Z + lspci 2024-06-26T07:07:03.1570636Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:03.1571419Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:03.1572372Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:03.1573165Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:03.1573923Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:03.1574683Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:03.1575408Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:03.1576194Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:03.1576785Z + lsmod 2024-06-26T07:07:03.1585250Z Module Size Used by 2024-06-26T07:07:03.1585676Z nvidia_modeset 1339392 0 2024-06-26T07:07:03.1586054Z veth 16384 0 2024-06-26T07:07:03.1586429Z nvidia_uvm 4599808 0 2024-06-26T07:07:03.1586880Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:03.1587373Z drm 421888 1 nvidia 2024-06-26T07:07:03.1588052Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:03.1588866Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:03.1589309Z xt_conntrack 16384 1 2024-06-26T07:07:03.1589698Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:03.1590125Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:03.1590591Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:03.1591041Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:03.1591550Z xfrm_user 45056 1 2024-06-26T07:07:03.1591933Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:03.1592350Z iptable_nat 16384 1 2024-06-26T07:07:03.1592740Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:03.1593174Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:03.1593643Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:03.1594165Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:03.1595200Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:03.1596199Z xt_addrtype 16384 2 2024-06-26T07:07:03.1596579Z iptable_filter 16384 1 2024-06-26T07:07:03.1596963Z br_netfilter 24576 0 2024-06-26T07:07:03.1597383Z bridge 172032 1 br_netfilter 2024-06-26T07:07:03.1597820Z stp 16384 1 bridge 2024-06-26T07:07:03.1598235Z llc 16384 2 bridge,stp 2024-06-26T07:07:03.1598652Z overlay 86016 0 2024-06-26T07:07:03.1599024Z sunrpc 393216 1 2024-06-26T07:07:03.1599386Z dm_mirror 28672 0 2024-06-26T07:07:03.1599785Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:03.1600276Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:03.1600788Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:03.1601230Z dax 69632 1 dm_mod 2024-06-26T07:07:03.1601792Z crc32_pclmul 16384 0 2024-06-26T07:07:03.1602175Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:03.1602556Z pcbc 16384 0 2024-06-26T07:07:03.1602927Z aesni_intel 188416 0 2024-06-26T07:07:03.1603301Z mousedev 24576 0 2024-06-26T07:07:03.1603700Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:03.1604154Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:03.1604658Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:03.1605211Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:03.1605875Z evdev 20480 3 2024-06-26T07:07:03.1606257Z psmouse 32768 0 2024-06-26T07:07:03.1606631Z button 16384 0 2024-06-26T07:07:03.1606989Z ena 139264 0 2024-06-26T07:07:03.1607362Z ptp 20480 1 ena 2024-06-26T07:07:03.1607760Z pps_core 20480 1 ptp 2024-06-26T07:07:03.1608163Z crc32c_intel 24576 0 2024-06-26T07:07:03.1608621Z autofs4 49152 2 2024-06-26T07:07:03.1608985Z + modinfo nvidia 2024-06-26T07:07:03.1609684Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:03.1610368Z alias: char-major-195-* 2024-06-26T07:07:03.1610750Z version: 550.54.15 2024-06-26T07:07:03.1611098Z supported: external 2024-06-26T07:07:03.1611436Z license: NVIDIA 2024-06-26T07:07:03.1611804Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:03.1612287Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:03.1612749Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:03.1613206Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:03.1613704Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:03.1614203Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:03.1614689Z depends: i2c-core,drm 2024-06-26T07:07:03.1615173Z retpoline: Y 2024-06-26T07:07:03.1615489Z name: nvidia 2024-06-26T07:07:03.1616040Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:03.1616700Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:03.1617337Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:03.1617924Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:03.1618367Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:03.1618801Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:03.1619249Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:03.1619687Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:03.1620123Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:03.1620651Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:03.1621209Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:03.1621690Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:03.1622131Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:03.1622550Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:03.1623003Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:03.1623532Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:03.1624110Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:03.1624662Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:03.1625330Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:03.1625928Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:03.1626537Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:03.1627124Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:03.1627625Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:03.1628388Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:03.1628959Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:03.1629571Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:03.1630031Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:03.1630523Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:03.1631000Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:03.1631435Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:03.1631985Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:03.1632513Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:03.1632991Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:03.1633470Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:03.1633969Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:03.1634453Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:03.1634958Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:03.1635426Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:03.1635847Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:03.1636315Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:03.1636776Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:03.1637231Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:03.1637707Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:03.1638192Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:03.1638639Z parm: rm_firmware_active:charp 2024-06-26T07:07:03.1639052Z + HAS_NVIDIA_DRIVER=0 2024-06-26T07:07:03.1639440Z ++ command -v nvidia-smi 2024-06-26T07:07:03.1639833Z + '[' -x /usr/bin/nvidia-smi ']' 2024-06-26T07:07:03.1640193Z + set +e 2024-06-26T07:07:03.1640690Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0 2024-06-26T07:07:05.1548468Z + INSTALLED_DRIVER_VERSION=550.54.15 2024-06-26T07:07:05.1549062Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:05.1549649Z + '[' 0 -ne 0 ']' 2024-06-26T07:07:05.1550124Z + '[' 550.54.15 '!=' 550.54.15 ']' 2024-06-26T07:07:05.1550776Z + HAS_NVIDIA_DRIVER=1 2024-06-26T07:07:05.1551500Z + echo 'NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation' 2024-06-26T07:07:05.1552253Z + set -e 2024-06-26T07:07:05.1552551Z + '[' 1 -eq 0 ']' 2024-06-26T07:07:05.1553122Z NVIDIA driver (550.54.15) has already been installed. Skipping NVIDIA driver installation 2024-06-26T07:07:05.1553802Z + post_install_nvidia_driver_common 2024-06-26T07:07:05.1554205Z + sudo modprobe nvidia 2024-06-26T07:07:05.1651153Z + echo 'After installing NVIDIA driver' 2024-06-26T07:07:05.1651628Z + lspci 2024-06-26T07:07:05.1652093Z After installing NVIDIA driver 2024-06-26T07:07:05.1729910Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] 2024-06-26T07:07:05.1730814Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] 2024-06-26T07:07:05.1731748Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08) 2024-06-26T07:07:05.1732569Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111 2024-06-26T07:07:05.1733336Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. Device 8061 2024-06-26T07:07:05.1734085Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA) 2024-06-26T07:07:05.1734812Z 00:1e.0 3D controller: NVIDIA Corporation Device 2237 (rev a1) 2024-06-26T07:07:05.1735596Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller 2024-06-26T07:07:05.1736188Z + lsmod 2024-06-26T07:07:05.1743535Z Module Size Used by 2024-06-26T07:07:05.1743960Z nvidia_modeset 1339392 0 2024-06-26T07:07:05.1744421Z veth 16384 0 2024-06-26T07:07:05.1744901Z nvidia_uvm 4599808 0 2024-06-26T07:07:05.1745490Z nvidia 53989376 2 nvidia_uvm,nvidia_modeset 2024-06-26T07:07:05.1746125Z drm 421888 1 nvidia 2024-06-26T07:07:05.1746586Z i2c_core 77824 2 nvidia,drm 2024-06-26T07:07:05.1747076Z backlight 16384 1 nvidia_modeset 2024-06-26T07:07:05.1747784Z xt_conntrack 16384 1 2024-06-26T07:07:05.1748520Z ipt_MASQUERADE 16384 1 2024-06-26T07:07:05.1749061Z nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE 2024-06-26T07:07:05.1749541Z nf_conntrack_netlink 49152 0 2024-06-26T07:07:05.1749980Z nfnetlink 16384 2 nf_conntrack_netlink 2024-06-26T07:07:05.1750443Z xfrm_user 45056 1 2024-06-26T07:07:05.1750836Z xfrm_algo 16384 1 xfrm_user 2024-06-26T07:07:05.1751284Z iptable_nat 16384 1 2024-06-26T07:07:05.1751787Z nf_conntrack_ipv4 16384 3 2024-06-26T07:07:05.1752405Z nf_defrag_ipv4 16384 1 nf_conntrack_ipv4 2024-06-26T07:07:05.1753043Z nf_nat_ipv4 16384 1 iptable_nat 2024-06-26T07:07:05.1753737Z nf_nat 32768 2 nf_nat_masquerade_ipv4,nf_nat_ipv4 2024-06-26T07:07:05.1755006Z nf_conntrack 155648 7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink 2024-06-26T07:07:05.1756157Z xt_addrtype 16384 2 2024-06-26T07:07:05.1756700Z iptable_filter 16384 1 2024-06-26T07:07:05.1757239Z br_netfilter 24576 0 2024-06-26T07:07:05.1757818Z bridge 172032 1 br_netfilter 2024-06-26T07:07:05.1758465Z stp 16384 1 bridge 2024-06-26T07:07:05.1759064Z llc 16384 2 bridge,stp 2024-06-26T07:07:05.1759652Z overlay 86016 0 2024-06-26T07:07:05.1760146Z sunrpc 393216 1 2024-06-26T07:07:05.1760634Z dm_mirror 28672 0 2024-06-26T07:07:05.1761121Z dm_region_hash 20480 1 dm_mirror 2024-06-26T07:07:05.1761837Z dm_log 20480 2 dm_region_hash,dm_mirror 2024-06-26T07:07:05.1762431Z dm_mod 143360 2 dm_log,dm_mirror 2024-06-26T07:07:05.1762888Z dax 69632 1 dm_mod 2024-06-26T07:07:05.1763297Z crc32_pclmul 16384 0 2024-06-26T07:07:05.1763676Z ghash_clmulni_intel 16384 0 2024-06-26T07:07:05.1764234Z pcbc 16384 0 2024-06-26T07:07:05.1764625Z aesni_intel 188416 0 2024-06-26T07:07:05.1765000Z mousedev 24576 0 2024-06-26T07:07:05.1765386Z aes_x86_64 20480 1 aesni_intel 2024-06-26T07:07:05.1765930Z crypto_simd 16384 1 aesni_intel 2024-06-26T07:07:05.1766437Z glue_helper 16384 1 aesni_intel 2024-06-26T07:07:05.1767000Z cryptd 28672 3 crypto_simd,ghash_clmulni_intel,aesni_intel 2024-06-26T07:07:05.1767553Z evdev 20480 3 2024-06-26T07:07:05.1767918Z psmouse 32768 0 2024-06-26T07:07:05.1768292Z button 16384 0 2024-06-26T07:07:05.1768661Z ena 139264 0 2024-06-26T07:07:05.1769029Z ptp 20480 1 ena 2024-06-26T07:07:05.1769418Z pps_core 20480 1 ptp 2024-06-26T07:07:05.1769814Z crc32c_intel 24576 0 2024-06-26T07:07:05.1770191Z autofs4 49152 2 2024-06-26T07:07:05.1770551Z + modinfo nvidia 2024-06-26T07:07:05.1771235Z filename: /lib/modules/4.14.336-257.562.amzn2.x86_64/kernel/drivers/video/nvidia.ko 2024-06-26T07:07:05.1771934Z alias: char-major-195-* 2024-06-26T07:07:05.1772329Z version: 550.54.15 2024-06-26T07:07:05.1772677Z supported: external 2024-06-26T07:07:05.1773025Z license: NVIDIA 2024-06-26T07:07:05.1773415Z firmware: nvidia/550.54.15/gsp_tu10x.bin 2024-06-26T07:07:05.1773905Z firmware: nvidia/550.54.15/gsp_ga10x.bin 2024-06-26T07:07:05.1774362Z srcversion: 833721318DA517F0C2FEC97 2024-06-26T07:07:05.1774827Z alias: pci:v000010DEd*sv*sd*bc06sc80i00* 2024-06-26T07:07:05.1775328Z alias: pci:v000010DEd*sv*sd*bc03sc02i00* 2024-06-26T07:07:05.1775819Z alias: pci:v000010DEd*sv*sd*bc03sc00i00* 2024-06-26T07:07:05.1776292Z depends: i2c-core,drm 2024-06-26T07:07:05.1776665Z retpoline: Y 2024-06-26T07:07:05.1776977Z name: nvidia 2024-06-26T07:07:05.1777537Z vermagic: 4.14.336-257.562.amzn2.x86_64 SMP mod_unload modversions 2024-06-26T07:07:05.1778296Z parm: NvSwitchRegDwords:NvSwitch regkey (charp) 2024-06-26T07:07:05.1778950Z parm: NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp) 2024-06-26T07:07:05.1779558Z parm: NVreg_ResmanDebugLevel:int 2024-06-26T07:07:05.1780080Z parm: NVreg_RmLogonRC:int 2024-06-26T07:07:05.1780532Z parm: NVreg_ModifyDeviceFiles:int 2024-06-26T07:07:05.1781044Z parm: NVreg_DeviceFileUID:int 2024-06-26T07:07:05.1781532Z parm: NVreg_DeviceFileGID:int 2024-06-26T07:07:05.1781973Z parm: NVreg_DeviceFileMode:int 2024-06-26T07:07:05.1782557Z parm: NVreg_InitializeSystemMemoryAllocations:int 2024-06-26T07:07:05.1783175Z parm: NVreg_UsePageAttributeTable:int 2024-06-26T07:07:05.1783663Z parm: NVreg_EnablePCIeGen3:int 2024-06-26T07:07:05.1784148Z parm: NVreg_EnableMSI:int 2024-06-26T07:07:05.1784577Z parm: NVreg_TCEBypassMode:int 2024-06-26T07:07:05.1785106Z parm: NVreg_EnableStreamMemOPs:int 2024-06-26T07:07:05.1785684Z parm: NVreg_RestrictProfilingToAdminUsers:int 2024-06-26T07:07:05.1786266Z parm: NVreg_PreserveVideoMemoryAllocations:int 2024-06-26T07:07:05.1786837Z parm: NVreg_EnableS0ixPowerManagement:int 2024-06-26T07:07:05.1787448Z parm: NVreg_S0ixPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:05.1788046Z parm: NVreg_DynamicPowerManagement:int 2024-06-26T07:07:05.1789015Z parm: NVreg_DynamicPowerManagementVideoMemoryThreshold:int 2024-06-26T07:07:05.1789611Z parm: NVreg_EnableGpuFirmware:int 2024-06-26T07:07:05.1790105Z parm: NVreg_EnableGpuFirmwareLogs:int 2024-06-26T07:07:05.1790633Z parm: NVreg_OpenRmEnableUnsupportedGpus:int 2024-06-26T07:07:05.1791186Z parm: NVreg_EnableUserNUMAManagement:int 2024-06-26T07:07:05.1791694Z parm: NVreg_MemoryPoolSize:int 2024-06-26T07:07:05.1792275Z parm: NVreg_KMallocHeapMaxSize:int 2024-06-26T07:07:05.1792779Z parm: NVreg_VMallocHeapMaxSize:int 2024-06-26T07:07:05.1793264Z parm: NVreg_IgnoreMMIOCheck:int 2024-06-26T07:07:05.1793721Z parm: NVreg_NvLinkDisable:int 2024-06-26T07:07:05.1794226Z parm: NVreg_EnablePCIERelaxedOrderingMode:int 2024-06-26T07:07:05.1794743Z parm: NVreg_RegisterPCIDriver:int 2024-06-26T07:07:05.1795221Z parm: NVreg_EnableResizableBar:int 2024-06-26T07:07:05.1795719Z parm: NVreg_EnableDbgBreakpoint:int 2024-06-26T07:07:05.1796220Z parm: NVreg_EnableNonblockingOpen:int 2024-06-26T07:07:05.1796699Z parm: NVreg_RegistryDwords:charp 2024-06-26T07:07:05.1797224Z parm: NVreg_RegistryDwordsPerDevice:charp 2024-06-26T07:07:05.1797710Z parm: NVreg_RmMsg:charp 2024-06-26T07:07:05.1798127Z parm: NVreg_GpuBlacklist:charp 2024-06-26T07:07:05.1798598Z parm: NVreg_TemporaryFilePath:charp 2024-06-26T07:07:05.1799083Z parm: NVreg_ExcludedGpus:charp 2024-06-26T07:07:05.1799560Z parm: NVreg_DmaRemapPeerMmio:int 2024-06-26T07:07:05.1800046Z parm: NVreg_RmNvlinkBandwidth:charp 2024-06-26T07:07:05.1800523Z parm: NVreg_ImexChannelCount:int 2024-06-26T07:07:05.1800982Z parm: rm_firmware_active:charp 2024-06-26T07:07:05.1801391Z + set +e 2024-06-26T07:07:05.1801703Z + nvidia-smi 2024-06-26T07:07:06.7491620Z Wed Jun 26 07:07:06 2024 2024-06-26T07:07:06.7492996Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:06.7493928Z | NVIDIA-SMI 550.54.15 Driver Version: 550.54.15 CUDA Version: 12.4 | 2024-06-26T07:07:06.7494699Z |-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:06.7495488Z | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | 2024-06-26T07:07:06.7496403Z | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | 2024-06-26T07:07:06.7497421Z | | | MIG M. | 2024-06-26T07:07:06.7497962Z |=========================================+========================+======================| 2024-06-26T07:07:06.7620388Z | 0 NVIDIA A10G Off | 00000000:00:1E.0 Off | 0 | 2024-06-26T07:07:06.7621157Z | 0% 33C P0 59W / 300W | 0MiB / 23028MiB | 0% Default | 2024-06-26T07:07:06.7621824Z | | | N/A | 2024-06-26T07:07:06.7622722Z +-----------------------------------------+------------------------+----------------------+ 2024-06-26T07:07:06.7623300Z 2024-06-26T07:07:06.7623951Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:06.7624618Z | Processes: | 2024-06-26T07:07:06.7625330Z | GPU GI CI PID Type Process name GPU Memory | 2024-06-26T07:07:06.7626020Z | ID ID Usage | 2024-06-26T07:07:06.7626587Z |=========================================================================================| 2024-06-26T07:07:06.7627224Z | No running processes found | 2024-06-26T07:07:06.7627969Z +-----------------------------------------------------------------------------------------+ 2024-06-26T07:07:07.3789966Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0 2024-06-26T07:07:08.9528402Z NVIDIA A10G 2024-06-26T07:07:09.4079238Z + NVIDIA_SMI_STATUS=0 2024-06-26T07:07:09.4079786Z + '[' 0 -eq 0 ']' 2024-06-26T07:07:09.4080196Z + echo 'INFO: Ignoring allowed status 0' 2024-06-26T07:07:09.4080947Z + set -e 2024-06-26T07:07:09.4081242Z INFO: Ignoring allowed status 0 2024-06-26T07:07:09.4082610Z == Installing nvidia container toolkit for amzn2 == 2024-06-26T07:07:09.4085473Z + sudo yum install -y yum-utils 2024-06-26T07:07:09.7125086Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:11.2494099Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version 2024-06-26T07:07:11.2494795Z Nothing to do 2024-06-26T07:07:11.4016478Z + [[ amzn2 == \a\m\z\n\2\0\2\3 ]] 2024-06-26T07:07:11.4017589Z + YUM_REPO_URL=https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:11.4018879Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:11.7246191Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:11.7490695Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo 2024-06-26T07:07:11.7491901Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:11.7492856Z repo saved to /etc/yum.repos.d/nvidia-docker.repo 2024-06-26T07:07:11.7603132Z + sudo yum install -y nvidia-docker2 2024-06-26T07:07:12.0679502Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd 2024-06-26T07:07:13.5359055Z Package nvidia-docker2-2.13.0-1.noarch already installed and latest version 2024-06-26T07:07:13.5359710Z Nothing to do 2024-06-26T07:07:13.6864784Z + sudo systemctl restart docker 2024-06-26T07:07:55.8211311Z Command completed after 1 attempt(s). 2024-06-26T07:07:55.8259254Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:07:55.8260094Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 2024-06-26T07:07:55.8260867Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 & 2024-06-26T07:07:55.8261606Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2024-06-26T07:07:55.8269788Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:07:55.8270354Z env: 2024-06-26T07:07:55.8270705Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:55.8271229Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:07:55.8271769Z ##[endgroup] 2024-06-26T07:07:56.0429367Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:07:56.0586560Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1) 2024-06-26T07:07:56.0684196Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.7/site-packages (11.525.84) 2024-06-26T07:07:56.1610855Z Prepare all required actions 2024-06-26T07:07:56.1611586Z Getting action download info 2024-06-26T07:07:56.2669150Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2024-06-26T07:07:57.7195048Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a) 2024-06-26T07:07:57.8283626Z ##[group]Run ./.github/actions/download-build-artifacts 2024-06-26T07:07:57.8284110Z with: 2024-06-26T07:07:57.8284460Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:07:57.8284927Z s3-bucket: gha-artifacts 2024-06-26T07:07:57.8285275Z env: 2024-06-26T07:07:57.8285556Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:57.8286018Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:07:57.8286646Z ##[endgroup] 2024-06-26T07:07:57.8317907Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:07:57.8318343Z with: 2024-06-26T07:07:57.8318718Z name: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:07:57.8319175Z s3-bucket: gha-artifacts 2024-06-26T07:07:57.8319524Z region: us-east-1 2024-06-26T07:07:57.8319825Z env: 2024-06-26T07:07:57.8320111Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:07:57.8320557Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:07:57.8321046Z ##[endgroup] 2024-06-26T07:07:58.2395615Z (node:2649) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:07:58.2396990Z 2024-06-26T07:07:58.2397504Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:07:58.2398394Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:07:58.2399379Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:07:58.3155195Z Found 1 objects with prefix pytorch/pytorch/9673646808/linux-focal-cuda12.1-py3.10-gcc9-sm86/ 2024-06-26T07:07:58.3157315Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:22.1649285Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2024-06-26T07:08:22.1658201Z Artifact download has finished successfully 2024-06-26T07:08:22.1802126Z ##[group]Run unzip -o artifacts.zip 2024-06-26T07:08:22.1802554Z unzip -o artifacts.zip 2024-06-26T07:08:22.1810312Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:22.1810815Z env: 2024-06-26T07:08:22.1811091Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:22.1811548Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:22.1812033Z ##[endgroup] 2024-06-26T07:08:22.1838284Z Archive: artifacts.zip 2024-06-26T07:08:22.1839909Z creating: dist/ 2024-06-26T07:08:24.1558376Z inflating: dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:24.1559064Z creating: build/custom_test_artifacts/ 2024-06-26T07:08:24.1559649Z creating: build/custom_test_artifacts/custom-op-build/ 2024-06-26T07:08:24.1560380Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2024-06-26T07:08:24.1561210Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.1565529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.1566862Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.1567792Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.1568773Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.1569732Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.1570818Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.1572007Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.1572988Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.1573970Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.1575103Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.1576258Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.1577382Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.1578497Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.1579638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.1580794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.1581818Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.1582810Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.1618297Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.1656329Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.1657749Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.1700584Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.1701979Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.1703379Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.1704829Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.1706279Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.1707638Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.1709147Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.1710506Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.1711835Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.1713101Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.1714327Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.1715658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.1716866Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.1718027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.1719220Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.1768754Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.1825766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.1826932Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.1827912Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.1828897Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.1829786Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.1830702Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2024-06-26T07:08:24.1831704Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:24.1832849Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2024-06-26T07:08:24.1833934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2024-06-26T07:08:24.1834953Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2024-06-26T07:08:24.1836017Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:24.1837084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2024-06-26T07:08:24.1838280Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:24.1839352Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2024-06-26T07:08:24.1840400Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2024-06-26T07:08:24.1851526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2024-06-26T07:08:24.1978448Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2024-06-26T07:08:24.1979437Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2024-06-26T07:08:24.1980508Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2024-06-26T07:08:24.1981698Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2024-06-26T07:08:24.1982831Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2024-06-26T07:08:24.1983912Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2024-06-26T07:08:24.1985026Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2024-06-26T07:08:24.1986152Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2024-06-26T07:08:24.1987273Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2024-06-26T07:08:24.1988944Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2024-06-26T07:08:24.1990275Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2024-06-26T07:08:24.2002679Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2024-06-26T07:08:24.2081593Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2024-06-26T07:08:24.2082791Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.2083855Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.2084984Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.2086124Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.2087040Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.2087939Z inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc 2024-06-26T07:08:24.2088794Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2024-06-26T07:08:24.2089576Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2024-06-26T07:08:24.2090377Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2024-06-26T07:08:24.2196347Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2024-06-26T07:08:24.2256257Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2024-06-26T07:08:24.2257005Z creating: build/custom_test_artifacts/jit-hook-build/ 2024-06-26T07:08:24.2257714Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2024-06-26T07:08:24.2258527Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.2263776Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.2264704Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.2265647Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.2266621Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.2267581Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.2268814Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.2269908Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.2270891Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.2271874Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.2273021Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.2274163Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.2275288Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.2276409Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.2277549Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.2278691Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.2279704Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.2280691Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.2315161Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.2352842Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.2354265Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.2396865Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.2398261Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.2399767Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.2401213Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.2402598Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.2403978Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.2405341Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.2406859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.2408195Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.2409448Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.2410659Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.2411862Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.2413068Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.2414237Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.2415436Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.2462881Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.2520584Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.2521759Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.2522740Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.2523599Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.2524479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.2525417Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2024-06-26T07:08:24.2526611Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2024-06-26T07:08:24.2527793Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2024-06-26T07:08:24.2528920Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2024-06-26T07:08:24.2529972Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2024-06-26T07:08:24.2531066Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2024-06-26T07:08:24.2532269Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2024-06-26T07:08:24.2533370Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2024-06-26T07:08:24.2534454Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2024-06-26T07:08:24.2535538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2024-06-26T07:08:24.2545312Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2024-06-26T07:08:24.2607665Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2024-06-26T07:08:24.2610007Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.2612101Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.2614010Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.2615769Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.2616826Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.2617707Z inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc 2024-06-26T07:08:24.2618538Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2024-06-26T07:08:24.2619303Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2024-06-26T07:08:24.2620105Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2024-06-26T07:08:24.2660424Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2024-06-26T07:08:24.2661172Z creating: build/custom_test_artifacts/custom-backend-build/ 2024-06-26T07:08:24.2661932Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2024-06-26T07:08:24.2662811Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2024-06-26T07:08:24.2667829Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2024-06-26T07:08:24.2669012Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/ 2024-06-26T07:08:24.2669994Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeSystem.cmake 2024-06-26T07:08:24.2671048Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/ 2024-06-26T07:08:24.2672076Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/tmp/ 2024-06-26T07:08:24.2673230Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/CMakeCCompilerId.c 2024-06-26T07:08:24.2674389Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdC/a.out 2024-06-26T07:08:24.2675443Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/ 2024-06-26T07:08:24.2676487Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/tmp/ 2024-06-26T07:08:24.2677692Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/CMakeCXXCompilerId.cpp 2024-06-26T07:08:24.2678898Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCXX/a.out 2024-06-26T07:08:24.2680069Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_C.bin 2024-06-26T07:08:24.2681279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCCompiler.cmake 2024-06-26T07:08:24.2682488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CXX.bin 2024-06-26T07:08:24.2683697Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCXXCompiler.cmake 2024-06-26T07:08:24.2684908Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/ 2024-06-26T07:08:24.2686067Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/ 2024-06-26T07:08:24.2719302Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii 2024-06-26T07:08:24.2756843Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp 2024-06-26T07:08:24.2758437Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id 2024-06-26T07:08:24.2800734Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii 2024-06-26T07:08:24.2802183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c 2024-06-26T07:08:24.2803649Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu 2024-06-26T07:08:24.2805152Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c 2024-06-26T07:08:24.2806671Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx 2024-06-26T07:08:24.2820241Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin 2024-06-26T07:08:24.2821685Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin 2024-06-26T07:08:24.2823106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c 2024-06-26T07:08:24.2824509Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o 2024-06-26T07:08:24.2825861Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin 2024-06-26T07:08:24.2827163Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.reg.c 2024-06-26T07:08:24.2828567Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin 2024-06-26T07:08:24.2829854Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.fatbin.c 2024-06-26T07:08:24.2831097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/tmp/a_dlink.o 2024-06-26T07:08:24.2832356Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/CMakeCUDACompilerId.cu 2024-06-26T07:08:24.2887718Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CompilerIdCUDA/a.out 2024-06-26T07:08:24.2945031Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeDetermineCompilerABI_CUDA.bin 2024-06-26T07:08:24.2946313Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.26.4/CMakeCUDACompiler.cmake 2024-06-26T07:08:24.2947343Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2024-06-26T07:08:24.2963035Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2024-06-26T07:08:24.2964036Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2024-06-26T07:08:24.2965079Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2024-06-26T07:08:24.2966323Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:24.2967721Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2024-06-26T07:08:24.2968920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2024-06-26T07:08:24.2970061Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2024-06-26T07:08:24.2971236Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:24.2972488Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2024-06-26T07:08:24.2973665Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:24.2974839Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2024-06-26T07:08:24.2976014Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2024-06-26T07:08:24.2977256Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2024-06-26T07:08:24.3076092Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2024-06-26T07:08:24.3077247Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2024-06-26T07:08:24.3078427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2024-06-26T07:08:24.3079737Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2024-06-26T07:08:24.3080982Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2024-06-26T07:08:24.3082184Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2024-06-26T07:08:24.3083407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2024-06-26T07:08:24.3084802Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2024-06-26T07:08:24.3086131Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2024-06-26T07:08:24.3087362Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2024-06-26T07:08:24.3088592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2024-06-26T07:08:24.3101811Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2024-06-26T07:08:24.3154871Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2024-06-26T07:08:24.3156162Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2024-06-26T07:08:24.3157290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2024-06-26T07:08:24.3158320Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2024-06-26T07:08:24.3159279Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2024-06-26T07:08:24.3160235Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2024-06-26T07:08:24.3161194Z inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc 2024-06-26T07:08:24.3162106Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2024-06-26T07:08:24.3162939Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2024-06-26T07:08:24.3163914Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2024-06-26T07:08:24.3262231Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2024-06-26T07:08:24.3303203Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2024-06-26T07:08:24.3303849Z creating: build/lib/ 2024-06-26T07:08:24.3388521Z inflating: build/lib/libprotobuf-lite.a 2024-06-26T07:08:24.3826663Z inflating: build/lib/libprotobuf.a 2024-06-26T07:08:24.3835970Z inflating: build/lib/libpthreadpool.a 2024-06-26T07:08:24.3843483Z inflating: build/lib/libcpuinfo.a 2024-06-26T07:08:24.3851180Z inflating: build/lib/libcpuinfo_internals.a 2024-06-26T07:08:24.3851667Z inflating: build/lib/libclog.a 2024-06-26T07:08:24.3869313Z inflating: build/lib/libnnpack.a 2024-06-26T07:08:24.3871310Z inflating: build/lib/libnnpack_reference_layers.a 2024-06-26T07:08:24.3933695Z inflating: build/lib/libgtest.a 2024-06-26T07:08:24.4005514Z inflating: build/lib/libbenchmark.a 2024-06-26T07:08:24.4067449Z inflating: build/lib/libasmjit.a 2024-06-26T07:08:24.4074683Z inflating: build/lib/libittnotify.a 2024-06-26T07:08:24.4100824Z inflating: build/lib/libtensorpipe_uv.a 2024-06-26T07:08:24.4222942Z inflating: build/lib/libgloo.a 2024-06-26T07:08:24.4244578Z inflating: build/lib/libfmt.a 2024-06-26T07:08:24.4333213Z inflating: build/lib/libc10.so 2024-06-26T07:08:24.4334475Z inflating: build/lib/libcaffe2_nvrtc.so 2024-06-26T07:08:24.4335407Z inflating: build/lib/libfoxi_loader.a 2024-06-26T07:08:24.4336142Z inflating: build/lib/libtorch_global_deps.so 2024-06-26T07:08:24.4355223Z inflating: build/lib/libpytorch_qnnpack.a 2024-06-26T07:08:24.4372422Z inflating: build/lib/libgmock.a 2024-06-26T07:08:24.4372870Z inflating: build/lib/libgtest_main.a 2024-06-26T07:08:24.4373565Z inflating: build/lib/libbenchmark_main.a 2024-06-26T07:08:24.4857432Z inflating: build/lib/libprotoc.a 2024-06-26T07:08:24.5226183Z inflating: build/lib/libgloo_cuda.a 2024-06-26T07:08:25.4931185Z inflating: build/lib/libdnnl.a 2024-06-26T07:08:25.5479501Z inflating: build/lib/libtensorpipe.a 2024-06-26T07:08:25.5532212Z inflating: build/lib/libc10_cuda.so 2024-06-26T07:08:25.5532668Z inflating: build/lib/libgmock_main.a 2024-06-26T07:08:25.6732444Z inflating: build/lib/libfbgemm.a 2024-06-26T07:08:25.6977166Z inflating: build/lib/libtensorpipe_cuda.a 2024-06-26T07:08:25.7475258Z inflating: build/lib/libkineto.a 2024-06-26T07:08:25.7651696Z inflating: build/lib/libXNNPACK.a 2024-06-26T07:08:25.7692258Z inflating: build/lib/libonnx_proto.a 2024-06-26T07:08:25.8356611Z inflating: build/lib/libonnx.a 2024-06-26T07:08:28.1592298Z inflating: build/lib/libtorch_cpu.so 2024-06-26T07:08:28.1596920Z inflating: build/lib/libunbox_lib.a 2024-06-26T07:08:28.1600659Z inflating: build/lib/libshm.so 2024-06-26T07:08:30.1428814Z inflating: build/lib/libtorch_cuda.so 2024-06-26T07:08:30.1429724Z inflating: build/lib/libtorch.so 2024-06-26T07:08:30.1431773Z inflating: build/lib/libc10d_cuda_test.so 2024-06-26T07:08:30.9364972Z inflating: build/lib/libtorch_cuda_linalg.so 2024-06-26T07:08:31.1257897Z inflating: build/lib/libtorch_python.so 2024-06-26T07:08:31.1324009Z inflating: build/lib/libtorchbind_test.so 2024-06-26T07:08:31.1344153Z inflating: build/lib/libjitbackend_test.so 2024-06-26T07:08:31.1368928Z inflating: build/lib/libbackend_with_compiler.so 2024-06-26T07:08:31.1402305Z inflating: build/lib/libnnapi_backend.so 2024-06-26T07:08:31.1402783Z creating: build/bin/ 2024-06-26T07:08:31.1451440Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2024-06-26T07:08:31.1501218Z inflating: build/bin/c10_DeviceGuard_test 2024-06-26T07:08:31.1550393Z inflating: build/bin/c10_Device_test 2024-06-26T07:08:31.1608208Z inflating: build/bin/c10_DispatchKeySet_test 2024-06-26T07:08:31.1659666Z inflating: build/bin/c10_Scalar_test 2024-06-26T07:08:31.1707100Z inflating: build/bin/c10_StreamGuard_test 2024-06-26T07:08:31.1756991Z inflating: build/bin/c10_SymInt_test 2024-06-26T07:08:31.1809986Z inflating: build/bin/c10_InlineDeviceGuard_test 2024-06-26T07:08:31.1863601Z inflating: build/bin/c10_InlineStreamGuard_test 2024-06-26T07:08:31.1918436Z inflating: build/bin/c10_SizesAndStrides_test 2024-06-26T07:08:31.1987009Z inflating: build/bin/c10_cow_test 2024-06-26T07:08:31.2038619Z inflating: build/bin/c10_Bitset_test 2024-06-26T07:08:31.2086023Z inflating: build/bin/c10_ConstexprCrc_test 2024-06-26T07:08:31.2135183Z inflating: build/bin/c10_DeadlockDetection_test 2024-06-26T07:08:31.2184026Z inflating: build/bin/c10_Half_test 2024-06-26T07:08:31.2238963Z inflating: build/bin/c10_LeftRight_test 2024-06-26T07:08:31.2292336Z inflating: build/bin/c10_Metaprogramming_test 2024-06-26T07:08:31.2341216Z inflating: build/bin/c10_Synchronized_test 2024-06-26T07:08:31.2395126Z inflating: build/bin/c10_ThreadLocal_test 2024-06-26T07:08:31.2444841Z inflating: build/bin/c10_TypeIndex_test 2024-06-26T07:08:31.2494771Z inflating: build/bin/c10_TypeList_test 2024-06-26T07:08:31.2542344Z inflating: build/bin/c10_TypeTraits_test 2024-06-26T07:08:31.2593441Z inflating: build/bin/c10_accumulate_test 2024-06-26T07:08:31.2646602Z inflating: build/bin/c10_bfloat16_test 2024-06-26T07:08:31.2695395Z inflating: build/bin/c10_bit_cast_test 2024-06-26T07:08:31.2750758Z inflating: build/bin/c10_complex_math_test 2024-06-26T07:08:31.2804493Z inflating: build/bin/c10_complex_test 2024-06-26T07:08:31.2855822Z inflating: build/bin/c10_exception_test 2024-06-26T07:08:31.2904128Z inflating: build/bin/c10_flags_test 2024-06-26T07:08:31.2952848Z inflating: build/bin/c10_generic_math_test 2024-06-26T07:08:31.3111678Z inflating: build/bin/c10_intrusive_ptr_test 2024-06-26T07:08:31.3161193Z inflating: build/bin/c10_irange_test 2024-06-26T07:08:31.3213338Z inflating: build/bin/c10_lazy_test 2024-06-26T07:08:31.3268533Z inflating: build/bin/c10_logging_test 2024-06-26T07:08:31.3340747Z inflating: build/bin/c10_optional_test 2024-06-26T07:08:31.3401675Z inflating: build/bin/c10_ordered_preserving_dict_test 2024-06-26T07:08:31.3453864Z inflating: build/bin/c10_registry_test 2024-06-26T07:08:31.3599773Z inflating: build/bin/c10_small_vector_test 2024-06-26T07:08:31.3650009Z inflating: build/bin/c10_ssize_test 2024-06-26T07:08:31.3700365Z inflating: build/bin/c10_string_util_test 2024-06-26T07:08:31.3757539Z inflating: build/bin/c10_string_view_test 2024-06-26T07:08:31.3805859Z inflating: build/bin/c10_tempfile_test 2024-06-26T07:08:31.3853087Z inflating: build/bin/c10_intrusive_ptr_benchmark 2024-06-26T07:08:31.3907560Z inflating: build/bin/c10_typeid_test 2024-06-26T07:08:31.4339937Z inflating: build/bin/protoc-3.13.0.0 2024-06-26T07:08:31.4771774Z inflating: build/bin/protoc 2024-06-26T07:08:31.4823035Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test 2024-06-26T07:08:31.4874863Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream 2024-06-26T07:08:31.4925144Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device 2024-06-26T07:08:31.4975401Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes 2024-06-26T07:08:31.5026277Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads 2024-06-26T07:08:31.5077927Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks 2024-06-26T07:08:31.5124854Z inflating: build/bin/c10_cuda_CUDATest 2024-06-26T07:08:31.5176031Z inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block 2024-06-26T07:08:31.5494942Z inflating: build/bin/vec_test_all_types_DEFAULT 2024-06-26T07:08:31.5826319Z inflating: build/bin/vec_test_all_types_AVX512 2024-06-26T07:08:31.6171365Z inflating: build/bin/vec_test_all_types_AVX2 2024-06-26T07:08:31.6222757Z inflating: build/bin/FileStoreTest 2024-06-26T07:08:31.6276854Z inflating: build/bin/TCPStoreTest 2024-06-26T07:08:31.6328436Z inflating: build/bin/HashStoreTest 2024-06-26T07:08:31.6341484Z inflating: build/bin/ProcessGroupMPITest 2024-06-26T07:08:31.6393980Z inflating: build/bin/test_edge_op_registration 2024-06-26T07:08:31.6397855Z inflating: build/bin/torch_shm_manager 2024-06-26T07:08:31.6400098Z inflating: build/bin/example_allreduce 2024-06-26T07:08:31.6453455Z inflating: build/bin/test_dist_autograd 2024-06-26T07:08:31.6520733Z inflating: build/bin/test_cpp_rpc 2024-06-26T07:08:31.6522853Z inflating: build/bin/parallel_benchmark 2024-06-26T07:08:31.6587838Z inflating: build/bin/test_mobile_nnc 2024-06-26T07:08:31.6596644Z inflating: build/bin/aot_model_compiler_test 2024-06-26T07:08:31.6929795Z inflating: build/bin/test_lazy 2024-06-26T07:08:31.8074277Z inflating: build/bin/test_api 2024-06-26T07:08:31.8145557Z inflating: build/bin/Dict_test 2024-06-26T07:08:31.8196449Z inflating: build/bin/Dimname_test 2024-06-26T07:08:31.8258336Z inflating: build/bin/MaybeOwned_test 2024-06-26T07:08:31.8313834Z inflating: build/bin/NamedTensor_test 2024-06-26T07:08:31.8371632Z inflating: build/bin/apply_utils_test 2024-06-26T07:08:31.8428492Z inflating: build/bin/atest 2024-06-26T07:08:31.8490042Z inflating: build/bin/basic 2024-06-26T07:08:31.8542619Z inflating: build/bin/broadcast_test 2024-06-26T07:08:31.8592281Z inflating: build/bin/cpu_allocator_test 2024-06-26T07:08:31.8648893Z inflating: build/bin/cpu_generator_test 2024-06-26T07:08:31.8700450Z inflating: build/bin/cpu_profiling_allocator_test 2024-06-26T07:08:31.8791258Z inflating: build/bin/cpu_rng_test 2024-06-26T07:08:31.8839657Z inflating: build/bin/dispatch_key_set_test 2024-06-26T07:08:31.8888477Z inflating: build/bin/dlconvertor_test 2024-06-26T07:08:31.8944903Z inflating: build/bin/extension_backend_test 2024-06-26T07:08:31.8998388Z inflating: build/bin/half_test 2024-06-26T07:08:31.9091140Z inflating: build/bin/ivalue_test 2024-06-26T07:08:31.9139207Z inflating: build/bin/lazy_tensor_test 2024-06-26T07:08:31.9192368Z inflating: build/bin/math_kernel_test 2024-06-26T07:08:31.9244568Z inflating: build/bin/memory_format_test 2024-06-26T07:08:31.9296502Z inflating: build/bin/memory_overlapping_test 2024-06-26T07:08:31.9347505Z inflating: build/bin/mobile_memory_cleanup 2024-06-26T07:08:31.9401966Z inflating: build/bin/native_test 2024-06-26T07:08:31.9450978Z inflating: build/bin/operator_name_test 2024-06-26T07:08:31.9500933Z inflating: build/bin/operators_test 2024-06-26T07:08:31.9551212Z inflating: build/bin/packedtensoraccessor_test 2024-06-26T07:08:31.9616396Z inflating: build/bin/pow_test 2024-06-26T07:08:31.9672002Z inflating: build/bin/quantized_test 2024-06-26T07:08:31.9719938Z inflating: build/bin/reduce_ops_test 2024-06-26T07:08:31.9769453Z inflating: build/bin/reportMemoryUsage_test 2024-06-26T07:08:31.9824082Z inflating: build/bin/scalar_tensor_test 2024-06-26T07:08:31.9881046Z inflating: build/bin/scalar_test 2024-06-26T07:08:31.9931835Z inflating: build/bin/StorageUtils_test 2024-06-26T07:08:31.9982341Z inflating: build/bin/stride_properties_test 2024-06-26T07:08:32.0058888Z inflating: build/bin/tensor_iterator_test 2024-06-26T07:08:32.0111271Z inflating: build/bin/test_parallel 2024-06-26T07:08:32.0114680Z inflating: build/bin/thread_init_test 2024-06-26T07:08:32.0168522Z inflating: build/bin/type_ptr_test 2024-06-26T07:08:32.0226260Z inflating: build/bin/type_test 2024-06-26T07:08:32.0277300Z inflating: build/bin/undefined_tensor_test 2024-06-26T07:08:32.0278299Z inflating: build/bin/verify_api_visibility 2024-06-26T07:08:32.0345615Z inflating: build/bin/legacy_vmap_test 2024-06-26T07:08:32.0395859Z inflating: build/bin/weakref_test 2024-06-26T07:08:32.0445735Z inflating: build/bin/wrapdim_test 2024-06-26T07:08:32.0496306Z inflating: build/bin/xla_tensor_test 2024-06-26T07:08:32.0554370Z inflating: build/bin/IListRef_test 2024-06-26T07:08:32.0657325Z inflating: build/bin/List_test 2024-06-26T07:08:32.0721857Z inflating: build/bin/KernelFunction_test 2024-06-26T07:08:32.0840071Z inflating: build/bin/kernel_function_legacy_test 2024-06-26T07:08:32.0933080Z inflating: build/bin/kernel_function_test 2024-06-26T07:08:32.1056944Z inflating: build/bin/kernel_lambda_legacy_test 2024-06-26T07:08:32.1157190Z inflating: build/bin/kernel_lambda_test 2024-06-26T07:08:32.1216679Z inflating: build/bin/kernel_stackbased_test 2024-06-26T07:08:32.1309312Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2024-06-26T07:08:32.1358559Z inflating: build/bin/CppSignature_test 2024-06-26T07:08:32.1412895Z inflating: build/bin/backend_fallback_test 2024-06-26T07:08:32.1460195Z inflating: build/bin/op_allowlist_test 2024-06-26T07:08:32.1521855Z inflating: build/bin/inline_container_test 2024-06-26T07:08:32.1813359Z inflating: build/bin/op_registration_test 2024-06-26T07:08:32.1864707Z inflating: build/bin/cuda_apply_test 2024-06-26T07:08:32.1914242Z inflating: build/bin/cuda_allocator_test 2024-06-26T07:08:32.1971487Z inflating: build/bin/cuda_atomic_ops_test 2024-06-26T07:08:32.2024227Z inflating: build/bin/cuda_caching_host_allocator_test 2024-06-26T07:08:32.2091857Z inflating: build/bin/cuda_complex_math_test 2024-06-26T07:08:32.2148922Z inflating: build/bin/cuda_complex_test 2024-06-26T07:08:32.2197313Z inflating: build/bin/cuda_device_test 2024-06-26T07:08:32.2252697Z inflating: build/bin/cuda_cub_test 2024-06-26T07:08:32.2301482Z inflating: build/bin/cuda_dlconvertor_test 2024-06-26T07:08:32.2364226Z inflating: build/bin/cuda_distributions_test 2024-06-26T07:08:32.2419075Z inflating: build/bin/cuda_generator_test 2024-06-26T07:08:32.2467271Z inflating: build/bin/cuda_half_test 2024-06-26T07:08:32.2517180Z inflating: build/bin/cuda_integer_divider_test 2024-06-26T07:08:32.2565873Z inflating: build/bin/cuda_optional_test 2024-06-26T07:08:32.2615896Z inflating: build/bin/cuda_packedtensoraccessor_test 2024-06-26T07:08:32.2666857Z inflating: build/bin/cuda_reportMemoryUsage_test 2024-06-26T07:08:32.2715814Z inflating: build/bin/cuda_allocatorTraceTracker_test 2024-06-26T07:08:32.2774219Z inflating: build/bin/cuda_stream_test 2024-06-26T07:08:32.2821969Z inflating: build/bin/cuda_cudnn_test 2024-06-26T07:08:32.2872413Z inflating: build/bin/cuda_vectorized_test 2024-06-26T07:08:32.2886779Z inflating: build/bin/tutorial_tensorexpr 2024-06-26T07:08:32.2950296Z inflating: build/bin/ProcessGroupGlooTest 2024-06-26T07:08:32.3005767Z inflating: build/bin/ProcessGroupGlooAsyncTest 2024-06-26T07:08:32.3067087Z inflating: build/bin/ProcessGroupNCCLTest 2024-06-26T07:08:32.3127635Z inflating: build/bin/ProcessGroupNCCLErrorsTest 2024-06-26T07:08:32.3931874Z inflating: build/bin/test_tensorexpr 2024-06-26T07:08:32.4490783Z inflating: build/bin/test_jit 2024-06-26T07:08:32.4491225Z creating: .additional_ci_files/ 2024-06-26T07:08:32.4544325Z inflating: .additional_ci_files/test-times.json 2024-06-26T07:08:32.4755964Z inflating: .additional_ci_files/test-class-times.json 2024-06-26T07:08:32.4781001Z ##[group]Run rm artifacts.zip 2024-06-26T07:08:32.4781387Z rm artifacts.zip 2024-06-26T07:08:32.4789236Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:32.4789741Z env: 2024-06-26T07:08:32.4790038Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:32.4790487Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:32.4790972Z ##[endgroup] 2024-06-26T07:08:33.0305834Z ##[group]Run df -H 2024-06-26T07:08:33.0306297Z df -H 2024-06-26T07:08:33.0316544Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:33.0317403Z env: 2024-06-26T07:08:33.0317789Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.0318398Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.0319035Z ##[endgroup] 2024-06-26T07:08:33.0349395Z Filesystem Size Used Avail Use% Mounted on 2024-06-26T07:08:33.0349922Z devtmpfs 34G 0 34G 0% /dev 2024-06-26T07:08:33.0350389Z tmpfs 34G 0 34G 0% /dev/shm 2024-06-26T07:08:33.0351113Z tmpfs 34G 455k 34G 1% /run 2024-06-26T07:08:33.0351597Z tmpfs 34G 0 34G 0% /sys/fs/cgroup 2024-06-26T07:08:33.0352081Z /dev/nvme0n1p1 162G 43G 119G 27% / 2024-06-26T07:08:33.0389193Z Prepare all required actions 2024-06-26T07:08:33.0389631Z Getting action download info 2024-06-26T07:08:33.1397806Z ##[group]Run ./.github/actions/download-td-artifacts 2024-06-26T07:08:33.1398269Z with: 2024-06-26T07:08:33.1398542Z env: 2024-06-26T07:08:33.1398861Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1399330Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1399810Z ##[endgroup] 2024-06-26T07:08:33.1430416Z ##[group]Run seemethere/download-artifact-s3@v4 2024-06-26T07:08:33.1430862Z with: 2024-06-26T07:08:33.1431130Z name: td_results 2024-06-26T07:08:33.1431453Z s3-bucket: gha-artifacts 2024-06-26T07:08:33.1431803Z region: us-east-1 2024-06-26T07:08:33.1432105Z env: 2024-06-26T07:08:33.1432379Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.1432834Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.1433317Z ##[endgroup] 2024-06-26T07:08:33.5546058Z (node:2674) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2024-06-26T07:08:33.5547146Z 2024-06-26T07:08:33.5547561Z Please migrate your code to use AWS SDK for JavaScript (v3). 2024-06-26T07:08:33.5549024Z For more information, check the migration guide at https://a.co/7PzMCcy 2024-06-26T07:08:33.5550336Z (Use `node --trace-warnings ...` to show where the warning was created) 2024-06-26T07:08:33.6275139Z Found 0 objects with prefix pytorch/pytorch/9673646808/td_results/ 2024-06-26T07:08:33.6281277Z Artifact download has finished successfully 2024-06-26T07:08:33.6408438Z ##[group]Run mkdir -p .additional_ci_files 2024-06-26T07:08:33.6408977Z mkdir -p .additional_ci_files 2024-06-26T07:08:33.6409542Z mv td_results.json .additional_ci_files/td_results.json 2024-06-26T07:08:33.6417394Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:33.6417887Z env: 2024-06-26T07:08:33.6418174Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.6418645Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.6419130Z ##[endgroup] 2024-06-26T07:08:33.6452607Z mv: cannot stat 'td_results.json': No such file or directory 2024-06-26T07:08:33.6464511Z ##[error]Process completed with exit code 1. 2024-06-26T07:08:33.6495428Z ##[group]Run .github/scripts/parse_ref.py 2024-06-26T07:08:33.6495906Z .github/scripts/parse_ref.py 2024-06-26T07:08:33.6502720Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:33.6503083Z env: 2024-06-26T07:08:33.6503368Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.6503836Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.6504322Z ##[endgroup] 2024-06-26T07:08:33.6742712Z Prepare all required actions 2024-06-26T07:08:33.6779747Z ##[group]Run ./.github/actions/get-workflow-job-id 2024-06-26T07:08:33.6780208Z with: 2024-06-26T07:08:33.6780753Z github-token: *** 2024-06-26T07:08:33.6781064Z env: 2024-06-26T07:08:33.6781346Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.6781803Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.6782289Z ##[endgroup] 2024-06-26T07:08:33.6797269Z ##[group]Run set -eux 2024-06-26T07:08:33.6797607Z set -eux 2024-06-26T07:08:33.6798186Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2024-06-26T07:08:33.6806229Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:33.6806870Z env: 2024-06-26T07:08:33.6807159Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:33.6807616Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:33.6808269Z GITHUB_TOKEN: *** 2024-06-26T07:08:33.6808582Z ##[endgroup] 2024-06-26T07:08:33.6827872Z + python3 .github/scripts/get_workflow_job_id.py 9673646808 i-01e2eadbbc5643a4a 2024-06-26T07:08:34.8510487Z setting job-id=26688911727 2024-06-26T07:08:34.8511330Z setting job-name=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:34.8704201Z Prepare all required actions 2024-06-26T07:08:34.8704648Z Getting action download info 2024-06-26T07:08:34.9848349Z ##[group]Run ./.github/actions/filter-test-configs 2024-06-26T07:08:34.9848807Z with: 2024-06-26T07:08:34.9849256Z github-token: *** 2024-06-26T07:08:34.9857326Z test-matrix: {"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]} 2024-06-26T07:08:34.9866019Z job-name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:34.9866750Z env: 2024-06-26T07:08:34.9867038Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:34.9867501Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:34.9867981Z ##[endgroup] 2024-06-26T07:08:34.9907925Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482 2024-06-26T07:08:34.9908634Z with: 2024-06-26T07:08:34.9908905Z shell: bash 2024-06-26T07:08:34.9909209Z timeout_minutes: 10 2024-06-26T07:08:34.9909539Z max_attempts: 5 2024-06-26T07:08:34.9909851Z retry_wait_seconds: 30 2024-06-26T07:08:34.9910957Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:34.9912120Z polling_interval_seconds: 1 2024-06-26T07:08:34.9912493Z warning_on_retry: true 2024-06-26T07:08:34.9912837Z continue_on_error: false 2024-06-26T07:08:34.9913176Z env: 2024-06-26T07:08:34.9913455Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:34.9914050Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:34.9914683Z GITHUB_TOKEN: *** 2024-06-26T07:08:34.9915008Z ##[endgroup] 2024-06-26T07:08:35.0412629Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1 2024-06-26T07:08:35.2555505Z Defaulting to user installation because normal site-packages is not writeable 2024-06-26T07:08:35.2706237Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.7/site-packages (2.27.1) 2024-06-26T07:08:35.2851612Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.7/site-packages (6.0.1) 2024-06-26T07:08:35.2859978Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (1.26.19) 2024-06-26T07:08:35.3057928Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2024.6.2) 2024-06-26T07:08:35.3064866Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (2.0.12) 2024-06-26T07:08:35.3086398Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.27.1) (3.7) 2024-06-26T07:08:36.0415639Z Command completed after 1 attempt(s). 2024-06-26T07:08:36.0458932Z ##[group]Run set -x 2024-06-26T07:08:36.0459272Z set -x 2024-06-26T07:08:36.0459565Z  2024-06-26T07:08:36.0460127Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:36.0460798Z # in runner workspace 2024-06-26T07:08:36.0461322Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2024-06-26T07:08:36.0469262Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.0469802Z env: 2024-06-26T07:08:36.0470081Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.0470544Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.0471024Z ##[endgroup] 2024-06-26T07:08:36.0491313Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2024-06-26T07:08:36.0688598Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:36.0689154Z echo "Workflow: ${GITHUB_WORKFLOW}" 2024-06-26T07:08:36.0689610Z echo "Job name: ${JOB_NAME}" 2024-06-26T07:08:36.0690004Z  2024-06-26T07:08:36.0690567Z # Use relative path here as this could be checked out anywhere, not necessarily 2024-06-26T07:08:36.0691242Z # in runner workspace 2024-06-26T07:08:36.0691807Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2024-06-26T07:08:36.0692440Z  --workflow "${GITHUB_WORKFLOW}" \ 2024-06-26T07:08:36.0692895Z  --job-name "${JOB_NAME}" \ 2024-06-26T07:08:36.0701255Z  --test-matrix "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \ 2024-06-26T07:08:36.0710071Z  --selected-test-configs "" \ 2024-06-26T07:08:36.0710540Z  --pr-number "${PR_NUMBER}" \ 2024-06-26T07:08:36.0710966Z  --tag "${TAG}" \ 2024-06-26T07:08:36.0711368Z  --event-name "${EVENT_NAME}" \ 2024-06-26T07:08:36.0711813Z  --schedule "${SCHEDULE}" \ 2024-06-26T07:08:36.0712246Z  --branch "${HEAD_BRANCH}" 2024-06-26T07:08:36.0719546Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.0720051Z env: 2024-06-26T07:08:36.0720331Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.0720800Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.0721474Z GITHUB_TOKEN: *** 2024-06-26T07:08:36.0722132Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:36.0722850Z PR_NUMBER: 2024-06-26T07:08:36.0723167Z TAG: ciflow/inductor/129470 2024-06-26T07:08:36.0723537Z EVENT_NAME: push 2024-06-26T07:08:36.0723856Z SCHEDULE: 2024-06-26T07:08:36.0724142Z HEAD_BRANCH: 2024-06-26T07:08:36.0724443Z ##[endgroup] 2024-06-26T07:08:36.0760142Z Workflow: inductor 2024-06-26T07:08:36.0761008Z Job name: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:36.2759737Z INFO:root:Found no test-config label on the PR, so all test configs are included 2024-06-26T07:08:36.4177355Z ##[group]Run echo "Filtered matrix:" 2024-06-26T07:08:36.4177825Z echo "Filtered matrix:" 2024-06-26T07:08:36.4185925Z echo "{"include": [{"config": "inductor", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_distributed", "shard": 1, "num_shards": 1, "runner": "linux.g5.12xlarge.nvidia.gpu"}, {"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "dynamic_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "aot_inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_cpp_wrapper_abi_compatible", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" 2024-06-26T07:08:36.4194802Z  2024-06-26T07:08:36.4195086Z echo 2024-06-26T07:08:36.4195473Z echo "Is the current job unstable? False" 2024-06-26T07:08:36.4195926Z  2024-06-26T07:08:36.4196206Z echo 2024-06-26T07:08:36.4196572Z echo "Is keep-going label set? False" 2024-06-26T07:08:36.4197011Z  2024-06-26T07:08:36.4197278Z echo 2024-06-26T07:08:36.4197603Z echo "Renabled issues? " 2024-06-26T07:08:36.4205245Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.4205901Z env: 2024-06-26T07:08:36.4206185Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.4206650Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.4207143Z ##[endgroup] 2024-06-26T07:08:36.4226686Z Filtered matrix: 2024-06-26T07:08:36.4236302Z {include: [{config: inductor, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_distributed, shard: 1, num_shards: 1, runner: linux.g5.12xlarge.nvidia.gpu}, {config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: dynamic_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: aot_inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_cpp_wrapper_abi_compatible, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}]} 2024-06-26T07:08:36.4244382Z 2024-06-26T07:08:36.4244547Z Is the current job unstable? False 2024-06-26T07:08:36.4244825Z 2024-06-26T07:08:36.4245087Z Is keep-going label set? False 2024-06-26T07:08:36.4245350Z 2024-06-26T07:08:36.4245473Z Renabled issues? 2024-06-26T07:08:36.4287478Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:36.4288181Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2024-06-26T07:08:36.4295886Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T07:08:36.4296386Z env: 2024-06-26T07:08:36.4296671Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.4297130Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.4297621Z JOB_TIMEOUT: 240 2024-06-26T07:08:36.4297925Z ##[endgroup] 2024-06-26T07:08:36.4369911Z ##[group]Run set -x 2024-06-26T07:08:36.4370460Z set -x 2024-06-26T07:08:36.4370763Z  2024-06-26T07:08:36.4371123Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2024-06-26T07:08:36.4371665Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2024-06-26T07:08:36.4372224Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2024-06-26T07:08:36.4372744Z  TEST_COMMAND=.ci/onnx/test.sh 2024-06-26T07:08:36.4373163Z else 2024-06-26T07:08:36.4373505Z  TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:36.4373929Z fi 2024-06-26T07:08:36.4374211Z  2024-06-26T07:08:36.4374686Z # detached container should get cleaned up by teardown_ec2_linux 2024-06-26T07:08:36.4375434Z # TODO: Stop building test binaries as part of the build phase 2024-06-26T07:08:36.4376115Z # Used for GPU_FLAG since that doesn't play nice 2024-06-26T07:08:36.4376710Z # shellcheck disable=SC2086,SC2090 2024-06-26T07:08:36.4377174Z container_name=$(docker run \ 2024-06-26T07:08:36.4377608Z  ${GPU_FLAG:-} \ 2024-06-26T07:08:36.4377992Z  -e BUILD_ENVIRONMENT \ 2024-06-26T07:08:36.4378395Z  -e PR_NUMBER \ 2024-06-26T07:08:36.4378758Z  -e GITHUB_ACTIONS \ 2024-06-26T07:08:36.4379161Z  -e GITHUB_REPOSITORY \ 2024-06-26T07:08:36.4379575Z  -e GITHUB_WORKFLOW \ 2024-06-26T07:08:36.4379967Z  -e GITHUB_JOB \ 2024-06-26T07:08:36.4380328Z  -e GITHUB_RUN_ID \ 2024-06-26T07:08:36.4380721Z  -e GITHUB_RUN_NUMBER \ 2024-06-26T07:08:36.4381131Z  -e GITHUB_RUN_ATTEMPT \ 2024-06-26T07:08:36.4381526Z  -e JOB_ID \ 2024-06-26T07:08:36.4381876Z  -e JOB_NAME \ 2024-06-26T07:08:36.4382236Z  -e BASE_SHA \ 2024-06-26T07:08:36.4382588Z  -e BRANCH \ 2024-06-26T07:08:36.4382921Z  -e SHA1 \ 2024-06-26T07:08:36.4383273Z  -e AWS_DEFAULT_REGION \ 2024-06-26T07:08:36.4383689Z  -e IN_WHEEL_TEST \ 2024-06-26T07:08:36.4384077Z  -e SHARD_NUMBER \ 2024-06-26T07:08:36.4384444Z  -e TEST_CONFIG \ 2024-06-26T07:08:36.4384818Z  -e NUM_TEST_SHARDS \ 2024-06-26T07:08:36.4385219Z  -e REENABLED_ISSUES \ 2024-06-26T07:08:36.4385636Z  -e CONTINUE_THROUGH_ERROR \ 2024-06-26T07:08:36.4386054Z  -e VERBOSE_TEST_LOGS \ 2024-06-26T07:08:36.4386464Z  -e NO_TEST_TIMEOUT \ 2024-06-26T07:08:36.4386844Z  -e NO_TD \ 2024-06-26T07:08:36.4387187Z  -e TD_DISTRIBUTED \ 2024-06-26T07:08:36.4387571Z  -e PR_LABELS \ 2024-06-26T07:08:36.4387982Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2024-06-26T07:08:36.4388682Z  -e SCCACHE_BUCKET \ 2024-06-26T07:08:36.4389074Z  -e SCCACHE_S3_KEY_PREFIX \ 2024-06-26T07:08:36.4389530Z  -e XLA_CUDA \ 2024-06-26T07:08:36.4389943Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2024-06-26T07:08:36.4390459Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2024-06-26T07:08:36.4390989Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2024-06-26T07:08:36.4391503Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2024-06-26T07:08:36.4391972Z  -e HUGGING_FACE_HUB_TOKEN \ 2024-06-26T07:08:36.4392398Z  -e DASHBOARD_TAG \ 2024-06-26T07:08:36.4392869Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2024-06-26T07:08:36.4393422Z  --security-opt seccomp=unconfined \ 2024-06-26T07:08:36.4393885Z  --cap-add=SYS_PTRACE \ 2024-06-26T07:08:36.4394276Z  --ipc=host \ 2024-06-26T07:08:36.4394630Z  --shm-size="${SHM_SIZE}" \ 2024-06-26T07:08:36.4395027Z  --tty \ 2024-06-26T07:08:36.4395352Z  --detach \ 2024-06-26T07:08:36.4395710Z  --name="${container_name}" \ 2024-06-26T07:08:36.4396124Z  --user jenkins \ 2024-06-26T07:08:36.4396720Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2024-06-26T07:08:36.4397341Z  -w /var/lib/jenkins/workspace \ 2024-06-26T07:08:36.4397785Z  "${DOCKER_IMAGE}" 2024-06-26T07:08:36.4398131Z ) 2024-06-26T07:08:36.4398539Z # Propagate download.pytorch.org IP to container 2024-06-26T07:08:36.4399457Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2024-06-26T07:08:36.4400426Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2024-06-26T07:08:36.4401327Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2024-06-26T07:08:36.4408738Z shell: /usr/bin/bash -e {0} 2024-06-26T07:08:36.4409090Z env: 2024-06-26T07:08:36.4409368Z GIT_DEFAULT_BRANCH: main 2024-06-26T07:08:36.4409815Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:36.4410423Z BUILD_ENVIRONMENT: linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:36.4410924Z PR_NUMBER: 2024-06-26T07:08:36.4411255Z GITHUB_REPOSITORY: pytorch/pytorch 2024-06-26T07:08:36.4411664Z GITHUB_WORKFLOW: inductor 2024-06-26T07:08:36.4412019Z GITHUB_JOB: test 2024-06-26T07:08:36.4412339Z GITHUB_RUN_ID: 9673646808 2024-06-26T07:08:36.4412697Z GITHUB_RUN_NUMBER: 76977 2024-06-26T07:08:36.4413044Z GITHUB_RUN_ATTEMPT: 1 2024-06-26T07:08:36.4413382Z JOB_ID: 26688911727 2024-06-26T07:08:36.4414039Z JOB_NAME: cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:36.4414751Z BRANCH: 2024-06-26T07:08:36.4415082Z SHA1: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:36.4415590Z BASE_SHA: b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:36.4416065Z TEST_CONFIG: dynamic_inductor_timm 2024-06-26T07:08:36.4416465Z SHARD_NUMBER: 1 2024-06-26T07:08:36.4416768Z NUM_TEST_SHARDS: 2 2024-06-26T07:08:36.4417090Z REENABLED_ISSUES: 2024-06-26T07:08:36.4417441Z CONTINUE_THROUGH_ERROR: False 2024-06-26T07:08:36.4417822Z VERBOSE_TEST_LOGS: False 2024-06-26T07:08:36.4418185Z NO_TEST_TIMEOUT: False 2024-06-26T07:08:36.4418523Z NO_TD: False 2024-06-26T07:08:36.4418825Z TD_DISTRIBUTED: False 2024-06-26T07:08:36.4419243Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2024-06-26T07:08:36.4419777Z SCCACHE_S3_KEY_PREFIX: inductor 2024-06-26T07:08:36.4420155Z SHM_SIZE: 2g 2024-06-26T07:08:36.4421227Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:36.4422366Z XLA_CUDA: 2024-06-26T07:08:36.4422849Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:36.4423465Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2024-06-26T07:08:36.4423906Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2024-06-26T07:08:36.4424308Z DASHBOARD_TAG: 2024-06-26T07:08:36.4424805Z HUGGING_FACE_HUB_TOKEN: *** 2024-06-26T07:08:36.4425181Z ##[endgroup] 2024-06-26T07:08:36.4444212Z + [[ dynamic_inductor_timm == \m\u\l\t\i\g\p\u ]] 2024-06-26T07:08:36.4444960Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *onnx* ]] 2024-06-26T07:08:36.4445483Z + TEST_COMMAND=.ci/pytorch/test.sh 2024-06-26T07:08:36.4451535Z +++ nproc --ignore=2 2024-06-26T07:08:36.4464530Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e DASHBOARD_TAG --env-file=/tmp/github_env_9673646808 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T07:08:47.5762899Z + container_name=dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T07:08:47.5764096Z + grep download.pytorch.org /etc/hosts 2024-06-26T07:08:47.5765576Z + docker exec -i dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 sudo bash -c '/bin/cat >> /etc/hosts' 2024-06-26T07:08:47.6387180Z + echo DOCKER_CONTAINER_ID=dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T07:08:47.6389796Z ++ echo dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl 2024-06-26T07:08:47.6391737Z + docker exec -t dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 sh -c 'pip install dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2024-06-26T07:08:48.0137184Z Processing ./dist/torch-2.5.0a0+gitb8c4c54-cp310-cp310-linux_x86_64.whl (from torch==2.5.0a0+gitb8c4c54) 2024-06-26T07:08:48.3260097Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.13.1) 2024-06-26T07:08:48.3265171Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (4.12.2) 2024-06-26T07:08:48.3267484Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.12.1) 2024-06-26T07:08:48.3270666Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.8.8) 2024-06-26T07:08:48.3274146Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.1.4) 2024-06-26T07:08:48.3277217Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2024.2.0) 2024-06-26T07:08:48.3302949Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (3.3.0) 2024-06-26T07:08:48.3360639Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.21.2) 2024-06-26T07:08:48.3791074Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (2.1.5) 2024-06-26T07:08:48.3946084Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch==2.5.0a0+gitb8c4c54->torch==2.5.0a0+gitb8c4c54) (1.3.0) 2024-06-26T07:08:49.6982610Z Installing collected packages: torch 2024-06-26T07:08:58.3636061Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2024-06-26T07:08:58.3637647Z timm 0.9.7 requires torchvision, which is not installed. 2024-06-26T07:08:58.3638325Z Successfully installed torch-2.5.0a0+gitb8c4c54 2024-06-26T07:08:58.4427694Z ++ dirname .ci/pytorch/test.sh 2024-06-26T07:08:58.4432605Z + source .ci/pytorch/common.sh 2024-06-26T07:08:58.4435615Z +++ dirname .ci/pytorch/common.sh 2024-06-26T07:08:58.4441696Z ++ source .ci/pytorch/common_utils.sh 2024-06-26T07:08:58.4443559Z +++ declare -f -t trap_add 2024-06-26T07:08:58.4448444Z ++ set -ex 2024-06-26T07:08:58.4449062Z ++ [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:08:58.4449646Z ++ BUILD_TEST_LIBTORCH=0 2024-06-26T07:08:58.4450284Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *rocm* ]] 2024-06-26T07:08:58.4451748Z ++ stat -c %u /var/lib/jenkins/workspace 2024-06-26T07:08:58.4459234Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2024-06-26T07:08:58.4459804Z + trap_add cleanup_workspace EXIT 2024-06-26T07:08:58.4460298Z + trap_add_cmd=cleanup_workspace 2024-06-26T07:08:58.4460666Z + shift 2024-06-26T07:08:58.4460948Z + for trap_add_name in "$@" 2024-06-26T07:08:58.4464231Z +++ trap -p EXIT 2024-06-26T07:08:58.4465690Z ++ eval 'extract_trap_cmd ' 2024-06-26T07:08:58.4466157Z +++ extract_trap_cmd 2024-06-26T07:08:58.4466539Z +++ printf '%s\n' '' 2024-06-26T07:08:58.4466971Z ++ printf '%s\n' cleanup_workspace 2024-06-26T07:08:58.4467505Z + trap -- ' 2024-06-26T07:08:58.4467879Z cleanup_workspace' EXIT 2024-06-26T07:08:58.4468657Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2024-06-26T07:08:58.8567574Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2024-06-26T07:08:58.8577935Z + echo 'Environment variables:' 2024-06-26T07:08:58.8578422Z Environment variables: 2024-06-26T07:08:58.8578769Z + env 2024-06-26T07:08:58.8584669Z INSTALLED_DB=yes 2024-06-26T07:08:58.8585340Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:08:58.8585830Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:08:58.8586257Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.8586986Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:08:58.8587646Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:08:58.8588385Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8588901Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8589523Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:58.8590152Z HOSTNAME=dd8cd24a36d5 2024-06-26T07:08:58.8591318Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8592401Z GITHUB_ACTION=__self 2024-06-26T07:08:58.8592877Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:08:58.8597601Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:08:58.8601480Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:08:58.8602043Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:08:58.8602407Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:08:58.8602779Z TEST_CONFIG=dynamic_inductor_timm 2024-06-26T07:08:58.8603198Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:08:58.8603666Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:08:58.8604133Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.8604581Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8605014Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8605452Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8606109Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:08:58.8606625Z GITHUB_REF_TYPE=tag 2024-06-26T07:08:58.8606947Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:08:58.8607336Z NCCL_VERSION=2.17.1-1 2024-06-26T07:08:58.8607723Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8608365Z XLA_CUDA= 2024-06-26T07:08:58.8608875Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:08:58.8610529Z *** 2024-06-26T07:08:58.8610851Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:08:58.8611265Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:08:58.8611634Z GITHUB_ACTIONS=true 2024-06-26T07:08:58.8611970Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:58.8612490Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:08:58.8613025Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:08:58.8613485Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8613988Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:08:58.8614472Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8615260Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8615970Z UCC_HOME=/usr 2024-06-26T07:08:58.8616308Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:08:58.8616712Z VERBOSE_TEST_LOGS=False 2024-06-26T07:08:58.8617071Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:08:58.8617523Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:08:58.8618039Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8618503Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:08:58.8618885Z SHARD_NUMBER=1 2024-06-26T07:08:58.8619189Z GITHUB_REF_PROTECTED=false 2024-06-26T07:08:58.8619540Z HOME=/var/lib/jenkins 2024-06-26T07:08:58.8619909Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:08:58.8620344Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:08:58.8620819Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:08:58.8621304Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:08:58.8621675Z CUDA_VERSION=12.1.1 2024-06-26T07:08:58.8622086Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:08:58.8622526Z NUM_TEST_SHARDS=2 2024-06-26T07:08:58.8622828Z UCX_HOME=/usr 2024-06-26T07:08:58.8623335Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:08:58.8624395Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8625629Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:58.8626830Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8627934Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:08:58.8628981Z GITHUB_EVENT_NAME=push 2024-06-26T07:08:58.8629326Z DASHBOARD_TAG= 2024-06-26T07:08:58.8629638Z GITHUB_RUN_ID=9673646808 2024-06-26T07:08:58.8630108Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:08:58.8630640Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:08:58.8631630Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8632541Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8632957Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8633342Z PR_NUMBER= 2024-06-26T07:08:58.8633636Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:08:58.8633983Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:08:58.8634422Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:08:58.8634888Z TERM=xterm 2024-06-26T07:08:58.8635228Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8635649Z INSTALLED_VISION=yes 2024-06-26T07:08:58.8635963Z BRANCH= 2024-06-26T07:08:58.8636260Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:08:58.8636668Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:08:58.8637098Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:08:58.8637836Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:08:58.8638594Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:08:58.8639086Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:08:58.8639552Z REENABLED_ISSUES= 2024-06-26T07:08:58.8639861Z SHLVL=1 2024-06-26T07:08:58.8640255Z MAX_JOBS=14 2024-06-26T07:08:58.8640645Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:08:58.8640999Z NVARCH=x86_64 2024-06-26T07:08:58.8641305Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:08:58.8641764Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8642302Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:08:58.8642785Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:08:58.8643447Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:58.8644021Z GITHUB_JOB=test 2024-06-26T07:08:58.8644412Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8644974Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:08:58.8645494Z NO_TEST_TIMEOUT=False 2024-06-26T07:08:58.8645977Z TD_DISTRIBUTED=False 2024-06-26T07:08:58.8646350Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:08:58.8646791Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:08:58.8647211Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:08:58.8647586Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:08:58.8647934Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:08:58.8648294Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:08:58.8649328Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:08:58.8650393Z GITHUB_BASE_REF= 2024-06-26T07:08:58.8650716Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:08:58.8651089Z CI=true 2024-06-26T07:08:58.8651408Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8651827Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:08:58.8652203Z JOB_ID=26688911727 2024-06-26T07:08:58.8652520Z INSTALLED_PROTOBUF=yes 2024-06-26T07:08:58.8652852Z GITHUB_HEAD_REF= 2024-06-26T07:08:58.8653154Z GITHUB_ACTION_REF= 2024-06-26T07:08:58.8653582Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:08:58.8654051Z GITHUB_WORKFLOW=inductor 2024-06-26T07:08:58.8654412Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:08:58.8655299Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8656128Z NO_TD=False 2024-06-26T07:08:58.8656438Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:08:58.8656805Z _=/usr/bin/env 2024-06-26T07:08:58.8657272Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2024-06-26T07:08:58.8729361Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2024-06-26T07:08:58.8730516Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2024-06-26T07:08:58.8731527Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2024-06-26T07:08:58.8732509Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2024-06-26T07:08:58.8733118Z + BUILD_DIR=build 2024-06-26T07:08:58.8733455Z + BUILD_RENAMED_DIR=build_renamed 2024-06-26T07:08:58.8733844Z + BUILD_BIN_DIR=build/bin 2024-06-26T07:08:58.8734175Z + SHARD_NUMBER=1 2024-06-26T07:08:58.8734492Z + NUM_TEST_SHARDS=2 2024-06-26T07:08:58.8734891Z + export VALGRIND=ON 2024-06-26T07:08:58.8735264Z + VALGRIND=ON 2024-06-26T07:08:58.8735749Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *clang9* ]] 2024-06-26T07:08:58.8736255Z + [[ 0 == \1 ]] 2024-06-26T07:08:58.8736556Z + [[ False == \1 ]] 2024-06-26T07:08:58.8737024Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *bazel* ]] 2024-06-26T07:08:58.8737547Z ++ realpath build/custom_test_artifacts 2024-06-26T07:08:58.8741995Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2024-06-26T07:08:58.8742720Z + [[ -n '' ]] 2024-06-26T07:08:58.8743066Z + echo 'Environment variables' 2024-06-26T07:08:58.8743453Z Environment variables 2024-06-26T07:08:58.8743775Z + env 2024-06-26T07:08:58.8747538Z INSTALLED_DB=yes 2024-06-26T07:08:58.8748055Z NV_LIBCUBLAS_VERSION=12.1.3.1-1 2024-06-26T07:08:58.8748924Z NVIDIA_VISIBLE_DEVICES=all 2024-06-26T07:08:58.8749614Z NV_NVML_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.8750682Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T07:08:58.8751550Z CONTINUE_THROUGH_ERROR=False 2024-06-26T07:08:58.8752060Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8752730Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8753475Z BUILD_ENVIRONMENT=linux-focal-cuda12.1-py3.10-gcc9-sm86 2024-06-26T07:08:58.8754049Z HOSTNAME=dd8cd24a36d5 2024-06-26T07:08:58.8755156Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8756249Z GITHUB_ACTION=__self 2024-06-26T07:08:58.8756745Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2024-06-26T07:08:58.8760688Z NVIDIA_REQUIRE_CUDA=cuda>=12.1 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=525,driver<526 brand=unknown,driver>=525,driver<526 brand=nvidia,driver>=525,driver<526 brand=nvidiartx,driver>=525,driver<526 brand=geforce,driver>=525,driver<526 brand=geforcertx,driver>=525,driver<526 brand=quadro,driver>=525,driver<526 brand=quadrortx,driver>=525,driver<526 brand=titan,driver>=525,driver<526 brand=titanrtx,driver>=525,driver<526 2024-06-26T07:08:58.8764512Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-1=12.1.3.1-1 2024-06-26T07:08:58.8765019Z NV_NVTX_VERSION=12.1.105-1 2024-06-26T07:08:58.8765377Z GITHUB_RUN_NUMBER=76977 2024-06-26T07:08:58.8765926Z TEST_CONFIG=dynamic_inductor_timm 2024-06-26T07:08:58.8766366Z GITHUB_REPOSITORY_OWNER_ID=21003710 2024-06-26T07:08:58.8766856Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2024-06-26T07:08:58.8767310Z NV_CUDA_CUDART_DEV_VERSION=12.1.105-1 2024-06-26T07:08:58.8767750Z NV_LIBCUSPARSE_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8768180Z NV_LIBNPP_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8768621Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8769116Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache 2024-06-26T07:08:58.8769580Z GITHUB_REF_TYPE=tag 2024-06-26T07:08:58.8769910Z TORCH_CUDA_ARCH_LIST=Maxwell 2024-06-26T07:08:58.8770287Z NCCL_VERSION=2.17.1-1 2024-06-26T07:08:58.8770670Z BASE_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8771108Z XLA_CUDA= 2024-06-26T07:08:58.8771585Z HUGGING_FACE_HUB_TOKEN=*** 2024-06-26T07:08:58.8771979Z *** 2024-06-26T07:08:58.8772253Z CARGO_NET_GIT_FETCH_WITH_CLI=true 2024-06-26T07:08:58.8772660Z GITHUB_REPOSITORY_ID=65600975 2024-06-26T07:08:58.8773021Z GITHUB_ACTIONS=true 2024-06-26T07:08:58.8773353Z NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T07:08:58.8773825Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-1=12.1.105-1 2024-06-26T07:08:58.8774356Z NV_LIBNPP_PACKAGE=libnpp-12-1=12.1.0.40-1 2024-06-26T07:08:58.8774824Z SHA1=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8775327Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev 2024-06-26T07:08:58.8775806Z GITHUB_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8776587Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8777296Z UCC_HOME=/usr 2024-06-26T07:08:58.8777622Z NV_LIBCUBLAS_DEV_VERSION=12.1.3.1-1 2024-06-26T07:08:58.8778021Z VERBOSE_TEST_LOGS=False 2024-06-26T07:08:58.8778372Z NVIDIA_PRODUCT_NAME=CUDA 2024-06-26T07:08:58.8778829Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-1 2024-06-26T07:08:58.8779318Z GITHUB_REF=refs/tags/ciflow/inductor/129470 2024-06-26T07:08:58.8779782Z NV_CUDA_CUDART_VERSION=12.1.105-1 2024-06-26T07:08:58.8780160Z SHARD_NUMBER=1 2024-06-26T07:08:58.8780469Z GITHUB_REF_PROTECTED=false 2024-06-26T07:08:58.8780810Z HOME=/var/lib/jenkins 2024-06-26T07:08:58.8781177Z GITHUB_API_URL=https://api.github.com 2024-06-26T07:08:58.8781621Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2024-06-26T07:08:58.8782198Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb 2024-06-26T07:08:58.8782692Z SCCACHE_S3_KEY_PREFIX=inductor 2024-06-26T07:08:58.8783131Z CUDA_VERSION=12.1.1 2024-06-26T07:08:58.8783546Z NV_LIBCUBLAS_PACKAGE=libcublas-12-1=12.1.3.1-1 2024-06-26T07:08:58.8783977Z NUM_TEST_SHARDS=2 2024-06-26T07:08:58.8784280Z UCX_HOME=/usr 2024-06-26T07:08:58.8784794Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-1=12.1.1-1 2024-06-26T07:08:58.8785860Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8787070Z JOB_NAME=cuda12.1-py3.10-gcc9-sm86 / test (dynamic_inductor_timm, 1, 2, linux.g5.4xlarge.nvidia.gpu) 2024-06-26T07:08:58.8788524Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8789637Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2024-06-26T07:08:58.8790283Z GITHUB_EVENT_NAME=push 2024-06-26T07:08:58.8790614Z DASHBOARD_TAG= 2024-06-26T07:08:58.8790920Z GITHUB_RUN_ID=9673646808 2024-06-26T07:08:58.8791374Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-1=12.1.0.40-1 2024-06-26T07:08:58.8791907Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-1 2024-06-26T07:08:58.8792882Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8793781Z GITHUB_ACTOR=pytorch-bot[bot] 2024-06-26T07:08:58.8794186Z NV_LIBNPP_DEV_VERSION=12.1.0.40-1 2024-06-26T07:08:58.8794561Z PR_NUMBER= 2024-06-26T07:08:58.8794839Z GITHUB_RUN_ATTEMPT=1 2024-06-26T07:08:58.8795161Z VALGRIND=ON 2024-06-26T07:08:58.8795457Z ANACONDA_PYTHON_VERSION=3.10 2024-06-26T07:08:58.8795897Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2024-06-26T07:08:58.8796346Z TERM=xterm 2024-06-26T07:08:58.8796677Z NV_LIBCUSPARSE_DEV_VERSION=12.1.0.106-1 2024-06-26T07:08:58.8797088Z INSTALLED_VISION=yes 2024-06-26T07:08:58.8797392Z BRANCH= 2024-06-26T07:08:58.8797680Z OPENSSL_ROOT_DIR=/opt/openssl 2024-06-26T07:08:58.8798082Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs 2024-06-26T07:08:58.8798514Z CUDA_PATH=/usr/local/cuda 2024-06-26T07:08:58.8799244Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2024-06-26T07:08:58.8799994Z GITHUB_SERVER_URL=https://github.com 2024-06-26T07:08:58.8800485Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b 2024-06-26T07:08:58.8800956Z REENABLED_ISSUES= 2024-06-26T07:08:58.8801248Z SHLVL=1 2024-06-26T07:08:58.8801513Z MAX_JOBS=14 2024-06-26T07:08:58.8801827Z NV_CUDA_LIB_VERSION=12.1.1-1 2024-06-26T07:08:58.8802219Z NVARCH=x86_64 2024-06-26T07:08:58.8802516Z GITHUB_ACTOR_ID=54816060 2024-06-26T07:08:58.8802963Z GITHUB_WORKFLOW_SHA=b8c4c54d347aa776934c60784e35936878ef18dc 2024-06-26T07:08:58.8803492Z GITHUB_REF_NAME=ciflow/inductor/129470 2024-06-26T07:08:58.8803966Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-12-1 2024-06-26T07:08:58.8804617Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2024-06-26T07:08:58.8805190Z GITHUB_JOB=test 2024-06-26T07:08:58.8805598Z NV_LIBNCCL_PACKAGE=libnccl2=2.17.1-1+cuda12.1 2024-06-26T07:08:58.8806301Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64 2024-06-26T07:08:58.8806816Z NO_TEST_TIMEOUT=False 2024-06-26T07:08:58.8807152Z TD_DISTRIBUTED=False 2024-06-26T07:08:58.8807572Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.1.1-1 2024-06-26T07:08:58.8808010Z GITHUB_REPOSITORY=pytorch/pytorch 2024-06-26T07:08:58.8808431Z NV_NVPROF_VERSION=12.1.105-1 2024-06-26T07:08:58.8808800Z GITHUB_RETENTION_DAYS=90 2024-06-26T07:08:58.8809145Z OPENSSL_DIR=/opt/openssl 2024-06-26T07:08:58.8809498Z GITHUB_ACTION_REPOSITORY= 2024-06-26T07:08:58.8810535Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:08:58.8811594Z GITHUB_BASE_REF= 2024-06-26T07:08:58.8811916Z NV_LIBNCCL_PACKAGE_NAME=libnccl2 2024-06-26T07:08:58.8812288Z CI=true 2024-06-26T07:08:58.8812725Z NV_LIBNCCL_PACKAGE_VERSION=2.17.1-1 2024-06-26T07:08:58.8813209Z GITHUB_REPOSITORY_OWNER=pytorch 2024-06-26T07:08:58.8813581Z JOB_ID=26688911727 2024-06-26T07:08:58.8826494Z INSTALLED_PROTOBUF=yes 2024-06-26T07:08:58.8826849Z GITHUB_HEAD_REF= 2024-06-26T07:08:58.8827157Z GITHUB_ACTION_REF= 2024-06-26T07:08:58.8827614Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2024-06-26T07:08:58.8828356Z GITHUB_WORKFLOW=inductor 2024-06-26T07:08:58.8828757Z DEBIAN_FRONTEND=noninteractive 2024-06-26T07:08:58.8829657Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_ca90832f-a0dc-4f0c-bd93-f74cfffdc152 2024-06-26T07:08:58.8830473Z NO_TD=False 2024-06-26T07:08:58.8830791Z SKIP_SCCACHE_INITIALIZATION=1 2024-06-26T07:08:58.8831161Z _=/usr/bin/env 2024-06-26T07:08:58.8831506Z + echo 'Testing pytorch' 2024-06-26T07:08:58.8831857Z Testing pytorch 2024-06-26T07:08:58.8832188Z + export LANG=C.UTF-8 2024-06-26T07:08:58.8832530Z + LANG=C.UTF-8 2024-06-26T07:08:58.8832821Z + PR_NUMBER= 2024-06-26T07:08:58.8833187Z + [[ dynamic_inductor_timm == \d\e\f\a\u\l\t ]] 2024-06-26T07:08:58.8833728Z + [[ dynamic_inductor_timm == \d\i\s\t\r\i\b\u\t\e\d ]] 2024-06-26T07:08:58.8834244Z + [[ dynamic_inductor_timm == \s\l\o\w ]] 2024-06-26T07:08:58.8834876Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *slow-gradcheck* ]] 2024-06-26T07:08:58.8835569Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *cuda* ]] 2024-06-26T07:08:58.8836123Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:08:58.8836596Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2024-06-26T07:08:58.8837052Z + [[ dynamic_inductor_timm == *crossref* ]] 2024-06-26T07:08:58.8837633Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *rocm* ]] 2024-06-26T07:08:58.8838265Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *xpu* ]] 2024-06-26T07:08:58.8838901Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:08:58.8839465Z + pip_install --user ninja==1.10.2 2024-06-26T07:08:58.8840004Z + pip install --progress-bar off --user ninja==1.10.2 2024-06-26T07:08:59.2381322Z Collecting ninja==1.10.2 2024-06-26T07:08:59.2534820Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2024-06-26T07:08:59.2666788Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2024-06-26T07:09:00.5716210Z Installing collected packages: ninja 2024-06-26T07:09:00.5780961Z  WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH. 2024-06-26T07:09:00.5782200Z Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location. 2024-06-26T07:09:00.5821689Z Successfully installed ninja-1.10.2 2024-06-26T07:09:00.6684911Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:00.6687288Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:00.6688737Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:00.6689241Z + install_tlparse 2024-06-26T07:09:00.6689612Z + pip_install --user tlparse==0.3.7 2024-06-26T07:09:00.6690153Z + pip install --progress-bar off --user tlparse==0.3.7 2024-06-26T07:09:01.0194553Z Collecting tlparse==0.3.7 2024-06-26T07:09:01.0352021Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (346 bytes) 2024-06-26T07:09:01.0442472Z Downloading tlparse-0.3.7-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.2 MB) 2024-06-26T07:09:02.3473314Z Installing collected packages: tlparse 2024-06-26T07:09:02.3833668Z Successfully installed tlparse-0.3.7 2024-06-26T07:09:02.4647583Z ++ python -m site --user-base 2024-06-26T07:09:02.4810415Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2024-06-26T07:09:02.4812184Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *asan* ]] 2024-06-26T07:09:02.4812827Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-debug* ]] 2024-06-26T07:09:02.4813561Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:02.4814565Z + echo 'We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass' 2024-06-26T07:09:02.4815696Z We are not in debug mode: linux-focal-cuda12.1-py3.10-gcc9-sm86. Expect the assertion to pass 2024-06-26T07:09:02.4816388Z + cd test 2024-06-26T07:09:02.4816908Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2024-06-26T07:09:03.9388607Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2024-06-26T07:09:03.9389448Z + [[ dynamic_inductor_timm == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2024-06-26T07:09:03.9390322Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 != *-bazel-* ]] 2024-06-26T07:09:03.9391029Z + pushd test 2024-06-26T07:09:03.9391423Z ~/workspace/test ~/workspace 2024-06-26T07:09:03.9391959Z ++ python -c 'import torch; print(torch.version.cuda)' 2024-06-26T07:09:05.3991292Z + CUDA_VERSION=12.1 2024-06-26T07:09:05.3991895Z + '[' 12.1 == 12.4 ']' 2024-06-26T07:09:05.3992237Z + ISCUDA124= 2024-06-26T07:09:05.3992526Z + popd 2024-06-26T07:09:05.3992799Z ~/workspace 2024-06-26T07:09:05.3995931Z + DYNAMO_BENCHMARK_FLAGS=() 2024-06-26T07:09:05.3996570Z + [[ dynamic_inductor_timm == *dynamo_eager* ]] 2024-06-26T07:09:05.3997191Z + [[ dynamic_inductor_timm == *aot_eager* ]] 2024-06-26T07:09:05.3997767Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-06-26T07:09:05.3998277Z + [[ dynamic_inductor_timm == *inductor* ]] 2024-06-26T07:09:05.3998751Z + [[ dynamic_inductor_timm != *perf* ]] 2024-06-26T07:09:05.3999399Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2024-06-26T07:09:05.4000021Z + [[ dynamic_inductor_timm == *dynamic* ]] 2024-06-26T07:09:05.4000667Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2024-06-26T07:09:05.4001267Z + [[ dynamic_inductor_timm == *cpu_inductor* ]] 2024-06-26T07:09:05.4001787Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2024-06-26T07:09:05.4025999Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:05.4026807Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *-bazel-* ]] 2024-06-26T07:09:05.4027302Z + cd test 2024-06-26T07:09:05.4027748Z + python -c 'import torch; print(torch.__config__.show())' 2024-06-26T07:09:06.7501884Z PyTorch built with: 2024-06-26T07:09:06.7502620Z - GCC 9.4 2024-06-26T07:09:06.7503063Z - C++ Version: 201703 2024-06-26T07:09:06.7503909Z - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:06.7504973Z - Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:06.7505652Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:06.7506164Z - LAPACK is enabled (usually provided by MKL) 2024-06-26T07:09:06.7506649Z - NNPACK is enabled 2024-06-26T07:09:06.7507027Z - CPU capability usage: AVX2 2024-06-26T07:09:06.7507431Z - CUDA Runtime 12.1 2024-06-26T07:09:06.7507941Z - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86 2024-06-26T07:09:06.7509033Z - CuDNN 90.1 (built against CUDA 12.4) 2024-06-26T07:09:06.7509481Z - Magma 2.6.1 2024-06-26T07:09:06.7516684Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=9.1.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=1 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_VERSION=2.5.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 2024-06-26T07:09:06.7523292Z 2024-06-26T07:09:07.0217076Z + cd test 2024-06-26T07:09:07.0218553Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2024-06-26T07:09:08.2244596Z ATen/Parallel: 2024-06-26T07:09:08.2245118Z at::get_num_threads() : 8 2024-06-26T07:09:08.2245787Z at::get_num_interop_threads() : 16 2024-06-26T07:09:08.2246251Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2024-06-26T07:09:08.2246655Z omp_get_max_threads() : 8 2024-06-26T07:09:08.2247610Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications 2024-06-26T07:09:08.2248404Z mkl_get_max_threads() : 8 2024-06-26T07:09:08.2249021Z Intel(R) MKL-DNN v3.4.2 (Git Hash 1137e04ec0b5251ca2b4400a4fd3c667ce843d67) 2024-06-26T07:09:08.2249701Z std::thread::hardware_concurrency() : 16 2024-06-26T07:09:08.2250123Z Environment variables: 2024-06-26T07:09:08.2250474Z OMP_NUM_THREADS : [not set] 2024-06-26T07:09:08.2250849Z MKL_NUM_THREADS : [not set] 2024-06-26T07:09:08.2251230Z ATen parallel backend: OpenMP 2024-06-26T07:09:08.2251483Z 2024-06-26T07:09:08.4670664Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *aarch64* ]] 2024-06-26T07:09:08.4671465Z + [[ dynamic_inductor_timm == *backward* ]] 2024-06-26T07:09:08.4672022Z + [[ dynamic_inductor_timm == *xla* ]] 2024-06-26T07:09:08.4672520Z + [[ dynamic_inductor_timm == *executorch* ]] 2024-06-26T07:09:08.4673194Z + [[ dynamic_inductor_timm == \j\i\t\_\l\e\g\a\c\y ]] 2024-06-26T07:09:08.4673923Z + [[ linux-focal-cuda12.1-py3.10-gcc9-sm86 == *libtorch* ]] 2024-06-26T07:09:08.4674527Z + [[ dynamic_inductor_timm == distributed ]] 2024-06-26T07:09:08.4675038Z + [[ dynamic_inductor_timm == *inductor_distributed* ]] 2024-06-26T07:09:08.4675613Z + [[ dynamic_inductor_timm == *inductor-halide* ]] 2024-06-26T07:09:08.4676216Z + [[ dynamic_inductor_timm == *inductor-micro-benchmark* ]] 2024-06-26T07:09:08.4676756Z + [[ dynamic_inductor_timm == *huggingface* ]] 2024-06-26T07:09:08.4677213Z + [[ dynamic_inductor_timm == *timm* ]] 2024-06-26T07:09:08.4677623Z + install_torchvision 2024-06-26T07:09:08.4677949Z + local orig_preload 2024-06-26T07:09:08.4678269Z + local commit 2024-06-26T07:09:08.4678576Z ++ get_pinned_commit vision 2024-06-26T07:09:08.4678950Z ++ cat .github/ci_commit_pins/vision.txt 2024-06-26T07:09:08.4685242Z + commit=d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.4685995Z + orig_preload= 2024-06-26T07:09:08.4686379Z + '[' -n '' ']' 2024-06-26T07:09:08.4687193Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.4688597Z + pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.7787047Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:08.7792091Z Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-7pkms8g_ 2024-06-26T07:09:08.7810596Z Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-7pkms8g_ 2024-06-26T07:09:10.1592437Z Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f' 2024-06-26T07:09:10.1608383Z Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:11.3674887Z Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:11.8996051Z Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f 2024-06-26T07:09:14.1778401Z Preparing metadata (setup.py) ... [?25l- \ done 2024-06-26T07:09:14.1826526Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.21.2) 2024-06-26T07:09:14.1829547Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.5.0a0+gitb8c4c54) 2024-06-26T07:09:14.1834905Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (10.3.0) 2024-06-26T07:09:14.2039329Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.13.1) 2024-06-26T07:09:14.2046045Z Requirement already satisfied: typing-extensions>=4.8.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.12.2) 2024-06-26T07:09:14.2048030Z Requirement already satisfied: sympy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.12.1) 2024-06-26T07:09:14.2051208Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8) 2024-06-26T07:09:14.2053793Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.4) 2024-06-26T07:09:14.2056691Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.2.0) 2024-06-26T07:09:14.2477269Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (2.1.5) 2024-06-26T07:09:14.2627811Z Requirement already satisfied: mpmath<1.4.0,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0) 2024-06-26T07:09:14.2699118Z Building wheels for collected packages: torchvision 2024-06-26T07:10:28.7351462Z Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ done 2024-06-26T07:10:28.7378147Z [?25h Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2112962 sha256=9b503063c7a6fab74265e412f91fd09c2b16a910129cb1a82fe8ede7f3f276dd 2024-06-26T07:10:28.7379796Z Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96 2024-06-26T07:10:28.7413501Z Successfully built torchvision 2024-06-26T07:10:29.9514627Z Installing collected packages: torchvision 2024-06-26T07:10:30.3312496Z Successfully installed torchvision-0.19.0a0+d23a6e1 2024-06-26T07:10:30.4389591Z + '[' -n '' ']' 2024-06-26T07:10:30.4390208Z + id=0 2024-06-26T07:10:30.4390810Z + test_dynamo_benchmark timm_models 0 2024-06-26T07:10:30.4391600Z ++ pwd 2024-06-26T07:10:30.4393332Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:30.4395049Z + local suite=timm_models 2024-06-26T07:10:30.4395637Z + shift 2024-06-26T07:10:30.4395982Z + local shard_id=0 2024-06-26T07:10:30.4396297Z + shift 2024-06-26T07:10:30.4396657Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-06-26T07:10:30.4398507Z + [[ dynamic_inductor_timm == *perf* ]] 2024-06-26T07:10:30.4399038Z + [[ dynamic_inductor_timm == *cpu_inductor* ]] 2024-06-26T07:10:30.4399909Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-06-26T07:10:30.4400824Z + test_single_dynamo_benchmark inference timm_models 0 --inference --bfloat16 2024-06-26T07:10:30.4401626Z ++ pwd 2024-06-26T07:10:30.4402102Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:30.4402845Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2024-06-26T07:10:30.4410217Z + local name=inference 2024-06-26T07:10:30.4410568Z + shift 2024-06-26T07:10:30.4410862Z + local suite=timm_models 2024-06-26T07:10:30.4411210Z + shift 2024-06-26T07:10:30.4411471Z + local shard_id=0 2024-06-26T07:10:30.4411773Z + shift 2024-06-26T07:10:30.4412053Z + partition_flags=() 2024-06-26T07:10:30.4412378Z + local partition_flags 2024-06-26T07:10:30.4412750Z + [[ -n 2 ]] 2024-06-26T07:10:30.4413073Z + [[ -n 0 ]] 2024-06-26T07:10:30.4413653Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2024-06-26T07:10:30.4414389Z + [[ dynamic_inductor_timm == *perf_compare* ]] 2024-06-26T07:10:30.4414880Z + [[ dynamic_inductor_timm == *perf* ]] 2024-06-26T07:10:30.4415348Z + [[ dynamic_inductor_timm == *aot_inductor* ]] 2024-06-26T07:10:30.4417140Z + python benchmarks/dynamo/timm_models.py --ci --accuracy --timing --explain --inductor --dynamic-shapes --dynamic-batch-only --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_timm_models.csv 2024-06-26T07:10:36.2024286Z 2024-06-26T07:10:36.5180108Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:10:36.5180470Z 2024-06-26T07:10:36.6542864Z model.safetensors: 0% 0.00/95.5M [00:00 will be ignored 2024-06-26T07:30:59.3655508Z E0626 07:30:59.364000 140560165134976 torch/_dynamo/utils.py:1470] RMSE (res-fp64): 0.00936, (ref-fp64): 0.00257 and shape=torch.Size([768]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.010000 2024-06-26T07:30:59.3658309Z E0626 07:30:59.364000 140560165134976 torch/_dynamo/utils.py:1384] Accuracy failed for key name blocks.0.attn.proj.bias.grad 2024-06-26T07:30:59.3675982Z fail_accuracy 2024-06-26T07:30:59.4172865Z TIMING: entire_frame_compile:102.7176 code_gen:26.16785 inductor_compile:56.66561 backend_compile:88.54291 2024-06-26T07:30:59.4175103Z STATS: call_* op count: 1054 | FakeTensor.__torch_dispatch__:16297 | FakeTensorMode.__torch_dispatch__:108805 | attempt fast:2461 | fast is_contiguous:2461 | ProxyTorchDispatchMode.__torch_dispatch__:21953 2024-06-26T07:30:59.4176444Z Dynamo produced 3 graphs covering 1054 ops with 7 graph breaks (5 unique) 2024-06-26T07:31:07.7070994Z 2024-06-26T07:31:08.5098110Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:08.5098607Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:31:08.5099058Z cuda train botnet26t_256 2024-06-26T07:32:17.3558091Z pass 2024-06-26T07:32:17.3661033Z TIMING: entire_frame_compile:63.2756 code_gen:19.55583 inductor_compile:38.07238 backend_compile:56.67397 2024-06-26T07:32:17.3662966Z STATS: call_* op count: 374 | FakeTensor.__torch_dispatch__:5785 | FakeTensorMode.__torch_dispatch__:42903 | attempt fast:1230 | fast is_contiguous:1204 | slow no contiguity match:26 | ProxyTorchDispatchMode.__torch_dispatch__:9029 2024-06-26T07:32:17.3664748Z Dynamo produced 2 graphs covering 374 ops with 6 graph breaks (5 unique) 2024-06-26T07:32:23.4549421Z 2024-06-26T07:32:27.1731750Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:27.1732231Z loading model: 0it [00:03, ?it/s] 2024-06-26T07:32:27.1732682Z cuda train cait_m36_384 2024-06-26T07:32:27.1737739Z Traceback (most recent call last): 2024-06-26T07:32:27.1738512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2321, in validate_model 2024-06-26T07:32:27.1739282Z self.model_iter_fn(model, example_inputs) 2024-06-26T07:32:27.1740120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in forward_and_backward_pass 2024-06-26T07:32:27.1740931Z pred = mod(*cloned_inputs) 2024-06-26T07:32:27.1742106Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:27.1743341Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:27.1744526Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:27.1745527Z return forward_call(*args, **kwargs) 2024-06-26T07:32:27.1746508Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 361, in forward 2024-06-26T07:32:27.1747261Z x = self.forward_features(x) 2024-06-26T07:32:27.1748410Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 346, in forward_features 2024-06-26T07:32:27.1749222Z x = self.blocks(x) 2024-06-26T07:32:27.1750111Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:27.1751003Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:27.1752237Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:27.1753169Z return forward_call(*args, **kwargs) 2024-06-26T07:32:27.1754078Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/container.py", line 249, in forward 2024-06-26T07:32:27.1754889Z input = module(input) 2024-06-26T07:32:27.1755785Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:27.1756670Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:27.1757587Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:27.1762611Z return forward_call(*args, **kwargs) 2024-06-26T07:32:27.1763850Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 197, in forward 2024-06-26T07:32:27.1764787Z x = x + self.drop_path(self.gamma_1 * self.attn(self.norm1(x))) 2024-06-26T07:32:27.1765993Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1657, in _wrapped_call_impl 2024-06-26T07:32:27.1766902Z return self._call_impl(*args, **kwargs) 2024-06-26T07:32:27.1767826Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T07:32:27.1768657Z return forward_call(*args, **kwargs) 2024-06-26T07:32:27.1769478Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/cait.py", line 151, in forward 2024-06-26T07:32:27.1770278Z x = (attn @ v).transpose(1, 2).reshape(B, N, C) 2024-06-26T07:32:27.1773471Z torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacity of 21.98 GiB of which 16.44 MiB is free. Process 16982 has 21.95 GiB memory in use. Of the allocated memory 21.21 GiB is allocated by PyTorch, and 430.66 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables) 2024-06-26T07:32:27.1776198Z 2024-06-26T07:32:27.1776492Z The above exception was the direct cause of the following exception: 2024-06-26T07:32:27.1776948Z 2024-06-26T07:32:27.1777093Z Traceback (most recent call last): 2024-06-26T07:32:27.1777756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 4229, in run 2024-06-26T07:32:27.1778411Z ) = runner.load_model( 2024-06-26T07:32:27.1779064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 303, in load_model 2024-06-26T07:32:27.1779819Z self.validate_model(model, example_inputs) 2024-06-26T07:32:27.1780567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 2323, in validate_model 2024-06-26T07:32:27.1781320Z raise RuntimeError("Eager run failed") from e 2024-06-26T07:32:27.1781790Z RuntimeError: Eager run failed 2024-06-26T07:32:27.1782049Z 2024-06-26T07:32:27.1782169Z eager_fail_to_run 2024-06-26T07:32:30.6997803Z 2024-06-26T07:32:31.4122399Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:31.4122865Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:32:31.4123311Z cuda train coat_lite_mini 2024-06-26T07:34:19.4369985Z pass 2024-06-26T07:34:19.4483023Z TIMING: entire_frame_compile:101.95291 code_gen:27.23048 inductor_compile:58.86668 backend_compile:90.52742 2024-06-26T07:34:19.4484640Z STATS: call_* op count: 538 | FakeTensor.__torch_dispatch__:6887 | FakeTensorMode.__torch_dispatch__:65573 | attempt fast:2208 | fast is_contiguous:2027 | slow no contiguity match:181 | ProxyTorchDispatchMode.__torch_dispatch__:14338 2024-06-26T07:34:19.4486171Z Dynamo produced 2 graphs covering 538 ops with 6 graph breaks (5 unique) 2024-06-26T07:34:27.8015339Z 2024-06-26T07:34:29.2838973Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:34:29.2839871Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:34:29.2840324Z cuda train convit_base 2024-06-26T07:34:29.3153792Z WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine 2024-06-26T07:37:38.3548002Z pass 2024-06-26T07:37:38.3771535Z TIMING: entire_frame_compile:183.73954 code_gen:33.09982 inductor_compile:91.38409 backend_compile:164.15807 2024-06-26T07:37:38.3773913Z STATS: call_* op count: 1308 | FakeTensor.__torch_dispatch__:16955 | FakeTensorMode.__torch_dispatch__:163105 | attempt fast:6894 | slow no contiguity match:301 | fast is_contiguous:6593 | ProxyTorchDispatchMode.__torch_dispatch__:40121 2024-06-26T07:37:38.3775535Z Dynamo produced 3 graphs covering 1308 ops with 7 graph breaks (5 unique) 2024-06-26T07:37:50.3857998Z 2024-06-26T07:37:51.1304955Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:51.1305417Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:37:51.1305854Z cuda train convmixer_768_32 2024-06-26T07:39:04.9729162Z pass 2024-06-26T07:39:04.9878001Z TIMING: entire_frame_compile:63.74054 code_gen:12.4543 inductor_compile:36.40373 backend_compile:57.36288 2024-06-26T07:39:04.9880638Z STATS: call_* op count: 235 | FakeTensor.__torch_dispatch__:12373 | FakeTensorMode.__torch_dispatch__:72527 | attempt fast:2379 | fast is_contiguous:2379 | ProxyTorchDispatchMode.__torch_dispatch__:15494 2024-06-26T07:39:04.9881967Z Dynamo produced 2 graphs covering 235 ops with 5 graph breaks (4 unique) 2024-06-26T07:39:11.7227257Z 2024-06-26T07:39:13.2808739Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:39:13.2809227Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:39:13.2809673Z cuda train convnext_base 2024-06-26T07:40:56.0498942Z W0626 07:40:56.048000 140551910830720 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:42:28.6683463Z pass 2024-06-26T07:42:28.7175825Z TIMING: entire_frame_compile:180.12465 code_gen:43.12661 inductor_compile:97.33817 backend_compile:152.17679 2024-06-26T07:42:28.7177722Z STATS: call_* op count: 1621 | FakeTensor.__torch_dispatch__:29579 | FakeTensorMode.__torch_dispatch__:180095 | attempt fast:4593 | fast is_contiguous:4023 | fast channels_last:284 | slow no contiguity match:286 | ProxyTorchDispatchMode.__torch_dispatch__:35464 2024-06-26T07:42:28.7179428Z Dynamo produced 3 graphs covering 1621 ops with 7 graph breaks (5 unique) 2024-06-26T07:42:40.8676885Z 2024-06-26T07:42:41.6289304Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:41.6289928Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:42:41.6290373Z cuda train crossvit_9_240 2024-06-26T07:44:37.0817639Z W0626 07:44:37.080000 139813740581504 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:45:49.6167626Z pass 2024-06-26T07:45:49.6299930Z TIMING: entire_frame_compile:176.34638 code_gen:41.98941 inductor_compile:98.59598 backend_compile:154.10098 2024-06-26T07:45:49.6301712Z STATS: call_* op count: 1441 | FakeTensor.__torch_dispatch__:25476 | FakeTensorMode.__torch_dispatch__:169448 | attempt fast:4696 | fast is_contiguous:4654 | slow no contiguity match:42 | ProxyTorchDispatchMode.__torch_dispatch__:34361 2024-06-26T07:45:49.6303192Z Dynamo produced 3 graphs covering 1441 ops with 7 graph breaks (5 unique) 2024-06-26T07:46:01.6520169Z 2024-06-26T07:46:02.6412618Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:02.6413077Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:46:02.6413529Z cuda train cspdarknet53 2024-06-26T07:47:39.2222492Z W0626 07:47:39.221000 139663196873344 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:48:36.5200447Z pass 2024-06-26T07:48:36.5353117Z TIMING: entire_frame_compile:141.5234 code_gen:34.7623 inductor_compile:81.0862 backend_compile:121.31062 2024-06-26T07:48:36.5355180Z STATS: call_* op count: 1070 | FakeTensor.__torch_dispatch__:23124 | FakeTensorMode.__torch_dispatch__:127201 | attempt fast:3256 | fast is_contiguous:3231 | slow no contiguity match:25 | ProxyTorchDispatchMode.__torch_dispatch__:24624 2024-06-26T07:48:36.5356909Z Dynamo produced 3 graphs covering 1070 ops with 7 graph breaks (5 unique) 2024-06-26T07:48:46.7029978Z 2024-06-26T07:48:48.1019135Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:48:48.1019610Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:48:48.1020073Z cuda train deit_base_distilled_patch16_224 2024-06-26T07:49:44.2277341Z W0626 07:49:44.226000 140667412345472 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:50:25.3541968Z pass 2024-06-26T07:50:25.4058545Z TIMING: entire_frame_compile:87.37253 code_gen:22.43745 inductor_compile:49.05472 backend_compile:75.96835 2024-06-26T07:50:25.4060289Z STATS: call_* op count: 869 | FakeTensor.__torch_dispatch__:14208 | FakeTensorMode.__torch_dispatch__:90883 | attempt fast:2348 | fast is_contiguous:2348 | ProxyTorchDispatchMode.__torch_dispatch__:18136 2024-06-26T07:50:25.4061619Z Dynamo produced 3 graphs covering 869 ops with 7 graph breaks (5 unique) 2024-06-26T07:50:32.9248525Z 2024-06-26T07:50:34.0398749Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:50:34.0399365Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:50:34.0399900Z cuda train dla102 2024-06-26T07:52:34.4163947Z W0626 07:52:34.415000 140664971620992 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T07:54:03.2640856Z pass 2024-06-26T07:54:03.2858001Z TIMING: entire_frame_compile:192.59432 code_gen:55.08978 inductor_compile:112.68028 backend_compile:170.41306 2024-06-26T07:54:03.2859651Z STATS: call_* op count: 1389 | FakeTensor.__torch_dispatch__:34776 | FakeTensorMode.__torch_dispatch__:190570 | attempt fast:3729 | fast is_contiguous:3651 | ProxyTorchDispatchMode.__torch_dispatch__:36994 | slow no contiguity match:78 2024-06-26T07:54:03.2861125Z Dynamo produced 3 graphs covering 1389 ops with 7 graph breaks (5 unique) 2024-06-26T07:54:15.7787716Z 2024-06-26T07:54:17.2856933Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:54:17.2857399Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:54:17.2859486Z cuda train dm_nfnet_f0 2024-06-26T07:55:43.6204803Z pass 2024-06-26T07:55:43.6706353Z TIMING: entire_frame_compile:78.70167 code_gen:17.84608 inductor_compile:45.44055 backend_compile:68.48167 2024-06-26T07:55:43.6707866Z STATS: call_* op count: 620 | FakeTensorMode.__torch_dispatch__:67531 | attempt fast:2906 | fast is_contiguous:2906 | FakeTensor.__torch_dispatch__:11878 | ProxyTorchDispatchMode.__torch_dispatch__:16005 2024-06-26T07:55:43.6712307Z Dynamo produced 2 graphs covering 620 ops with 6 graph breaks (5 unique) 2024-06-26T07:55:50.8596151Z 2024-06-26T07:55:52.2295088Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:55:52.2295638Z loading model: 0it [00:01, ?it/s] 2024-06-26T07:55:52.2296079Z cuda train dpn107 2024-06-26T07:58:57.0621018Z pass 2024-06-26T07:58:57.1230245Z TIMING: entire_frame_compile:165.85475 code_gen:41.11701 inductor_compile:93.50169 backend_compile:147.01767 2024-06-26T07:58:57.1233869Z STATS: call_* op count: 746 | FakeTensor.__torch_dispatch__:18918 | FakeTensorMode.__torch_dispatch__:127330 | attempt fast:4040 | fast is_contiguous:3741 | slow no contiguity match:299 | ProxyTorchDispatchMode.__torch_dispatch__:28636 2024-06-26T07:58:57.1235781Z Dynamo produced 2 graphs covering 746 ops with 6 graph breaks (5 unique) 2024-06-26T07:59:08.0227351Z 2024-06-26T07:59:08.7905690Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:08.7906163Z loading model: 0it [00:00, ?it/s] 2024-06-26T07:59:08.7909991Z cuda train eca_botnext26ts_256 2024-06-26T08:00:27.2446317Z W0626 08:00:27.243000 140048879604352 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:00:59.9927598Z pass 2024-06-26T08:01:00.0040431Z TIMING: entire_frame_compile:102.72282 code_gen:29.99351 inductor_compile:60.13087 backend_compile:89.57693 2024-06-26T08:01:00.0042054Z STATS: call_* op count: 760 | FakeTensor.__torch_dispatch__:12987 | FakeTensorMode.__torch_dispatch__:80455 | attempt fast:1825 | fast is_contiguous:1764 | slow no contiguity match:61 | ProxyTorchDispatchMode.__torch_dispatch__:15134 2024-06-26T08:01:00.0043573Z Dynamo produced 3 graphs covering 760 ops with 7 graph breaks (5 unique) 2024-06-26T08:01:08.0300551Z 2024-06-26T08:01:08.8227011Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:08.8228684Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:01:08.8229574Z cuda train eca_halonext26ts 2024-06-26T08:02:38.4570885Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:02:38.4573156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:02:38.4574385Z pred = mod(*cloned_inputs) 2024-06-26T08:02:38.4575374Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:38.4576307Z return forward_call(*args, **kwargs) 2024-06-26T08:02:38.4577440Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1257, in forward 2024-06-26T08:02:38.4578310Z x = self.forward_features(x) 2024-06-26T08:02:38.4579348Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 1249, in forward_features 2024-06-26T08:02:38.4580158Z x = self.stages(x) 2024-06-26T08:02:38.4580983Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:38.4581828Z return forward_call(*args, **kwargs) 2024-06-26T08:02:38.4582884Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/byobnet.py", line 883, in forward 2024-06-26T08:02:38.4583866Z x = self.self_attn(x) 2024-06-26T08:02:38.4584796Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:02:38.4585621Z return forward_call(*args, **kwargs) 2024-06-26T08:02:38.4586483Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/layers/halo_attn.py", line 191, in forward 2024-06-26T08:02:38.4587545Z kv = kv.unfold(2, self.win_size, self.block_size).unfold(3, self.win_size, self.block_size).reshape( 2024-06-26T08:02:38.4588481Z 2024-06-26T08:02:43.4695233Z W0626 08:02:43.468000 140644856939136 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:03:15.8645436Z pass 2024-06-26T08:03:15.8712610Z TIMING: entire_frame_compile:119.27143 code_gen:37.05567 inductor_compile:73.11306 backend_compile:104.32942 2024-06-26T08:03:15.8714255Z STATS: call_* op count: 790 | FakeTensor.__torch_dispatch__:12770 | FakeTensorMode.__torch_dispatch__:83185 | attempt fast:1828 | fast is_contiguous:1766 | slow no contiguity match:62 | ProxyTorchDispatchMode.__torch_dispatch__:15740 2024-06-26T08:03:15.8715723Z Dynamo produced 3 graphs covering 790 ops with 7 graph breaks (5 unique) 2024-06-26T08:03:24.3660019Z 2024-06-26T08:03:25.0232608Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:25.0233210Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:03:25.0234072Z cuda train ese_vovnet19b_dw 2024-06-26T08:04:11.9587753Z W0626 08:04:11.958000 139711619461760 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T08:04:42.0582758Z pass 2024-06-26T08:04:42.0654189Z TIMING: entire_frame_compile:69.95143 code_gen:25.5876 inductor_compile:44.6605 backend_compile:60.80162 2024-06-26T08:04:42.0657725Z STATS: call_* op count: 459 | FakeTensor.__torch_dispatch__:9656 | FakeTensorMode.__torch_dispatch__:55157 | attempt fast:925 | fast is_contiguous:901 | ProxyTorchDispatchMode.__torch_dispatch__:10392 | slow no contiguity match:24 2024-06-26T08:04:42.0659928Z Dynamo produced 4 graphs covering 459 ops with 7 graph breaks (5 unique) 2024-06-26T08:04:48.4535008Z 2024-06-26T08:04:49.2018486Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:04:49.2019298Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:04:49.2019903Z cuda train fbnetc_100 2024-06-26T08:06:02.5886922Z W0626 08:06:02.587000 139866795840128 torch/_logging/_internal.py:1040] [7/0] Profiler function will be ignored 2024-06-26T08:07:01.8072304Z pass 2024-06-26T08:07:01.8115018Z TIMING: entire_frame_compile:121.09391 code_gen:33.31034 inductor_compile:68.91295 backend_compile:103.89267 2024-06-26T08:07:01.8118066Z STATS: call_* op count: 993 | FakeTensor.__torch_dispatch__:20699 | FakeTensorMode.__torch_dispatch__:114042 | attempt fast:2264 | fast is_contiguous:2264 | ProxyTorchDispatchMode.__torch_dispatch__:22083 2024-06-26T08:07:01.8119409Z Dynamo produced 4 graphs covering 993 ops with 7 graph breaks (5 unique) 2024-06-26T08:07:10.8286568Z 2024-06-26T08:07:11.7328783Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:07:11.7329424Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:07:11.7330142Z cuda train fbnetv3_b 2024-06-26T08:09:53.7383457Z pass 2024-06-26T08:09:53.7464243Z TIMING: entire_frame_compile:151.9737 code_gen:31.2217 inductor_compile:85.90928 backend_compile:132.50423 2024-06-26T08:09:53.7467721Z STATS: call_* op count: 610 | FakeTensor.__torch_dispatch__:20844 | FakeTensorMode.__torch_dispatch__:122481 | attempt fast:5511 | fast is_contiguous:5511 | ProxyTorchDispatchMode.__torch_dispatch__:23345 2024-06-26T08:09:53.7471060Z Dynamo produced 2 graphs covering 610 ops with 6 graph breaks (5 unique) 2024-06-26T08:10:04.7675011Z 2024-06-26T08:10:05.8112898Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:10:05.8113946Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:10:05.8114902Z cuda train gernet_l 2024-06-26T08:11:11.6666461Z pass 2024-06-26T08:11:11.6884957Z TIMING: entire_frame_compile:59.78436 code_gen:12.30286 inductor_compile:32.99849 backend_compile:51.93281 2024-06-26T08:11:11.6887126Z STATS: call_* op count: 410 | FakeTensor.__torch_dispatch__:9236 | FakeTensorMode.__torch_dispatch__:55341 | attempt fast:2021 | fast is_contiguous:2021 | ProxyTorchDispatchMode.__torch_dispatch__:12116 2024-06-26T08:11:11.6888448Z Dynamo produced 2 graphs covering 410 ops with 6 graph breaks (5 unique) 2024-06-26T08:11:18.0835684Z 2024-06-26T08:11:18.8788608Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:18.8789377Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:11:18.8792847Z cuda train ghostnet_100 2024-06-26T08:13:13.9532678Z pass 2024-06-26T08:13:13.9578075Z TIMING: entire_frame_compile:107.63459 code_gen:27.36294 inductor_compile:64.91124 backend_compile:97.40333 2024-06-26T08:13:13.9581068Z STATS: call_* op count: 360 | FakeTensor.__torch_dispatch__:15456 | FakeTensorMode.__torch_dispatch__:92512 | attempt fast:3045 | fast is_contiguous:2981 | ProxyTorchDispatchMode.__torch_dispatch__:19864 | slow no contiguity match:64 2024-06-26T08:13:13.9582573Z Dynamo produced 2 graphs covering 360 ops with 6 graph breaks (5 unique) 2024-06-26T08:13:22.8168947Z 2024-06-26T08:13:23.9198795Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:13:23.9199274Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:13:23.9199713Z cuda train gluon_inception_v3 2024-06-26T08:14:19.6866826Z W0626 08:14:19.685000 140583964521088 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:15:37.7809877Z pass 2024-06-26T08:15:37.8034243Z TIMING: entire_frame_compile:122.57821 inductor_compile:45.56115 backend_compile:97.35416 code_gen:21.59255 2024-06-26T08:15:37.8037283Z STATS: call_* op count: 1376 | FakeTensor.__torch_dispatch__:17669 | FakeTensorMode.__torch_dispatch__:136254 | attempt fast:2205 | fast is_contiguous:2205 | ProxyTorchDispatchMode.__torch_dispatch__:33213 2024-06-26T08:15:37.8038723Z Dynamo produced 3 graphs covering 1376 ops with 7 graph breaks (5 unique) 2024-06-26T08:15:46.8223408Z 2024-06-26T08:15:47.6898145Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:47.6898626Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:15:47.6903076Z cuda train gmixer_24_224 2024-06-26T08:17:47.6185976Z pass 2024-06-26T08:17:47.6431743Z TIMING: entire_frame_compile:112.05757 code_gen:20.49815 inductor_compile:58.35028 backend_compile:99.15081 2024-06-26T08:17:47.6435170Z STATS: call_* op count: 684 | FakeTensor.__torch_dispatch__:12729 | FakeTensorMode.__torch_dispatch__:110658 | attempt fast:4737 | fast is_contiguous:4017 | slow no contiguity match:720 | ProxyTorchDispatchMode.__torch_dispatch__:23778 2024-06-26T08:17:47.6437292Z Dynamo produced 2 graphs covering 684 ops with 6 graph breaks (5 unique) 2024-06-26T08:17:56.6644378Z 2024-06-26T08:17:57.4004600Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:17:57.4005342Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:17:57.4006025Z cuda train gmlp_s16_224 2024-06-26T08:19:45.9312048Z W0626 08:19:45.930000 140063520973440 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:21:03.3509619Z pass 2024-06-26T08:21:03.3709243Z TIMING: entire_frame_compile:173.324 code_gen:40.05501 inductor_compile:95.05574 backend_compile:151.24269 2024-06-26T08:21:03.3710927Z STATS: call_* op count: 1458 | FakeTensor.__torch_dispatch__:28861 | FakeTensorMode.__torch_dispatch__:180365 | attempt fast:5367 | fast is_contiguous:4917 | slow no contiguity match:450 | ProxyTorchDispatchMode.__torch_dispatch__:35364 2024-06-26T08:21:03.3712470Z Dynamo produced 3 graphs covering 1458 ops with 7 graph breaks (5 unique) 2024-06-26T08:21:15.3530507Z 2024-06-26T08:21:16.9401184Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:21:16.9401671Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:21:16.9402129Z cuda train hrnet_w18 2024-06-26T08:26:33.6814422Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:26:33.6815615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:26:33.6817226Z pred = mod(*cloned_inputs) 2024-06-26T08:26:33.6818530Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:26:33.6819751Z return forward_call(*args, **kwargs) 2024-06-26T08:26:33.6820704Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 792, in forward 2024-06-26T08:26:33.6821563Z y = self.forward_features(x) 2024-06-26T08:26:33.6822742Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 770, in forward_features 2024-06-26T08:26:33.6823553Z yl = self.stages(x) 2024-06-26T08:26:33.6824337Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 751, in stages 2024-06-26T08:26:33.6825092Z yl = self.stage2(xl) 2024-06-26T08:26:33.6825941Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:26:33.6826781Z return forward_call(*args, **kwargs) 2024-06-26T08:26:33.6827811Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 506, in forward 2024-06-26T08:26:33.6828886Z x = module(x) 2024-06-26T08:26:33.6829711Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:26:33.6830547Z return forward_call(*args, **kwargs) 2024-06-26T08:26:33.6831392Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/hrnet.py", line 484, in forward 2024-06-26T08:26:33.6832406Z y = y + f(x[j]) 2024-06-26T08:26:33.6832619Z 2024-06-26T08:26:47.4802137Z pass 2024-06-26T08:26:47.4983037Z TIMING: entire_frame_compile:314.13268 code_gen:72.74736 inductor_compile:182.38343 backend_compile:289.2293 2024-06-26T08:26:47.4984740Z STATS: call_* op count: 1394 | FakeTensor.__torch_dispatch__:52796 | FakeTensorMode.__torch_dispatch__:319842 | attempt fast:11855 | fast is_contiguous:11855 | ProxyTorchDispatchMode.__torch_dispatch__:71274 2024-06-26T08:26:47.4986255Z Dynamo produced 2 graphs covering 1394 ops with 5 graph breaks (4 unique) 2024-06-26T08:27:07.4039885Z 2024-06-26T08:27:08.5318925Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:27:08.5319470Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:27:08.5319974Z cuda train inception_v3 2024-06-26T08:28:06.1733219Z pass 2024-06-26T08:28:06.1976596Z TIMING: entire_frame_compile:50.64309 inductor_compile:5.5929 backend_compile:38.05662 2024-06-26T08:28:06.1978005Z STATS: call_* op count: 506 | FakeTensor.__torch_dispatch__:2985 | FakeTensorMode.__torch_dispatch__:60332 | attempt fast:2205 | fast is_contiguous:2205 | ProxyTorchDispatchMode.__torch_dispatch__:20409 2024-06-26T08:28:06.1979316Z Dynamo produced 2 graphs covering 506 ops with 6 graph breaks (5 unique) 2024-06-26T08:28:11.8669735Z 2024-06-26T08:28:13.2314492Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:28:13.2315104Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:28:13.2315653Z cuda train jx_nest_base 2024-06-26T08:31:04.5930177Z W0626 08:31:04.592000 139838837334656 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:32:23.8169286Z pass 2024-06-26T08:32:23.8627311Z TIMING: entire_frame_compile:231.26757 code_gen:49.80851 inductor_compile:122.04074 backend_compile:201.35425 2024-06-26T08:32:23.8630308Z STATS: call_* op count: 1852 | FakeTensor.__torch_dispatch__:29052 | FakeTensorMode.__torch_dispatch__:215199 | attempt fast:5712 | fast is_contiguous:5712 | ProxyTorchDispatchMode.__torch_dispatch__:42081 2024-06-26T08:32:23.8632399Z Dynamo produced 3 graphs covering 1852 ops with 7 graph breaks (5 unique) 2024-06-26T08:32:38.3700042Z 2024-06-26T08:32:38.9455561Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:38.9456065Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:32:38.9456648Z cuda train lcnet_050 2024-06-26T08:33:29.4699620Z pass 2024-06-26T08:33:29.4723965Z TIMING: entire_frame_compile:46.94474 code_gen:10.46483 inductor_compile:26.74893 backend_compile:40.87679 2024-06-26T08:33:29.4725985Z STATS: call_* op count: 168 | FakeTensor.__torch_dispatch__:6295 | FakeTensorMode.__torch_dispatch__:36876 | attempt fast:1707 | fast is_contiguous:1707 | ProxyTorchDispatchMode.__torch_dispatch__:6908 2024-06-26T08:33:29.4727395Z Dynamo produced 2 graphs covering 168 ops with 6 graph breaks (5 unique) 2024-06-26T08:33:35.2938809Z 2024-06-26T08:33:36.0832825Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:36.0833328Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:33:36.0833769Z cuda train levit_128 2024-06-26T08:35:47.2476308Z skipping cudagraphs due to deterministic index put. Found from : 2024-06-26T08:35:47.2478588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/timm_models.py", line 371, in torch_dynamo_resume_in_forward_and_backward_pass_at_369 2024-06-26T08:35:47.2480465Z pred = mod(*cloned_inputs) 2024-06-26T08:35:47.2481906Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:47.2482765Z return forward_call(*args, **kwargs) 2024-06-26T08:35:47.2483964Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 652, in forward 2024-06-26T08:35:47.2484879Z x = self.forward_features(x) 2024-06-26T08:35:47.2485752Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 643, in forward_features 2024-06-26T08:35:47.2486950Z x = self.stages(x) 2024-06-26T08:35:47.2487791Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:47.2488758Z return forward_call(*args, **kwargs) 2024-06-26T08:35:47.2489597Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 518, in forward 2024-06-26T08:35:47.2490326Z x = self.blocks(x) 2024-06-26T08:35:47.2491148Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:47.2491977Z return forward_call(*args, **kwargs) 2024-06-26T08:35:47.2492817Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 456, in forward 2024-06-26T08:35:47.2493587Z x = x + self.drop_path1(self.attn(x)) 2024-06-26T08:35:47.2494504Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1668, in _call_impl 2024-06-26T08:35:47.2495334Z return forward_call(*args, **kwargs) 2024-06-26T08:35:47.2496168Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 236, in forward 2024-06-26T08:35:47.2497037Z attn = q @ k * self.scale + self.get_attention_biases(x.device) 2024-06-26T08:35:47.2498071Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/timm/models/levit.py", line 211, in get_attention_biases 2024-06-26T08:35:47.2498998Z return self.attention_biases[:, self.attention_bias_idxs] 2024-06-26T08:35:47.2499386Z 2024-06-26T08:35:51.6283706Z W0626 08:35:51.627000 140281942282880 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:36:46.8689676Z pass 2024-06-26T08:36:46.8725146Z TIMING: entire_frame_compile:181.40392 code_gen:48.5677 inductor_compile:102.90409 backend_compile:160.12571 2024-06-26T08:36:46.8727901Z STATS: call_* op count: 1257 | FakeTensor.__torch_dispatch__:24160 | FakeTensorMode.__torch_dispatch__:155969 | attempt fast:3507 | fast is_contiguous:3459 | slow no contiguity match:46 | ProxyTorchDispatchMode.__torch_dispatch__:32073 | fast channels_last:2 2024-06-26T08:36:46.8729512Z Dynamo produced 3 graphs covering 1257 ops with 7 graph breaks (5 unique) 2024-06-26T08:36:59.3567119Z 2024-06-26T08:37:00.4434172Z loading model: 0it [00:00, ?it/s] 2024-06-26T08:37:00.4434644Z loading model: 0it [00:01, ?it/s] 2024-06-26T08:37:00.4435083Z cuda train mixer_b16_224 2024-06-26T08:37:59.1843159Z W0626 08:37:59.183000 140188363530880 torch/_logging/_internal.py:1040] [6/0] Profiler function will be ignored 2024-06-26T08:38:39.1850569Z pass 2024-06-26T08:38:39.2267432Z TIMING: entire_frame_compile:89.7782 code_gen:22.79633 inductor_compile:50.2901 backend_compile:78.52476 2024-06-26T08:38:39.2270957Z STATS: call_* op count: 720 | FakeTensor.__torch_dispatch__:14328 | FakeTensorMode.__torch_dispatch__:88843 | attempt fast:2769 | fast is_contiguous:2625 | slow no contiguity match:144 | ProxyTorchDispatchMode.__torch_dispatch__:17844 2024-06-26T08:38:39.2273761Z Dynamo produced 3 graphs covering 720 ops with 7 graph breaks (5 unique) 2024-06-26T08:38:43.9955486Z accuracy pass_rate=93.33% 2024-06-26T08:38:43.9958188Z calls_captured gmean=0.00x mean=852.267x 2024-06-26T08:38:43.9961450Z unique_graphs gmean=0.00x mean=2.533x 2024-06-26T08:38:43.9965067Z graph_breaks gmean=0.00x mean=6.267x 2024-06-26T08:38:43.9968756Z unique_graph_breaks gmean=0.00x mean=4.767x 2024-06-26T08:38:43.9971931Z autograd_captures gmean=0.00x mean=0.000x 2024-06-26T08:38:43.9975122Z autograd_compiles gmean=0.00x mean=0.000x 2024-06-26T08:38:43.9978425Z cudagraph_skips gmean=0.00x mean=0.133x 2024-06-26T08:38:45.4446811Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_timm_training.csv 2024-06-26T08:38:45.7002748Z adv_inception_v3 PASS 2024-06-26T08:38:45.7006481Z beit_base_patch16_224 XFAIL 2024-06-26T08:38:45.7011944Z botnet26t_256 PASS 2024-06-26T08:38:45.7017179Z cait_m36_384 XFAIL 2024-06-26T08:38:45.7022314Z coat_lite_mini PASS 2024-06-26T08:38:45.7027351Z convit_base PASS 2024-06-26T08:38:45.7032757Z convmixer_768_32 PASS 2024-06-26T08:38:45.7037817Z convnext_base PASS 2024-06-26T08:38:45.7043035Z crossvit_9_240 PASS 2024-06-26T08:38:45.7048076Z cspdarknet53 PASS 2024-06-26T08:38:45.7053066Z deit_base_distilled_patch16_224 PASS 2024-06-26T08:38:45.7057960Z dla102 PASS 2024-06-26T08:38:45.7063012Z dm_nfnet_f0 PASS 2024-06-26T08:38:45.7067979Z dpn107 PASS 2024-06-26T08:38:45.7073407Z eca_botnext26ts_256 PASS 2024-06-26T08:38:45.7078108Z eca_halonext26ts PASS 2024-06-26T08:38:45.7083052Z ese_vovnet19b_dw PASS 2024-06-26T08:38:45.7088241Z fbnetc_100 PASS 2024-06-26T08:38:45.7093214Z fbnetv3_b PASS 2024-06-26T08:38:45.7098148Z gernet_l PASS 2024-06-26T08:38:45.7103094Z ghostnet_100 PASS 2024-06-26T08:38:45.7108242Z gluon_inception_v3 PASS 2024-06-26T08:38:45.7113279Z gmixer_24_224 PASS 2024-06-26T08:38:45.7118305Z gmlp_s16_224 PASS 2024-06-26T08:38:45.7123160Z hrnet_w18 PASS 2024-06-26T08:38:45.7128463Z inception_v3 PASS 2024-06-26T08:38:45.7133349Z jx_nest_base PASS 2024-06-26T08:38:45.7138284Z lcnet_050 PASS 2024-06-26T08:38:45.7143176Z levit_128 PASS 2024-06-26T08:38:45.7148380Z mixer_b16_224 PASS 2024-06-26T08:38:45.7527185Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_timm_models.csv --expected benchmarks/dynamo/ci_expected_accuracy//dynamic_inductor_timm_training.csv 2024-06-26T08:38:46.0299223Z adv_inception_v3 PASS 2024-06-26T08:38:46.0303713Z beit_base_patch16_224 PASS 2024-06-26T08:38:46.0307364Z botnet26t_256 PASS 2024-06-26T08:38:46.0312523Z cait_m36_384 PASS 2024-06-26T08:38:46.0316829Z coat_lite_mini PASS 2024-06-26T08:38:46.0321364Z convit_base PASS 2024-06-26T08:38:46.0325948Z convmixer_768_32 PASS 2024-06-26T08:38:46.0330674Z convnext_base PASS 2024-06-26T08:38:46.0335086Z crossvit_9_240 PASS 2024-06-26T08:38:46.0339712Z cspdarknet53 PASS 2024-06-26T08:38:46.0344567Z deit_base_distilled_patch16_224 PASS 2024-06-26T08:38:46.0348982Z dla102 PASS 2024-06-26T08:38:46.0353531Z dm_nfnet_f0 PASS 2024-06-26T08:38:46.0357872Z dpn107 PASS 2024-06-26T08:38:46.0362268Z eca_botnext26ts_256 PASS 2024-06-26T08:38:46.0366813Z eca_halonext26ts PASS 2024-06-26T08:38:46.0371358Z ese_vovnet19b_dw PASS 2024-06-26T08:38:46.0375845Z fbnetc_100 PASS 2024-06-26T08:38:46.0380294Z fbnetv3_b PASS 2024-06-26T08:38:46.0384640Z gernet_l PASS 2024-06-26T08:38:46.0389401Z ghostnet_100 PASS 2024-06-26T08:38:46.0393666Z gluon_inception_v3 PASS 2024-06-26T08:38:46.0398359Z gmixer_24_224 PASS 2024-06-26T08:38:46.0402930Z gmlp_s16_224 PASS 2024-06-26T08:38:46.0407342Z hrnet_w18 PASS 2024-06-26T08:38:46.0412042Z inception_v3 PASS 2024-06-26T08:38:46.0416202Z jx_nest_base PASS 2024-06-26T08:38:46.0420610Z lcnet_050 PASS 2024-06-26T08:38:46.0425111Z levit_128 PASS 2024-06-26T08:38:46.0429552Z mixer_b16_224 PASS 2024-06-26T08:38:46.0800740Z + cleanup_workspace 2024-06-26T08:38:46.0801616Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2024-06-26T08:38:46.0802696Z sudo may print the following warning message that can be ignored. The chown command will still run. 2024-06-26T08:38:46.0803601Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2024-06-26T08:38:46.0804252Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2024-06-26T08:38:46.0805057Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2024-06-26T08:38:46.0806306Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2024-06-26T08:38:46.0807144Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2024-06-26T08:38:46.6024974Z ##[group]Run cat test/**/*_toprint.log || true 2024-06-26T08:38:46.6025512Z cat test/**/*_toprint.log || true 2024-06-26T08:38:46.6033649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:46.6034151Z env: 2024-06-26T08:38:46.6034437Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.6034888Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.6035650Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.6036306Z ##[endgroup] 2024-06-26T08:38:46.6098046Z cat: test/**/*_toprint.log: No such file or directory 2024-06-26T08:38:46.6145173Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:38:46.6145649Z kill "$MONITOR_SCRIPT_PID" 2024-06-26T08:38:46.6153635Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:46.6154152Z env: 2024-06-26T08:38:46.6154432Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.6154912Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.6155727Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.6156412Z MONITOR_SCRIPT_PID: 2618 2024-06-26T08:38:46.6156758Z ##[endgroup] 2024-06-26T08:38:46.6341485Z Prepare all required actions 2024-06-26T08:38:46.6341934Z Getting action download info 2024-06-26T08:38:46.7968655Z Download action repository 'actions/upload-artifact@v3' (SHA:a8a3f3ad30e3422c9c7b888a15615d19a852ae32) 2024-06-26T08:38:46.9440149Z ##[group]Run ./.github/actions/upload-test-artifacts 2024-06-26T08:38:46.9440629Z with: 2024-06-26T08:38:46.9441167Z file-suffix: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727 2024-06-26T08:38:46.9441838Z s3-bucket: gha-artifacts 2024-06-26T08:38:46.9442184Z env: 2024-06-26T08:38:46.9442459Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.9442920Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.9443708Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.9444371Z ##[endgroup] 2024-06-26T08:38:46.9469417Z ##[group]Run # Remove any previous test jsons if they exist 2024-06-26T08:38:46.9470078Z # Remove any previous test jsons if they exist 2024-06-26T08:38:46.9470595Z rm -f test-jsons-*.zip 2024-06-26T08:38:46.9471122Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json' 2024-06-26T08:38:46.9478958Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:46.9479458Z env: 2024-06-26T08:38:46.9479750Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.9480211Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.9480968Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.9481880Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727 2024-06-26T08:38:46.9482525Z ##[endgroup] 2024-06-26T08:38:46.9653568Z adding: test/allowlist_for_publicAPI.json (deflated 79%) 2024-06-26T08:38:46.9682872Z adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%) 2024-06-26T08:38:46.9683563Z adding: test/minioptest_failures_dict.json (deflated 70%) 2024-06-26T08:38:46.9689594Z adding: test/profiler/profiler_utils_mock_events.json (deflated 87%) 2024-06-26T08:38:46.9714998Z ##[group]Run # Remove any previous test reports if they exist 2024-06-26T08:38:46.9715636Z # Remove any previous test reports if they exist 2024-06-26T08:38:46.9716157Z rm -f test-reports-*.zip 2024-06-26T08:38:46.9716747Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv' 2024-06-26T08:38:46.9724372Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:46.9724875Z env: 2024-06-26T08:38:46.9725144Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.9725762Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.9726521Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.9727445Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727 2024-06-26T08:38:46.9728085Z ##[endgroup] 2024-06-26T08:38:46.9885560Z adding: test/test-reports/inference_timm_models.csv (deflated 69%) 2024-06-26T08:38:46.9887191Z adding: test/test-reports/training_timm_models.csv (deflated 65%) 2024-06-26T08:38:46.9890646Z adding: test/test-reports/training_timm_model_graph_breaks.csv (deflated 97%) 2024-06-26T08:38:46.9892497Z adding: test/test-reports/training_timm_model_graph_break_deduped.csv (deflated 73%) 2024-06-26T08:38:46.9917759Z ##[group]Run # Remove any previous usage logs if they exist 2024-06-26T08:38:46.9918610Z # Remove any previous usage logs if they exist 2024-06-26T08:38:46.9919119Z rm -f logs-*.zip 2024-06-26T08:38:46.9919794Z # this workflow is also run in bazel build test, but we dont generate usage reports for it 2024-06-26T08:38:46.9920600Z # so check to see if the file exists first 2024-06-26T08:38:46.9921115Z if [ -f 'usage_log.txt' ]; then 2024-06-26T08:38:46.9921647Z  zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' 2024-06-26T08:38:46.9922138Z fi 2024-06-26T08:38:46.9922637Z if ls test/**/*.log 1> /dev/null 2>&1; then 2024-06-26T08:38:46.9923222Z  zip -r "logs-${FILE_SUFFIX}.zip" test -i '*.log' 2024-06-26T08:38:46.9923721Z fi 2024-06-26T08:38:46.9931142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:46.9931640Z env: 2024-06-26T08:38:46.9931924Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:46.9932380Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:46.9933137Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:46.9934045Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727 2024-06-26T08:38:46.9934689Z ##[endgroup] 2024-06-26T08:38:47.0011454Z adding: usage_log.txt (deflated 92%) 2024-06-26T08:38:47.0079753Z ##[group]Run # Remove any previous debugging artifacts if they exist 2024-06-26T08:38:47.0080466Z # Remove any previous debugging artifacts if they exist 2024-06-26T08:38:47.0081019Z rm -f debug-*.zip 2024-06-26T08:38:47.0081411Z if [ -d 'test/debug' ]; then 2024-06-26T08:38:47.0081922Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2024-06-26T08:38:47.0082400Z fi 2024-06-26T08:38:47.0089318Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:47.0089813Z env: 2024-06-26T08:38:47.0090093Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:47.0090546Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:47.0091300Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:47.0092212Z FILE_SUFFIX: test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727 2024-06-26T08:38:47.0093000Z ##[endgroup] 2024-06-26T08:38:47.0167442Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:38:47.0167882Z with: 2024-06-26T08:38:47.0168161Z s3-bucket: gha-artifacts 2024-06-26T08:38:47.0168598Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:47.0169067Z retention-days: 14 2024-06-26T08:38:47.0169394Z if-no-files-found: warn 2024-06-26T08:38:47.0169747Z path: test-jsons-*.zip 2024-06-26T08:38:47.0170086Z name: artifact 2024-06-26T08:38:47.0170387Z region: us-east-1 2024-06-26T08:38:47.0170680Z env: 2024-06-26T08:38:47.0170956Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:47.0171406Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:47.0172159Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:47.0172813Z ##[endgroup] 2024-06-26T08:38:47.3216649Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:38:47.3217404Z With the provided path, there will be 1 file uploaded 2024-06-26T08:38:47.3218052Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:47.3243993Z Starting upload of test-jsons-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:47.4643313Z Finished upload of test-jsons-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:47.4774714Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:38:47.4775157Z with: 2024-06-26T08:38:47.4775440Z s3-bucket: gha-artifacts 2024-06-26T08:38:47.4775865Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:47.4776337Z retention-days: 14 2024-06-26T08:38:47.4776667Z if-no-files-found: error 2024-06-26T08:38:47.4777035Z path: test-reports-*.zip 2024-06-26T08:38:47.4777381Z name: artifact 2024-06-26T08:38:47.4777679Z region: us-east-1 2024-06-26T08:38:47.4777985Z env: 2024-06-26T08:38:47.4778262Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:47.4778717Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:47.4779476Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:47.4780135Z ##[endgroup] 2024-06-26T08:38:47.7839204Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:38:47.7840221Z With the provided path, there will be 1 file uploaded 2024-06-26T08:38:47.7840839Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:47.7878520Z Starting upload of test-reports-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:47.8679121Z Finished upload of test-reports-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:47.8817438Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:38:47.8817889Z with: 2024-06-26T08:38:47.8818173Z s3-bucket: gha-artifacts 2024-06-26T08:38:47.8818609Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:47.8819076Z retention-days: 14 2024-06-26T08:38:47.8819420Z if-no-files-found: ignore 2024-06-26T08:38:47.8819781Z path: logs-*.zip 2024-06-26T08:38:47.8820091Z name: artifact 2024-06-26T08:38:47.8820394Z region: us-east-1 2024-06-26T08:38:47.8820688Z env: 2024-06-26T08:38:47.8820969Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:47.8821438Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:47.8822196Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:47.8822844Z ##[endgroup] 2024-06-26T08:38:48.1869227Z NOTE: s3-prefix specified, ignoring name parameter 2024-06-26T08:38:48.1869935Z With the provided path, there will be 1 file uploaded 2024-06-26T08:38:48.1870604Z Uploading to s3 prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:48.1896499Z Starting upload of logs-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:48.3230450Z Finished upload of logs-test-dynamic_inductor_timm-1-2-linux.g5.4xlarge.nvidia.gpu_26688911727.zip 2024-06-26T08:38:48.3358595Z ##[group]Run seemethere/upload-artifact-s3@v5 2024-06-26T08:38:48.3359037Z with: 2024-06-26T08:38:48.3359323Z s3-bucket: gha-artifacts 2024-06-26T08:38:48.3359753Z s3-prefix: pytorch/pytorch/9673646808/1/artifact 2024-06-26T08:38:48.3360233Z retention-days: 14 2024-06-26T08:38:48.3360567Z if-no-files-found: ignore 2024-06-26T08:38:48.3360929Z path: debug-*.zip 2024-06-26T08:38:48.3361240Z name: artifact 2024-06-26T08:38:48.3361535Z region: us-east-1 2024-06-26T08:38:48.3361839Z env: 2024-06-26T08:38:48.3362114Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:48.3362568Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:48.3363316Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:48.3363979Z ##[endgroup] 2024-06-26T08:38:48.6427226Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2024-06-26T08:38:48.6560239Z ##[group]Run # shellcheck disable=SC2156 2024-06-26T08:38:48.6560739Z # shellcheck disable=SC2156 2024-06-26T08:38:48.6561533Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2024-06-26T08:38:48.6569538Z shell: /usr/bin/bash -e {0} 2024-06-26T08:38:48.6569896Z env: 2024-06-26T08:38:48.6570174Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:48.6570632Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:48.6571391Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:48.6572057Z ##[endgroup] 2024-06-26T08:38:48.8172083Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2024-06-26T08:38:48.8172640Z with: 2024-06-26T08:38:48.8172910Z env: 2024-06-26T08:38:48.8173200Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:48.8173665Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:48.8174414Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:48.8175095Z ##[endgroup] 2024-06-26T08:38:48.8192698Z ##[group]Run set -eou pipefail 2024-06-26T08:38:48.8193105Z set -eou pipefail 2024-06-26T08:38:48.8193461Z  2024-06-26T08:38:48.8193973Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2024-06-26T08:38:48.8194620Z for _ in $(seq 1440); do 2024-06-26T08:38:48.8195093Z  # Break if no ssh session exists anymore 2024-06-26T08:38:48.8195595Z  if [ "$(who)" = "" ]; then 2024-06-26T08:38:48.8196033Z  break 2024-06-26T08:38:48.8196400Z  fi 2024-06-26T08:38:48.8196714Z  echo "." 2024-06-26T08:38:48.8197053Z  sleep 5 2024-06-26T08:38:48.8197379Z done 2024-06-26T08:38:48.8204677Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:48.8205174Z env: 2024-06-26T08:38:48.8205456Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:48.8206030Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:48.8206791Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:48.8207447Z ##[endgroup] 2024-06-26T08:38:48.8226624Z Holding runner for 2 hours until all ssh sessions have logged out 2024-06-26T08:38:48.8262458Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:38:48.8263218Z # ignore expansion of "docker ps -q" since it could be empty 2024-06-26T08:38:48.8263801Z # shellcheck disable=SC2046 2024-06-26T08:38:48.8264271Z docker stop $(docker ps -q) || true 2024-06-26T08:38:48.8264755Z # Prune all of the docker images 2024-06-26T08:38:48.8265207Z docker system prune -af 2024-06-26T08:38:48.8272295Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:48.8272793Z env: 2024-06-26T08:38:48.8273078Z GIT_DEFAULT_BRANCH: main 2024-06-26T08:38:48.8273534Z GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all 2024-06-26T08:38:48.8274407Z DOCKER_CONTAINER_ID: dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:48.8275068Z ##[endgroup] 2024-06-26T08:38:49.1145875Z dd8cd24a36d5 2024-06-26T08:38:49.8614282Z Deleted Containers: 2024-06-26T08:38:49.8614988Z dd8cd24a36d56bffdda10dc0e21a0e8dc1ff0ac5ffe70b20503b6968a153a425 2024-06-26T08:38:49.8615537Z 2024-06-26T08:38:54.8895281Z Deleted Images: 2024-06-26T08:38:54.8898414Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks:91382da70d5719cd7007b6b80b71d2f48398f6b7 2024-06-26T08:38:54.8901160Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.1-cudnn9-py3-gcc9-inductor-benchmarks@sha256:216c88a4f79ebfa71a5dcf4d9a946548184354a3ea1f677f4b1065d28705debb 2024-06-26T08:38:54.8902687Z deleted: sha256:0058b15e517a024817bccd921dc9af264c964f500b3cb9b3f4649a8ac22cc2dd 2024-06-26T08:38:54.8903521Z deleted: sha256:754ead7012f0b7691885a9660f02e80be6cee8c5665486fb995c1fb06adb6e92 2024-06-26T08:38:54.8904362Z deleted: sha256:fbe4496d8830e11be86d8f222522e7956f32f85bfc06b05ea53961f34a58af74 2024-06-26T08:38:54.8905195Z deleted: sha256:1c83814aa1a1a7e0cc998e3479bdab8e8766b975560cbf00690acf39ed899a80 2024-06-26T08:38:54.8906042Z deleted: sha256:e9ea8e61942c7f8ae4aab97b8d102b01d6bb4b6cf2a605808cd34130a17d8a0f 2024-06-26T08:38:54.8906873Z deleted: sha256:acc5fc32588cd95e082a6ce35e48d0f303ae8f96e3ab65c4e375b85e1f3c4a92 2024-06-26T08:38:54.8907678Z deleted: sha256:a93e53400904d29e4185d3be942ea34f19572ea438962788ae4e9b3d9bf4c498 2024-06-26T08:38:54.8909023Z deleted: sha256:c97092b88b5e4c77c68bbc4b51afab64e38719af1ff58013ec49a696cdd40a6e 2024-06-26T08:38:54.8909858Z deleted: sha256:b132889c90d67ffa53655846603e3fcc2372c23f04dbdbc54690a39d304eb89b 2024-06-26T08:38:54.8910661Z deleted: sha256:915d54081ed607710624abe3f2490114c745858831f4fc13396bd3b4b9a121bc 2024-06-26T08:38:54.8911458Z deleted: sha256:bfa8d446c498c87f535aef102695564de1b22c755e6c63944ae2f5073a2b31ec 2024-06-26T08:38:54.8912280Z deleted: sha256:e71460fb363c29e70bbe357d680aaa7ae694f094ef2f2713609158ad90611744 2024-06-26T08:38:54.8913123Z deleted: sha256:84079bb1b4ee1d0119ae3d3e701de460a1324912297c675425c9a36d30fcdb0f 2024-06-26T08:38:54.8913944Z deleted: sha256:d86ac3ca148d4758cb03b493fdd49109d8f1a2c7c85c4b755493c3c4529553e5 2024-06-26T08:38:54.8914861Z deleted: sha256:532b3ab97c44a673a0d5f62ee46fa6a83248a84de8877754a612355be56ed0a3 2024-06-26T08:38:54.8915819Z deleted: sha256:617e2a35f25ae24500c88c0c9b8475f0687fbcbed3d200013b4c99c6d8941aff 2024-06-26T08:38:54.8916688Z deleted: sha256:b88120bcd5a26a660c09c5df13176e0ef098fb2c8cdb57d74ab0796a3d1194d6 2024-06-26T08:38:54.8917511Z deleted: sha256:c016b2e7764735243205fb76fa9a273bb3dd4bf05532288e3a613ce022131ac3 2024-06-26T08:38:54.8918347Z deleted: sha256:bcaca462c390deddd2995de2d245a7495e7d2e1e94f0edad30a573c83d0588fd 2024-06-26T08:38:54.8919187Z deleted: sha256:9e260eba003d5c25d7b410e5c4c3093845d9e46611896840901c689d7fbf8ee6 2024-06-26T08:38:54.8920015Z deleted: sha256:3e62b86f8b911a9b59c62d55f063e71683ae6a01c22418db5d27930712adaad6 2024-06-26T08:38:54.8920826Z deleted: sha256:905920ccb1d88449b8dde686c34a4b2a8b6f59e78ac44916b189ea3593b9fbbb 2024-06-26T08:38:54.8921638Z deleted: sha256:961201237e03acd5061406238dd7a3c1343ac888d7583212cd0ab5ae1b4f0540 2024-06-26T08:38:54.8922454Z deleted: sha256:70047a2aac62159fa28fb4dbd4567ec2ffbab68fe046967e09acb0eb3e5cb9cb 2024-06-26T08:38:54.8923292Z deleted: sha256:1ec680ad781436ae65fa987f5fa1ed032ecd5c82a6e35eea6707de501f62c0f4 2024-06-26T08:38:54.8924128Z deleted: sha256:ef859623a50e85abf6e6c1dbd6188aa765a66cc7b2bd618a1cbe4825b7b2bd91 2024-06-26T08:38:54.8924962Z deleted: sha256:404ade5cacff59773b93ab438bfa64cd3a171d07f6f168bdc4a5d4bfa0b43526 2024-06-26T08:38:54.8925910Z deleted: sha256:5b27de6ee3d9f2f18217bf18dec725f7b4954c0e48c85d35752634c22d62209b 2024-06-26T08:38:54.8926754Z deleted: sha256:498e39c575b3fa069acaac7e46b91b4e54ae3a05f2ada5c4347091f346f25d4c 2024-06-26T08:38:54.8927553Z deleted: sha256:1405919d2694d65669b1f4c842282021afe89208ce86570e96464903733da2d0 2024-06-26T08:38:54.8928560Z deleted: sha256:d5b41e5d70896c0472fc3e6da0516a17d89cd6635dc75010ecddc31ce9690fad 2024-06-26T08:38:54.8929374Z deleted: sha256:18e9379b62b9d0bb40293d28a652eafaee66a499f66f0776bf14e37a513101b7 2024-06-26T08:38:54.8930192Z deleted: sha256:a781f80ed00ff2926a699fd01a309bb0b4fb7d5655eb652c205bb9c7f811035f 2024-06-26T08:38:54.8930994Z deleted: sha256:c0430973b7cf759075330ee29c79bfd1312c91e6033d9dd883b4b1c29743a171 2024-06-26T08:38:54.8931818Z deleted: sha256:e733b3d66efb9342e5a283376c0e1ddffa47a42fecc26e61ecaf0a3472885d97 2024-06-26T08:38:54.8932646Z deleted: sha256:8929c0b2047f74b84c82c39eff9077a92ae1a828cde00c9ab78fb3e0ed5e0283 2024-06-26T08:38:54.8933458Z deleted: sha256:f5b56155fb58a0a60fe6391d3581cf91610ed69d4f0a321257c19944da296a52 2024-06-26T08:38:54.8934262Z deleted: sha256:b54d2cb355238d1aa91e58df93064447b80f53706d1ca62823acc834f5ac54fe 2024-06-26T08:38:54.8935071Z deleted: sha256:e7311285af9eeddb51aa5629aa98386e89f1456ff32843b6aee6443247edf66c 2024-06-26T08:38:54.8935888Z deleted: sha256:67657ceec347da8dc8085d5731633efeac115ba677fa4b7c5966c4260f7abab0 2024-06-26T08:38:54.8936702Z deleted: sha256:b496e50409dbf414930fc9b4437aa0aa4ce27bc7840e050ae459aaffcf9aca86 2024-06-26T08:38:54.8937504Z deleted: sha256:8f5276170cc4a60d36ad7b964a502306b29546254cedcbcb101ea9123497d2c0 2024-06-26T08:38:54.8938299Z deleted: sha256:53d524d1564745e96a7e515462171b73184d9f1ff5454b32fcfbbca39d361467 2024-06-26T08:38:54.8939123Z deleted: sha256:2acbadb00d69b6aca575f04baba4ea26ae2b5c2d85f09945004ad93a78a8eb75 2024-06-26T08:38:54.8940040Z deleted: sha256:02665efeff878aeb603c1edb2e21cb700c171c025d64f5aae1b2b20b12f3964a 2024-06-26T08:38:54.8940879Z deleted: sha256:19bd23e9a51dfe883a8b8d2ff6d66ff1b3476b8912856406bb79b81762aa6d4b 2024-06-26T08:38:54.8941693Z deleted: sha256:73aa8637bf9299d6298dabd858480c274a62117388f2ca1d5e9eb15c6d1e4155 2024-06-26T08:38:54.8942499Z deleted: sha256:5f8968406305f348b1caebc0e035187d6e9965ace7b9d5f2f21114d37c9fe2e1 2024-06-26T08:38:54.8943313Z deleted: sha256:d827104914db0c503cf5c81f13e2456bf4efce42d4cf1f8d5d70838207278bee 2024-06-26T08:38:54.8944126Z deleted: sha256:867b219724f2f1aaf56f42e3e9f03a5d64bdca8a6a72407efd10328aeea5d510 2024-06-26T08:38:54.8944945Z deleted: sha256:b9ff2a7cceb9237b26bcb17728edf8a558eec5a76f63626e84f80541dfc0f1d4 2024-06-26T08:38:54.8945777Z deleted: sha256:be73f509106819007ce2b0f5a1301ac56ae583cb6c1fb69bee93efa0840368f1 2024-06-26T08:38:54.8946596Z deleted: sha256:2a11c07c915c3fdbb4e73096dea73ea950c970d50fe1a5764060bd62b7b7cf2c 2024-06-26T08:38:54.8947417Z deleted: sha256:5e9989622e2abc1c72f8184343507b45d75e4ff971edb68b96cbcd75ae484ff5 2024-06-26T08:38:54.8948459Z deleted: sha256:05ce642bd9cf5a9205bff107df28b1d6cef7d5f6ab3ea944d28c032ec61270df 2024-06-26T08:38:54.8949278Z deleted: sha256:017702617f6c07b73f2088678f4710b794cfe897a213a5cf122a0e2519eae0ac 2024-06-26T08:38:54.8950077Z deleted: sha256:44b3b37bb1ca39dc5cf356de41a23a5c12f605a1992813e408087fb2b0ad6d1c 2024-06-26T08:38:54.8950882Z deleted: sha256:8c387982e9f27db357a1d149dc149cabc0bd04668ad6956387a5072682dada67 2024-06-26T08:38:54.8951686Z deleted: sha256:10d415dca4e3fbb88a747b40317f3c6333c041ba8f73855f67fb9fe981fae5ee 2024-06-26T08:38:54.8952496Z deleted: sha256:6c65469bf0f53dcd625faefe363b8f3136c08388a6cf64d9158e8576e4ee89b1 2024-06-26T08:38:54.8953320Z deleted: sha256:027db5d8b76deeefbd8c34b485ab7ff14ed2a3e6800425c1ce0d77c2e9fdd4ab 2024-06-26T08:38:54.8954137Z deleted: sha256:24941234ad28cf5c8174d7cf18326c10b4e17f2c2137672ea0322df461597dd7 2024-06-26T08:38:54.8954948Z deleted: sha256:240ea2eebfa8519d6c2d88f4c99e66ce4e038d000f93ce8d03726c01d81180ef 2024-06-26T08:38:54.8955769Z deleted: sha256:09e823cb391a69f6a195e3ca8a41624b5f8b34eee14c6eb36277de814a91966e 2024-06-26T08:38:54.8956564Z deleted: sha256:730479d695169a6593bf0ac16b377491dce65175a6d158e012422367d04577fd 2024-06-26T08:38:54.8957361Z deleted: sha256:b0a30f9f7bd5459ff662453b9346a7ba8ea8db1191528345c1953eab4aa2f54e 2024-06-26T08:38:54.8958174Z deleted: sha256:75cefa87effaa3ad361c74642839923101f0f9982f814f3c9e2ed4ad52f2a008 2024-06-26T08:38:54.8959072Z deleted: sha256:6f97db87ba9512e615e871c6982000602982e1e5a5be334a56e7292251e84e00 2024-06-26T08:38:54.8959882Z deleted: sha256:35a033279d5c9bd6b862bfa9331d8fcbf98bbaf778a778fabc882384db9204f8 2024-06-26T08:38:54.8960699Z deleted: sha256:ab9b00c496073d62e69e032178858d3d6d4c4bab9e87065d3785c6da57351d00 2024-06-26T08:38:54.8961497Z deleted: sha256:7ed947299e109c2459de6e240d86f049c30f93f2380659ce441d8737b8cd065f 2024-06-26T08:38:54.8962298Z deleted: sha256:410d5a7f7a9a1cb4551c106b9cc728a9dcff598f9be231a351ce6a0a33f81e64 2024-06-26T08:38:54.8963122Z deleted: sha256:5f0021bb56efa14bb93978c01513e2a1187ba30e69bdb0546d0ef39b30873f88 2024-06-26T08:38:54.8963930Z deleted: sha256:c1601aa97eb84151c14da3aeea351201bd99144d36d66b397f6555d80245d86d 2024-06-26T08:38:54.8964753Z deleted: sha256:72af05a89be22accdc1ca5d66dcbbb33993a9ef5997f849df1d4ba4c48049f25 2024-06-26T08:38:54.8965696Z deleted: sha256:38b90c9663dcaa2bc57a4dd3008298e7ea93e9535fa42312b4dc4246e7491af9 2024-06-26T08:38:54.8966553Z deleted: sha256:bda61b6cefb3ec8eeb74fef1ca1c7f9a5845fe5e8f07b8123323e897425d5c29 2024-06-26T08:38:54.8967379Z deleted: sha256:5a18e1aa877074529a84cbddf19f8d5403787823378ceae6b72fb62f78d43037 2024-06-26T08:38:54.8968187Z deleted: sha256:6c3e7df31590f02f10cb71fc4eb27653e9b428df2e6e5421a455b062bd2e39f9 2024-06-26T08:38:54.8968670Z 2024-06-26T08:38:54.8968829Z Total reclaimed space: 36.35GB 2024-06-26T08:38:54.9031257Z Post job cleanup. 2024-06-26T08:38:54.9095610Z Post job cleanup. 2024-06-26T08:38:54.9916581Z [command]/usr/bin/git version 2024-06-26T08:38:54.9957495Z git version 2.40.1 2024-06-26T08:38:54.9997037Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/16216604-219f-42b0-b101-4f52528b80ed' before making global git config changes 2024-06-26T08:38:54.9998261Z Adding repository directory to the temporary git global config as a safe directory 2024-06-26T08:38:55.0002578Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2024-06-26T08:38:55.0036864Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2024-06-26T08:38:55.0060937Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2024-06-26T08:38:55.0270193Z Entering 'android/libs/fbjni' 2024-06-26T08:38:55.0305395Z Entering 'third_party/FP16' 2024-06-26T08:38:55.0342448Z Entering 'third_party/FXdiv' 2024-06-26T08:38:55.0379426Z Entering 'third_party/NNPACK' 2024-06-26T08:38:55.0415885Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:38:55.0449008Z Entering 'third_party/XNNPACK' 2024-06-26T08:38:55.0495678Z Entering 'third_party/benchmark' 2024-06-26T08:38:55.0530787Z Entering 'third_party/cpp-httplib' 2024-06-26T08:38:55.0566062Z Entering 'third_party/cpuinfo' 2024-06-26T08:38:55.0600196Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:38:55.0635083Z Entering 'third_party/cutlass' 2024-06-26T08:38:55.0676753Z Entering 'third_party/eigen' 2024-06-26T08:38:55.0713617Z Entering 'third_party/fbgemm' 2024-06-26T08:38:55.0749576Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:38:55.0783180Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:38:55.0818525Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:38:55.0858857Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:38:55.0892577Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:38:55.0928282Z Entering 'third_party/flatbuffers' 2024-06-26T08:38:55.0965307Z Entering 'third_party/fmt' 2024-06-26T08:38:55.0999472Z Entering 'third_party/foxi' 2024-06-26T08:38:55.1033656Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:38:55.1067179Z Entering 'third_party/gloo' 2024-06-26T08:38:55.1102362Z Entering 'third_party/googletest' 2024-06-26T08:38:55.1141855Z Entering 'third_party/ideep' 2024-06-26T08:38:55.1176053Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:38:55.1214907Z Entering 'third_party/ittapi' 2024-06-26T08:38:55.1251282Z Entering 'third_party/kineto' 2024-06-26T08:38:55.1287658Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:38:55.1320867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:38:55.1356357Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:38:55.1390996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:38:55.1423727Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:38:55.1456782Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:38:55.1492544Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:38:55.1526121Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:38:55.1561684Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:38:55.1596221Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:38:55.1631357Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:38:55.1664085Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:38:55.1700006Z Entering 'third_party/mimalloc' 2024-06-26T08:38:55.1734154Z Entering 'third_party/nccl/nccl' 2024-06-26T08:38:55.1768940Z Entering 'third_party/nlohmann' 2024-06-26T08:38:55.1806317Z Entering 'third_party/onnx' 2024-06-26T08:38:55.1855465Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:38:55.1889348Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:38:55.1926570Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:38:55.1964788Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:38:55.1999966Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:38:55.2032356Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:38:55.2064949Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:38:55.2099598Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:38:55.2133940Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:38:55.2169276Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:38:55.2204267Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:38:55.2241881Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:38:55.2278611Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:38:55.2329489Z Entering 'third_party/pocketfft' 2024-06-26T08:38:55.2364164Z Entering 'third_party/protobuf' 2024-06-26T08:38:55.2401053Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:38:55.2435873Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:38:55.2468678Z Entering 'third_party/psimd' 2024-06-26T08:38:55.2504377Z Entering 'third_party/pthreadpool' 2024-06-26T08:38:55.2540308Z Entering 'third_party/pybind11' 2024-06-26T08:38:55.2576692Z Entering 'third_party/python-peachpy' 2024-06-26T08:38:55.2610554Z Entering 'third_party/sleef' 2024-06-26T08:38:55.2645758Z Entering 'third_party/tensorpipe' 2024-06-26T08:38:55.2681152Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:38:55.2713592Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:38:55.2747989Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:38:55.2780818Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:38:55.2814011Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:38:55.2864833Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2024-06-26T08:38:55.2887079Z http.https://github.com/.extraheader 2024-06-26T08:38:55.2895922Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2024-06-26T08:38:55.2925386Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2024-06-26T08:38:55.3129015Z Entering 'android/libs/fbjni' 2024-06-26T08:38:55.3150252Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3174636Z Entering 'third_party/FP16' 2024-06-26T08:38:55.3196226Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3219417Z Entering 'third_party/FXdiv' 2024-06-26T08:38:55.3241495Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3264985Z Entering 'third_party/NNPACK' 2024-06-26T08:38:55.3286742Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3309991Z Entering 'third_party/VulkanMemoryAllocator' 2024-06-26T08:38:55.3330648Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3354546Z Entering 'third_party/XNNPACK' 2024-06-26T08:38:55.3377720Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3412043Z Entering 'third_party/benchmark' 2024-06-26T08:38:55.3434514Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3457299Z Entering 'third_party/cpp-httplib' 2024-06-26T08:38:55.3478985Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3502171Z Entering 'third_party/cpuinfo' 2024-06-26T08:38:55.3524197Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3549456Z Entering 'third_party/cudnn_frontend' 2024-06-26T08:38:55.3571351Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3595375Z Entering 'third_party/cutlass' 2024-06-26T08:38:55.3617029Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3645898Z Entering 'third_party/eigen' 2024-06-26T08:38:55.3666925Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3691337Z Entering 'third_party/fbgemm' 2024-06-26T08:38:55.3712872Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3735886Z Entering 'third_party/fbgemm/third_party/asmjit' 2024-06-26T08:38:55.3756300Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3779637Z Entering 'third_party/fbgemm/third_party/cpuinfo' 2024-06-26T08:38:55.3800687Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3825676Z Entering 'third_party/fbgemm/third_party/cutlass' 2024-06-26T08:38:55.3846699Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3873586Z Entering 'third_party/fbgemm/third_party/googletest' 2024-06-26T08:38:55.3894441Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3918170Z Entering 'third_party/fbgemm/third_party/hipify_torch' 2024-06-26T08:38:55.3939849Z http.https://github.com/.extraheader 2024-06-26T08:38:55.3963415Z Entering 'third_party/flatbuffers' 2024-06-26T08:38:55.3985991Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4011956Z Entering 'third_party/fmt' 2024-06-26T08:38:55.4033220Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4056374Z Entering 'third_party/foxi' 2024-06-26T08:38:55.4079232Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4102512Z Entering 'third_party/gemmlowp/gemmlowp' 2024-06-26T08:38:55.4126181Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4148954Z Entering 'third_party/gloo' 2024-06-26T08:38:55.4170545Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4193483Z Entering 'third_party/googletest' 2024-06-26T08:38:55.4215246Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4238165Z Entering 'third_party/ideep' 2024-06-26T08:38:55.4260288Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4282139Z Entering 'third_party/ideep/mkl-dnn' 2024-06-26T08:38:55.4303336Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4331808Z Entering 'third_party/ittapi' 2024-06-26T08:38:55.4353209Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4376403Z Entering 'third_party/kineto' 2024-06-26T08:38:55.4398215Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4420090Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2024-06-26T08:38:55.4441936Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4465671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2024-06-26T08:38:55.4487404Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4510366Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2024-06-26T08:38:55.4531308Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4553436Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2024-06-26T08:38:55.4573365Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4595612Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2024-06-26T08:38:55.4617984Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4640190Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2024-06-26T08:38:55.4661842Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4686791Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2024-06-26T08:38:55.4707178Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4731309Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2024-06-26T08:38:55.4751859Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4775916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2024-06-26T08:38:55.4799514Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4823753Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2024-06-26T08:38:55.4844820Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4870153Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2024-06-26T08:38:55.4891590Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4914806Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2024-06-26T08:38:55.4935368Z http.https://github.com/.extraheader 2024-06-26T08:38:55.4958027Z Entering 'third_party/mimalloc' 2024-06-26T08:38:55.4980767Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5004003Z Entering 'third_party/nccl/nccl' 2024-06-26T08:38:55.5026073Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5050263Z Entering 'third_party/nlohmann' 2024-06-26T08:38:55.5072267Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5094823Z Entering 'third_party/onnx' 2024-06-26T08:38:55.5115998Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5152083Z Entering 'third_party/onnx/third_party/benchmark' 2024-06-26T08:38:55.5173156Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5196184Z Entering 'third_party/onnx/third_party/pybind11' 2024-06-26T08:38:55.5217374Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5243066Z Entering 'third_party/opentelemetry-cpp' 2024-06-26T08:38:55.5264817Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5290231Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2024-06-26T08:38:55.5311837Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5334960Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2024-06-26T08:38:55.5356100Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5379614Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2024-06-26T08:38:55.5400716Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5423304Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2024-06-26T08:38:55.5444518Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5469226Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2024-06-26T08:38:55.5490532Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5513465Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2024-06-26T08:38:55.5534232Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5557056Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2024-06-26T08:38:55.5578051Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5599904Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2024-06-26T08:38:55.5621016Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5644765Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2024-06-26T08:38:55.5665591Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5690018Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2024-06-26T08:38:55.5711585Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5751149Z Entering 'third_party/pocketfft' 2024-06-26T08:38:55.5772835Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5795082Z Entering 'third_party/protobuf' 2024-06-26T08:38:55.5817652Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5843372Z Entering 'third_party/protobuf/third_party/benchmark' 2024-06-26T08:38:55.5864269Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5888106Z Entering 'third_party/protobuf/third_party/googletest' 2024-06-26T08:38:55.5909846Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5933044Z Entering 'third_party/psimd' 2024-06-26T08:38:55.5954349Z http.https://github.com/.extraheader 2024-06-26T08:38:55.5977037Z Entering 'third_party/pthreadpool' 2024-06-26T08:38:55.5999515Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6023379Z Entering 'third_party/pybind11' 2024-06-26T08:38:55.6045510Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6070420Z Entering 'third_party/python-peachpy' 2024-06-26T08:38:55.6092014Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6116047Z Entering 'third_party/sleef' 2024-06-26T08:38:55.6136780Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6159350Z Entering 'third_party/tensorpipe' 2024-06-26T08:38:55.6180322Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6202454Z Entering 'third_party/tensorpipe/third_party/googletest' 2024-06-26T08:38:55.6221715Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6243458Z Entering 'third_party/tensorpipe/third_party/libnop' 2024-06-26T08:38:55.6264176Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6287113Z Entering 'third_party/tensorpipe/third_party/libuv' 2024-06-26T08:38:55.6307974Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6331048Z Entering 'third_party/tensorpipe/third_party/pybind11' 2024-06-26T08:38:55.6352550Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6375359Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2024-06-26T08:38:55.6396659Z http.https://github.com/.extraheader 2024-06-26T08:38:55.6496268Z A job completed hook has been configured by the self-hosted runner administrator 2024-06-26T08:38:55.6513974Z ##[group]Run '/home/ec2-user/runner-scripts/cleanup.sh' 2024-06-26T08:38:55.6520623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2024-06-26T08:38:55.6521136Z ##[endgroup] 2024-06-26T08:38:56.2184052Z Cleaning up orphan processes