2025-04-21T21:42:05.6197399Z Current runner version: '2.323.0'
2025-04-21T21:42:05.6204015Z Runner name: 'i-0e08e74978c5f6915'
2025-04-21T21:42:05.6205040Z Machine name: 'ip-10-0-23-151'
2025-04-21T21:42:05.6209529Z ##[group]GITHUB_TOKEN Permissions
2025-04-21T21:42:05.6212184Z Actions: read
2025-04-21T21:42:05.6212734Z Attestations: read
2025-04-21T21:42:05.6213259Z Checks: read
2025-04-21T21:42:05.6213745Z Contents: read
2025-04-21T21:42:05.6214236Z Deployments: read
2025-04-21T21:42:05.6214745Z Discussions: read
2025-04-21T21:42:05.6215246Z Issues: read
2025-04-21T21:42:05.6215725Z Metadata: read
2025-04-21T21:42:05.6216227Z Models: read
2025-04-21T21:42:05.6216716Z Packages: read
2025-04-21T21:42:05.6217191Z Pages: read
2025-04-21T21:42:05.6217681Z PullRequests: read
2025-04-21T21:42:05.6218227Z RepositoryProjects: read
2025-04-21T21:42:05.6218801Z SecurityEvents: read
2025-04-21T21:42:05.6219336Z Statuses: read
2025-04-21T21:42:05.6219829Z ##[endgroup]
2025-04-21T21:42:05.6222586Z Secret source: Actions
2025-04-21T21:42:05.6223794Z Prepare workflow directory
2025-04-21T21:42:06.0954013Z Prepare all required actions
2025-04-21T21:42:06.0996678Z Getting action download info
2025-04-21T21:42:06.3229350Z Download action repository 'pytorch/test-infra@main' (SHA:d0e960f2e16f063377744672abd0cabc2c9a0167)
2025-04-21T21:42:07.9433883Z Download action repository 'pytorch/pytorch@main' (SHA:c312d8c5018f6c742fa9df7223fee74c71c4a68b)
2025-04-21T21:42:22.1198251Z Download action repository 'aws-actions/configure-aws-credentials@v4' (SHA:e3dd6a429d7300a6a4c196c26e071d42e0343502)
2025-04-21T21:42:22.3237846Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-04-21T21:42:22.5985766Z Getting action download info
2025-04-21T21:42:22.6995524Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2025-04-21T21:42:22.9193967Z Getting action download info
2025-04-21T21:42:23.0183442Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-04-21T21:42:23.2018224Z Getting action download info
2025-04-21T21:42:23.2830991Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2025-04-21T21:42:23.4356269Z Getting action download info
2025-04-21T21:42:23.5547041Z Complete job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:42:23.6092402Z A job started hook has been configured by the self-hosted runner administrator
2025-04-21T21:42:23.6188686Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh'
2025-04-21T21:42:23.6199653Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:23.6200241Z ##[endgroup]
2025-04-21T21:42:25.4228531Z Runner Type: linux.g5.4xlarge.nvidia.gpu
2025-04-21T21:42:25.4229019Z Instance Type: g5.4xlarge
2025-04-21T21:42:25.4229287Z AMI Name: unknown
2025-04-21T21:42:25.4268211Z AMI ID: ami-071226ecf16aa7d96
2025-04-21T21:42:30.7915884Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2025-04-21T21:42:30.7916297Z with:
2025-04-21T21:42:30.7916828Z   github-secret: ***
2025-04-21T21:42:30.7917506Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2025-04-21T21:42:30.7918233Z   activate-with-label: false
2025-04-21T21:42:30.7918498Z   label: with-ssh
2025-04-21T21:42:30.7918742Z   remove-existing-keys: true
2025-04-21T21:42:30.7919029Z   fail-silently: true
2025-04-21T21:42:30.7919290Z env:
2025-04-21T21:42:30.7919513Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:30.7919772Z ##[endgroup]
2025-04-21T21:42:30.9140581Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info.
2025-04-21T21:42:30.9141898Z ciflow reference detected, attempting to extract PR number
2025-04-21T21:42:31.3022454Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2025-04-21T21:42:31.3880422Z No SSH keys found for user pytorch-bot[bot]
2025-04-21T21:42:31.3880897Z Grabbing public ssh keys from https://github.com/laithsakka.keys
2025-04-21T21:42:31.4526759Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2025-04-21T21:42:31.4542105Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2025-04-21T21:42:31.4578230Z Login using: ssh ec2-user@ec2-3-222-116-14.compute-1.amazonaws.com
2025-04-21T21:42:31.4578785Z All testing is done inside the container, to start an interactive session run:
2025-04-21T21:42:31.4579316Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2025-04-21T21:42:31.4745745Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-04-21T21:42:31.4746178Z with:
2025-04-21T21:42:31.4746398Z   no-sudo: true
2025-04-21T21:42:31.4746630Z   submodules: recursive
2025-04-21T21:42:31.4746900Z   fetch-depth: 0
2025-04-21T21:42:31.4747120Z env:
2025-04-21T21:42:31.4747326Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:31.4747609Z ##[endgroup]
2025-04-21T21:42:31.4888351Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:42:31.4889336Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:42:31.4900803Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:31.4901181Z env:
2025-04-21T21:42:31.4901406Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:31.4901674Z ##[endgroup]
2025-04-21T21:42:31.4995922Z ##[group]Run # Use all available CPUs for fetching
2025-04-21T21:42:31.4996366Z # Use all available CPUs for fetching
2025-04-21T21:42:31.4996706Z cd "${GITHUB_WORKSPACE}"
2025-04-21T21:42:31.4997055Z git config --global fetch.parallel 0
2025-04-21T21:42:31.4997451Z git config --global submodule.fetchJobs 0
2025-04-21T21:42:31.4997800Z 
2025-04-21T21:42:31.4998147Z # Clean workspace. The default checkout action should also do this, but
2025-04-21T21:42:31.4998636Z # do it here as well just in case
2025-04-21T21:42:31.4999024Z if [[ -d .git ]]; then
2025-04-21T21:42:31.4999325Z   if [ -z "${NO_SUDO}" ]; then
2025-04-21T21:42:31.4999637Z     sudo git clean -ffdx
2025-04-21T21:42:31.4999919Z   else
2025-04-21T21:42:31.5000245Z     git clean -ffdx
2025-04-21T21:42:31.5000529Z   fi
2025-04-21T21:42:31.5000762Z fi
2025-04-21T21:42:31.5009497Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:31.5009868Z env:
2025-04-21T21:42:31.5010101Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:31.5010373Z   NO_SUDO: true
2025-04-21T21:42:31.5010595Z ##[endgroup]
2025-04-21T21:42:31.8226854Z Removing .additional_ci_files/
2025-04-21T21:42:31.8227184Z Removing .pytest_cache/
2025-04-21T21:42:31.8227630Z Removing benchmarks/functional_autograd_benchmark/__pycache__/
2025-04-21T21:42:31.8228145Z Removing build/
2025-04-21T21:42:31.8228477Z Removing dist/
2025-04-21T21:42:31.8228965Z Removing logs-test-slow-3-3-linux.g5.4xlarge.nvidia.gpu_40896347932.zip
2025-04-21T21:42:31.8229740Z Removing test-jsons-test-slow-3-3-linux.g5.4xlarge.nvidia.gpu_40896347932.zip
2025-04-21T21:42:31.8230397Z Removing test-reports-test-slow-3-3-linux.g5.4xlarge.nvidia.gpu_40896347932.zip
2025-04-21T21:42:31.8230928Z Removing test/.pytorch-disabled-tests.json
2025-04-21T21:42:31.8231268Z Removing test/__pycache__/
2025-04-21T21:42:31.8231574Z Removing test/ao/sparsity/__pycache__/
2025-04-21T21:42:31.8231931Z Removing test/benchmark_utils/__pycache__/
2025-04-21T21:42:31.8232268Z Removing test/dynamo/__pycache__/
2025-04-21T21:42:31.8232591Z Removing test/export/__pycache__/
2025-04-21T21:42:31.8232912Z Removing test/functorch/__pycache__/
2025-04-21T21:42:31.8233311Z Removing test/inductor/__pycache__/
2025-04-21T21:42:31.8233696Z Removing test/inductor/extension_backends/cpp/__pycache__/
2025-04-21T21:42:31.8234430Z Removing test/inductor/extension_backends/triton/__pycache__/
2025-04-21T21:42:31.8234847Z Removing test/jit/__pycache__/
2025-04-21T21:42:31.8235168Z Removing test/optim/__pycache__/
2025-04-21T21:42:31.8235491Z Removing test/quantization/__pycache__/
2025-04-21T21:42:31.8235875Z Removing test/quantization/ao_migration/__pycache__/
2025-04-21T21:42:31.8236262Z Removing test/quantization/bc/__pycache__/
2025-04-21T21:42:31.8236625Z Removing test/quantization/core/__pycache__/
2025-04-21T21:42:31.8237045Z Removing test/quantization/core/experimental/__pycache__/
2025-04-21T21:42:31.8237657Z Removing test/quantization/eager/__pycache__/
2025-04-21T21:42:31.8238021Z Removing test/quantization/fx/__pycache__/
2025-04-21T21:42:31.8238376Z Removing test/quantization/jit/__pycache__/
2025-04-21T21:42:31.8238748Z Removing test/quantization/pt2e/__pycache__/
2025-04-21T21:42:31.8239091Z Removing test/test-reports/
2025-04-21T21:42:31.8239478Z Removing test/torch_np/__pycache__/
2025-04-21T21:42:31.8239803Z Removing tools/__pycache__/
2025-04-21T21:42:31.8240090Z Removing tools/stats/__pycache__/
2025-04-21T21:42:31.8240471Z Removing tools/stats/upload_utilization_stats/__pycache__/
2025-04-21T21:42:31.8240876Z Removing tools/testing/__pycache__/
2025-04-21T21:42:31.8241258Z Removing tools/testing/target_determination/__pycache__/
2025-04-21T21:42:31.8241752Z Removing tools/testing/target_determination/heuristics/__pycache__/
2025-04-21T21:42:31.8242184Z Removing usage_log.txt
2025-04-21T21:42:31.8309580Z ##[group]Run actions/checkout@v4
2025-04-21T21:42:31.8309864Z with:
2025-04-21T21:42:31.8310117Z   ref: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:42:31.8310444Z   fetch-depth: 0
2025-04-21T21:42:31.8310684Z   submodules: recursive
2025-04-21T21:42:31.8310935Z   show-progress: false
2025-04-21T21:42:31.8311197Z   repository: pytorch/pytorch
2025-04-21T21:42:31.8311604Z   token: ***
2025-04-21T21:42:31.8311824Z   ssh-strict: true
2025-04-21T21:42:31.8312054Z   ssh-user: git
2025-04-21T21:42:31.8312296Z   persist-credentials: true
2025-04-21T21:42:31.8312562Z   clean: true
2025-04-21T21:42:31.8312801Z   sparse-checkout-cone-mode: true
2025-04-21T21:42:31.8313093Z   fetch-tags: false
2025-04-21T21:42:31.8313462Z   lfs: false
2025-04-21T21:42:31.8313687Z   set-safe-directory: true
2025-04-21T21:42:31.8313945Z env:
2025-04-21T21:42:31.8314160Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:31.8314407Z ##[endgroup]
2025-04-21T21:42:31.9447531Z Syncing repository: pytorch/pytorch
2025-04-21T21:42:31.9448695Z ##[group]Getting Git version info
2025-04-21T21:42:31.9449171Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2025-04-21T21:42:31.9449812Z [command]/usr/bin/git version
2025-04-21T21:42:31.9450108Z git version 2.47.1
2025-04-21T21:42:31.9473118Z ##[endgroup]
2025-04-21T21:42:31.9483366Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/b926d891-4120-48ac-9b2f-e3e48d26173d/.gitconfig'
2025-04-21T21:42:31.9503770Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/b926d891-4120-48ac-9b2f-e3e48d26173d' before making global git config changes
2025-04-21T21:42:31.9504679Z Adding repository directory to the temporary git global config as a safe directory
2025-04-21T21:42:31.9508601Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-04-21T21:42:31.9562180Z [command]/usr/bin/git config --local --get remote.origin.url
2025-04-21T21:42:31.9586474Z https://github.com/pytorch/pytorch
2025-04-21T21:42:31.9605018Z ##[group]Removing previously created refs, to avoid conflicts
2025-04-21T21:42:31.9608659Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-04-21T21:42:31.9634751Z HEAD
2025-04-21T21:42:31.9677380Z ##[endgroup]
2025-04-21T21:42:31.9680554Z [command]/usr/bin/git submodule status
2025-04-21T21:42:32.0086783Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-04-21T21:42:32.0198379Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-04-21T21:42:32.0295233Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-04-21T21:42:32.0412463Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-04-21T21:42:32.0442440Z  e170594ac7cf1dac584da473d4ca9301087090c1 third_party/NVTX (v3.1.0)
2025-04-21T21:42:32.0532140Z  a6bfc237255a6bac1513f7c1ebde6d8aed6b5191 third_party/VulkanMemoryAllocator (v2.1.0-705-ga6bfc23)
2025-04-21T21:42:32.1120734Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a010365)
2025-04-21T21:42:32.1152146Z  0d98dba29d66e93259db7daa53a9327df767a415 third_party/benchmark (v1.6.1)
2025-04-21T21:42:32.1186825Z  8086bbe3a78d931eb96fe12fdc014082e18d18d3 third_party/composable_kernel (rocm-6.4.0-6-g8086bbe3a)
2025-04-21T21:42:32.1350711Z  3b6597bba913d51161383657829b7e644e59c006 third_party/cpp-httplib (v0.15.3-20-g3b6597b)
2025-04-21T21:42:32.1510645Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-04-21T21:42:32.1555373Z  8801fd7b31c2f798732ed1e1dd4711a9cde4217d third_party/cudnn_frontend (v0.5-46-g8801fd7)
2025-04-21T21:42:32.1662404Z  afa1772203677c5118fcd82537a9c8fefbcc7008 third_party/cutlass (v3.8.0)
2025-04-21T21:42:32.2500878Z  3147391d946bb4b6c68edd901f2add6ac1f31f8c third_party/eigen (3.4.0)
2025-04-21T21:42:32.2902292Z  dbc3157bf256f1339b3fa1fef2be89ac4078be0e third_party/fbgemm (v0.4.1-446-gdbc3157b)
2025-04-21T21:42:32.3004623Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-04-21T21:42:32.3036415Z  01834de25e4bf3975a9a00e816292b1ad0fe184b third_party/flatbuffers (v23.3.3)
2025-04-21T21:42:32.3543511Z  123913715afeb8a437e6388b4473fcc4753e1c9a third_party/fmt (11.1.4)
2025-04-21T21:42:32.3686666Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-04-21T21:42:32.3787011Z  c61070427610ccd923efe3e7f8b3eca12bbcc31a third_party/gloo (heads/main)
2025-04-21T21:42:32.4074566Z  b514bdc898e2951020cbdca1304b75f5950d1f59 third_party/googletest (release-1.8.0-3484-gb514bdc8)
2025-04-21T21:42:32.4170838Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-04-21T21:42:32.4245706Z  5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42 third_party/ittapi (v3.23.0-14-g5b8a7d7)
2025-04-21T21:42:32.4558304Z  5fa4bd8c8fb581a621f72957d0cb11431cc4714f third_party/kineto (remotes/origin/sraikund/test-64-g5fa4bd8)
2025-04-21T21:42:32.4591120Z  ef685a13cfbe8d418aa2ed34350e21e4938358b6 third_party/kleidiai (v1.3.0)
2025-04-21T21:42:32.4620256Z  b66e3214d8a104669c2ec05ae91ebc26a8f5ab78 third_party/mimalloc (v1.8.2)
2025-04-21T21:42:32.5181205Z  87cda1d6646592ac5866dc703c8e1839046a6806 third_party/nlohmann (v3.10.1-113-g87cda1d6)
2025-04-21T21:42:32.5516637Z  b8baa8446686496da4cc8fda09f2b6fe65c2a02c third_party/onnx (v1.17.0)
2025-04-21T21:42:32.5549503Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-04-21T21:42:32.5582580Z  9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe third_party/pocketfft (release_for_eigen-11-g9d3ab05)
2025-04-21T21:42:32.6033438Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-04-21T21:42:32.6129013Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-04-21T21:42:32.6202122Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-04-21T21:42:32.6261888Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-04-21T21:42:32.6356054Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-04-21T21:42:32.6495350Z  56e1f79cb140fb9326d612d0be06b5250565cade third_party/sleef (3.7-33-g56e1f79)
2025-04-21T21:42:32.6587237Z  52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master)
2025-04-21T21:42:32.6608200Z ##[group]Cleaning the repository
2025-04-21T21:42:32.6612277Z [command]/usr/bin/git clean -ffdx
2025-04-21T21:42:32.6843048Z [command]/usr/bin/git reset --hard HEAD
2025-04-21T21:42:33.7462545Z Updating files:  54% (8997/16510)
2025-04-21T21:42:33.7553818Z Updating files:  55% (9081/16510)
2025-04-21T21:42:33.7640654Z Updating files:  56% (9246/16510)
2025-04-21T21:42:33.7727951Z Updating files:  57% (9411/16510)
2025-04-21T21:42:33.7815345Z Updating files:  58% (9576/16510)
2025-04-21T21:42:33.7902854Z Updating files:  59% (9741/16510)
2025-04-21T21:42:33.7990362Z Updating files:  60% (9906/16510)
2025-04-21T21:42:33.8078011Z Updating files:  61% (10072/16510)
2025-04-21T21:42:33.8165443Z Updating files:  62% (10237/16510)
2025-04-21T21:42:33.8256304Z Updating files:  63% (10402/16510)
2025-04-21T21:42:33.8341266Z Updating files:  64% (10567/16510)
2025-04-21T21:42:33.8443164Z Updating files:  65% (10732/16510)
2025-04-21T21:42:33.8714178Z Updating files:  66% (10897/16510)
2025-04-21T21:42:33.8963917Z Updating files:  67% (11062/16510)
2025-04-21T21:42:33.9038595Z Updating files:  68% (11227/16510)
2025-04-21T21:42:33.9121210Z Updating files:  69% (11392/16510)
2025-04-21T21:42:33.9329757Z Updating files:  70% (11557/16510)
2025-04-21T21:42:33.9665532Z Updating files:  71% (11723/16510)
2025-04-21T21:42:33.9966531Z Updating files:  72% (11888/16510)
2025-04-21T21:42:34.0453568Z Updating files:  73% (12053/16510)
2025-04-21T21:42:34.0656016Z Updating files:  74% (12218/16510)
2025-04-21T21:42:34.0801715Z Updating files:  75% (12383/16510)
2025-04-21T21:42:34.1088311Z Updating files:  76% (12548/16510)
2025-04-21T21:42:34.1318052Z Updating files:  77% (12713/16510)
2025-04-21T21:42:34.1653661Z Updating files:  78% (12878/16510)
2025-04-21T21:42:34.1941121Z Updating files:  79% (13043/16510)
2025-04-21T21:42:34.2231110Z Updating files:  80% (13208/16510)
2025-04-21T21:42:34.2423104Z Updating files:  81% (13374/16510)
2025-04-21T21:42:34.2575618Z Updating files:  82% (13539/16510)
2025-04-21T21:42:34.2780403Z Updating files:  83% (13704/16510)
2025-04-21T21:42:34.2988314Z Updating files:  84% (13869/16510)
2025-04-21T21:42:34.3170230Z Updating files:  85% (14034/16510)
2025-04-21T21:42:34.3373997Z Updating files:  86% (14199/16510)
2025-04-21T21:42:34.3542971Z Updating files:  87% (14364/16510)
2025-04-21T21:42:34.3731721Z Updating files:  88% (14529/16510)
2025-04-21T21:42:34.3969789Z Updating files:  89% (14694/16510)
2025-04-21T21:42:34.4165117Z Updating files:  90% (14859/16510)
2025-04-21T21:42:34.4342342Z Updating files:  91% (15025/16510)
2025-04-21T21:42:34.4525958Z Updating files:  92% (15190/16510)
2025-04-21T21:42:34.4770754Z Updating files:  93% (15355/16510)
2025-04-21T21:42:34.5022727Z Updating files:  94% (15520/16510)
2025-04-21T21:42:34.5235861Z Updating files:  95% (15685/16510)
2025-04-21T21:42:34.5458843Z Updating files:  96% (15850/16510)
2025-04-21T21:42:34.5799145Z Updating files:  97% (16015/16510)
2025-04-21T21:42:34.6006971Z Updating files:  98% (16180/16510)
2025-04-21T21:42:34.6207348Z Updating files:  99% (16345/16510)
2025-04-21T21:42:34.6208576Z Updating files: 100% (16510/16510)
2025-04-21T21:42:34.6209371Z Updating files: 100% (16510/16510), done.
2025-04-21T21:42:34.6387445Z HEAD is now at b7c70007280 Ensure runners have the required prefix (#151815)
2025-04-21T21:42:34.6556345Z ##[endgroup]
2025-04-21T21:42:34.6558276Z ##[group]Disabling automatic garbage collection
2025-04-21T21:42:34.6563988Z [command]/usr/bin/git config --local gc.auto 0
2025-04-21T21:42:34.6599789Z ##[endgroup]
2025-04-21T21:42:34.6600213Z ##[group]Setting up auth
2025-04-21T21:42:34.6605975Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-04-21T21:42:34.6650139Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-04-21T21:42:34.7012373Z Entering 'android/libs/fbjni'
2025-04-21T21:42:34.7081871Z Entering 'third_party/FP16'
2025-04-21T21:42:34.7152182Z Entering 'third_party/FXdiv'
2025-04-21T21:42:34.7220892Z Entering 'third_party/NNPACK'
2025-04-21T21:42:34.7290248Z Entering 'third_party/NVTX'
2025-04-21T21:42:34.7360640Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:34.7431697Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:34.7515174Z Entering 'third_party/benchmark'
2025-04-21T21:42:34.7585104Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:34.7661305Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:34.7730462Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:34.7798942Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:34.7868064Z Entering 'third_party/cutlass'
2025-04-21T21:42:34.7945979Z Entering 'third_party/eigen'
2025-04-21T21:42:34.8017584Z Entering 'third_party/fbgemm'
2025-04-21T21:42:34.8088969Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:34.8155693Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:34.8221855Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:34.8296342Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:34.8368100Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:34.8438701Z Entering 'third_party/flash-attention'
2025-04-21T21:42:34.8508161Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:34.8583583Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:34.8661965Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:34.8735465Z Entering 'third_party/fmt'
2025-04-21T21:42:34.8804634Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:34.8874463Z Entering 'third_party/gloo'
2025-04-21T21:42:34.8942493Z Entering 'third_party/googletest'
2025-04-21T21:42:34.9011067Z Entering 'third_party/ideep'
2025-04-21T21:42:34.9078029Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:34.9154424Z Entering 'third_party/ittapi'
2025-04-21T21:42:34.9223024Z Entering 'third_party/kineto'
2025-04-21T21:42:34.9291710Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:34.9358170Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:34.9428795Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:34.9497217Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:34.9564425Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:34.9630250Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:34.9702148Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:34.9770064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:34.9838726Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:34.9908167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:34.9979000Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:35.0044912Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:35.0121114Z Entering 'third_party/kleidiai'
2025-04-21T21:42:35.0190822Z Entering 'third_party/mimalloc'
2025-04-21T21:42:35.0261539Z Entering 'third_party/nlohmann'
2025-04-21T21:42:35.0332399Z Entering 'third_party/onnx'
2025-04-21T21:42:35.0415601Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:35.0488565Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:35.0558403Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:35.0628547Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:35.0698238Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:35.0764163Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:35.0833759Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:35.0900733Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:35.0966429Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:35.1035623Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:35.1103761Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:35.1181227Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:35.1273964Z Entering 'third_party/pocketfft'
2025-04-21T21:42:35.1343470Z Entering 'third_party/protobuf'
2025-04-21T21:42:35.1415058Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:35.1484120Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:35.1554310Z Entering 'third_party/psimd'
2025-04-21T21:42:35.1623469Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:35.1692395Z Entering 'third_party/pybind11'
2025-04-21T21:42:35.1763254Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:35.1832294Z Entering 'third_party/sleef'
2025-04-21T21:42:35.1901174Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:35.1968783Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:35.2035814Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:35.2101968Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:35.2169164Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:35.2237086Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:35.2334014Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-04-21T21:42:35.2366276Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-04-21T21:42:35.2727775Z Entering 'android/libs/fbjni'
2025-04-21T21:42:35.2796331Z Entering 'third_party/FP16'
2025-04-21T21:42:35.2866209Z Entering 'third_party/FXdiv'
2025-04-21T21:42:35.2934765Z Entering 'third_party/NNPACK'
2025-04-21T21:42:35.3002617Z Entering 'third_party/NVTX'
2025-04-21T21:42:35.3071933Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:35.3140030Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:35.3224121Z Entering 'third_party/benchmark'
2025-04-21T21:42:35.3291835Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:35.3366036Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:35.3434081Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:35.3506541Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:35.3577137Z Entering 'third_party/cutlass'
2025-04-21T21:42:35.3658531Z Entering 'third_party/eigen'
2025-04-21T21:42:35.3733486Z Entering 'third_party/fbgemm'
2025-04-21T21:42:35.3801988Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:35.3868477Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:35.3934718Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:35.4006681Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:35.4074858Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:35.4144269Z Entering 'third_party/flash-attention'
2025-04-21T21:42:35.4211117Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:35.4286119Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:35.4365468Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:35.4436249Z Entering 'third_party/fmt'
2025-04-21T21:42:35.4508879Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:35.4582057Z Entering 'third_party/gloo'
2025-04-21T21:42:35.4657753Z Entering 'third_party/googletest'
2025-04-21T21:42:35.4727765Z Entering 'third_party/ideep'
2025-04-21T21:42:35.4793443Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:35.4870274Z Entering 'third_party/ittapi'
2025-04-21T21:42:35.4939761Z Entering 'third_party/kineto'
2025-04-21T21:42:35.5008713Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:35.5074398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:35.5148964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:35.5216685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:35.5289057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:35.5354872Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:35.5431195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:35.5499633Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:35.5568821Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:35.5638562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:35.5711659Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:35.5780358Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:35.5851287Z Entering 'third_party/kleidiai'
2025-04-21T21:42:35.5924809Z Entering 'third_party/mimalloc'
2025-04-21T21:42:35.5994029Z Entering 'third_party/nlohmann'
2025-04-21T21:42:35.6068436Z Entering 'third_party/onnx'
2025-04-21T21:42:35.6150170Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:35.6226703Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:35.6294489Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:35.6362170Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:35.6431453Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:35.6498212Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:35.6565553Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:35.6633442Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:35.6700146Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:35.6768700Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:35.6839571Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:35.6910908Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:35.7000061Z Entering 'third_party/pocketfft'
2025-04-21T21:42:35.7068741Z Entering 'third_party/protobuf'
2025-04-21T21:42:35.7138635Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:35.7204919Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:35.7277332Z Entering 'third_party/psimd'
2025-04-21T21:42:35.7344875Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:35.7416855Z Entering 'third_party/pybind11'
2025-04-21T21:42:35.7486347Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:35.7554587Z Entering 'third_party/sleef'
2025-04-21T21:42:35.7628084Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:35.7694965Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:35.7760489Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:35.7827502Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:35.7894284Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:35.7958678Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:35.8059721Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-04-21T21:42:35.8111636Z ##[endgroup]
2025-04-21T21:42:35.8112058Z ##[group]Fetching the repository
2025-04-21T21:42:35.8119240Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-04-21T21:42:39.4861282Z From https://github.com/pytorch/pytorch
2025-04-21T21:42:39.4862149Z  - [deleted]                 (none)                  -> ciflow/inductor/149690
2025-04-21T21:42:39.5561958Z  - [deleted]                 (none)                  -> ciflow/inductor/149711
2025-04-21T21:42:39.5562437Z  - [deleted]                 (none)                  -> ciflow/trunk/149711
2025-04-21T21:42:39.5563527Z  - [deleted]                 (none)                  -> ciflow/trunk/151046
2025-04-21T21:42:39.5564672Z  - [deleted]                 (none)                  -> ciflow/trunk/151716
2025-04-21T21:42:39.5565876Z  - [deleted]                 (none)                  -> ciflow/trunk/151794
2025-04-21T21:42:40.6898067Z  * [new branch]              eqy-patch-6             -> origin/eqy-patch-6
2025-04-21T21:42:40.6938750Z    b95d854985a..3e5aca84441  gh/laithsakka/119/base  -> origin/gh/laithsakka/119/base
2025-04-21T21:42:40.6945478Z    2e26d078915..b68c0effe97  gh/laithsakka/119/head  -> origin/gh/laithsakka/119/head
2025-04-21T21:42:40.6952690Z  + 19dc41198c9...55d0c158068 gh/laithsakka/119/orig  -> origin/gh/laithsakka/119/orig  (forced update)
2025-04-21T21:42:40.6962108Z    eaba5732110..a9a371e2cbb  gh/laithsakka/143/base  -> origin/gh/laithsakka/143/base
2025-04-21T21:42:40.6970028Z    12685772308..e408d0201d5  gh/laithsakka/143/head  -> origin/gh/laithsakka/143/head
2025-04-21T21:42:40.6978037Z  + 5a4cbcd8a74...e3fa1be9109 gh/laithsakka/143/orig  -> origin/gh/laithsakka/143/orig  (forced update)
2025-04-21T21:42:40.7004521Z    bbd593720e2..0de0a192d66  gh/laithsakka/144/base  -> origin/gh/laithsakka/144/base
2025-04-21T21:42:40.7011865Z    8b8f4016ba3..90c985a64c3  gh/laithsakka/144/head  -> origin/gh/laithsakka/144/head
2025-04-21T21:42:40.7019831Z  + 5cae6c1dc5e...c76ceea1d31 gh/laithsakka/144/orig  -> origin/gh/laithsakka/144/orig  (forced update)
2025-04-21T21:42:40.7027665Z    8b8f4016ba3..91110def275  gh/laithsakka/145/base  -> origin/gh/laithsakka/145/base
2025-04-21T21:42:40.7036191Z    61c653a8a41..79bd94ff19f  gh/laithsakka/145/head  -> origin/gh/laithsakka/145/head
2025-04-21T21:42:40.7043938Z  + 1c79f4f257d...cf312ce1ac8 gh/laithsakka/145/orig  -> origin/gh/laithsakka/145/orig  (forced update)
2025-04-21T21:42:40.7058198Z    5c603ac1ba8..f71a3d90cd1  gh/mlazos/29/base       -> origin/gh/mlazos/29/base
2025-04-21T21:42:40.7065832Z    d5699eb0ca4..ba361774d76  gh/mlazos/29/head       -> origin/gh/mlazos/29/head
2025-04-21T21:42:40.7073494Z  + ceacf9edd65...7463b297ab7 gh/mlazos/29/orig       -> origin/gh/mlazos/29/orig  (forced update)
2025-04-21T21:42:40.7081296Z    fea3e205db8..f0051bf4cf3  gh/mlazos/30/base       -> origin/gh/mlazos/30/base
2025-04-21T21:42:40.7089700Z    b367c377e05..36d13fd7746  gh/mlazos/30/head       -> origin/gh/mlazos/30/head
2025-04-21T21:42:40.7098481Z  + 16f0eb41fb0...386d8af2dab gh/mlazos/30/orig       -> origin/gh/mlazos/30/orig  (forced update)
2025-04-21T21:42:40.7107313Z    4a9a383ded5..7423b341c9f  gh/mlazos/31/base       -> origin/gh/mlazos/31/base
2025-04-21T21:42:40.7115264Z    9dd543831cf..c2cf8791922  gh/mlazos/31/head       -> origin/gh/mlazos/31/head
2025-04-21T21:42:40.7123581Z  + 0ca0b7d756f...71abc0a02df gh/mlazos/31/orig       -> origin/gh/mlazos/31/orig  (forced update)
2025-04-21T21:42:40.7137909Z    edd39a0d4ee..4447800f753  gh/syed-ahmed/1/head    -> origin/gh/syed-ahmed/1/head
2025-04-21T21:42:40.7145444Z  + 469a4dc35e1...0f317493c86 gh/syed-ahmed/1/orig    -> origin/gh/syed-ahmed/1/orig  (forced update)
2025-04-21T21:42:40.7163040Z    1f0d764b65d..c312d8c5018  main                    -> origin/main
2025-04-21T21:42:40.7171962Z  + 0ca0b7d756f...71abc0a02df mlazos/hc-fixes4        -> origin/mlazos/hc-fixes4  (forced update)
2025-04-21T21:42:40.7175983Z  * [new branch]              pianpwk/01_auto_warning -> origin/pianpwk/01_auto_warning
2025-04-21T21:42:40.7178219Z  * [new branch]              pianpwk/dynamo_export_ctx -> origin/pianpwk/dynamo_export_ctx
2025-04-21T21:42:40.7188819Z    2eacdb91c3d..33808f0ebdc  viable/strict           -> origin/viable/strict
2025-04-21T21:42:40.7196144Z  t [tag update]              ciflow/inductor/148893  -> ciflow/inductor/148893
2025-04-21T21:42:40.7199904Z  t [tag update]              ciflow/inductor/151023  -> ciflow/inductor/151023
2025-04-21T21:42:40.7201261Z  t [tag update]              ciflow/inductor/151038  -> ciflow/inductor/151038
2025-04-21T21:42:40.7202908Z  t [tag update]              ciflow/inductor/151041  -> ciflow/inductor/151041
2025-04-21T21:42:40.7205406Z  t [tag update]              ciflow/inductor/151293  -> ciflow/inductor/151293
2025-04-21T21:42:40.7206736Z  t [tag update]              ciflow/inductor/151294  -> ciflow/inductor/151294
2025-04-21T21:42:40.7208665Z  t [tag update]              ciflow/inductor/151295  -> ciflow/inductor/151295
2025-04-21T21:42:40.7211701Z  * [new tag]                 ciflow/inductor/151825  -> ciflow/inductor/151825
2025-04-21T21:42:40.7213094Z  * [new tag]                 ciflow/inductor/151828  -> ciflow/inductor/151828
2025-04-21T21:42:40.7215030Z  * [new tag]                 ciflow/inductor/151833  -> ciflow/inductor/151833
2025-04-21T21:42:40.7223152Z  t [tag update]              ciflow/mps/147644       -> ciflow/mps/147644
2025-04-21T21:42:40.7228024Z  * [new tag]                 ciflow/trunk/147476     -> ciflow/trunk/147476
2025-04-21T21:42:40.7230401Z  * [new tag]                 ciflow/trunk/150897     -> ciflow/trunk/150897
2025-04-21T21:42:40.7231758Z  t [tag update]              ciflow/trunk/151023     -> ciflow/trunk/151023
2025-04-21T21:42:40.7233487Z  t [tag update]              ciflow/trunk/151253     -> ciflow/trunk/151253
2025-04-21T21:42:40.7234913Z  t [tag update]              ciflow/trunk/151295     -> ciflow/trunk/151295
2025-04-21T21:42:40.7236593Z  * [new tag]                 ciflow/trunk/151701     -> ciflow/trunk/151701
2025-04-21T21:42:40.7238024Z  * [new tag]                 ciflow/trunk/151795     -> ciflow/trunk/151795
2025-04-21T21:42:40.7239131Z  * [new tag]                 ciflow/trunk/151797     -> ciflow/trunk/151797
2025-04-21T21:42:40.7240775Z  t [tag update]              ciflow/trunk/151798     -> ciflow/trunk/151798
2025-04-21T21:42:40.7241916Z  * [new tag]                 ciflow/trunk/151816     -> ciflow/trunk/151816
2025-04-21T21:42:40.8068670Z [command]/usr/bin/git rev-parse --verify --quiet b68c0effe97351b0c24b2f7aef671559be6aa977^{object}
2025-04-21T21:42:40.8098469Z b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:42:40.8104679Z ##[endgroup]
2025-04-21T21:42:40.8105102Z ##[group]Determining the checkout info
2025-04-21T21:42:40.8105562Z ##[endgroup]
2025-04-21T21:42:40.8109989Z [command]/usr/bin/git sparse-checkout disable
2025-04-21T21:42:41.0628893Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-04-21T21:42:41.0663024Z ##[group]Checking out the ref
2025-04-21T21:42:41.0666874Z [command]/usr/bin/git checkout --progress --force b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:42:41.2309724Z Previous HEAD position was b7c70007280 Ensure runners have the required prefix (#151815)
2025-04-21T21:42:41.2319091Z HEAD is now at b68c0effe97 Update on "use statically known true instead of guard size oblivious in bmm and mm decompositions . "
2025-04-21T21:42:41.2376516Z ##[endgroup]
2025-04-21T21:42:41.2377268Z ##[group]Setting up auth for fetching submodules
2025-04-21T21:42:41.2384389Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-04-21T21:42:41.2437618Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-04-21T21:42:41.2471381Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-04-21T21:42:41.2512832Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-04-21T21:42:41.2543690Z ##[endgroup]
2025-04-21T21:42:41.2544470Z ##[group]Fetching submodules
2025-04-21T21:42:41.2548127Z [command]/usr/bin/git submodule sync --recursive
2025-04-21T21:42:41.2917942Z Synchronizing submodule url for 'android/libs/fbjni'
2025-04-21T21:42:41.2945924Z Synchronizing submodule url for 'third_party/FP16'
2025-04-21T21:42:41.2972790Z Synchronizing submodule url for 'third_party/FXdiv'
2025-04-21T21:42:41.2998826Z Synchronizing submodule url for 'third_party/NNPACK'
2025-04-21T21:42:41.3025147Z Synchronizing submodule url for 'third_party/NVTX'
2025-04-21T21:42:41.3051074Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:41.3076988Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-04-21T21:42:41.3123394Z Synchronizing submodule url for 'third_party/benchmark'
2025-04-21T21:42:41.3149026Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-04-21T21:42:41.3185201Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-04-21T21:42:41.3210374Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-04-21T21:42:41.3237355Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-04-21T21:42:41.3263011Z Synchronizing submodule url for 'third_party/cutlass'
2025-04-21T21:42:41.3297219Z Synchronizing submodule url for 'third_party/eigen'
2025-04-21T21:42:41.3326123Z Synchronizing submodule url for 'third_party/fbgemm'
2025-04-21T21:42:41.3350145Z Synchronizing submodule url for 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:41.3378950Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:41.3404329Z Synchronizing submodule url for 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:41.3436635Z Synchronizing submodule url for 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:41.3461967Z Synchronizing submodule url for 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:41.3490289Z Synchronizing submodule url for 'third_party/flash-attention'
2025-04-21T21:42:41.3514994Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:41.3548755Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:41.3591927Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-04-21T21:42:41.3620379Z Synchronizing submodule url for 'third_party/fmt'
2025-04-21T21:42:41.3649688Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:41.3676013Z Synchronizing submodule url for 'third_party/gloo'
2025-04-21T21:42:41.3702446Z Synchronizing submodule url for 'third_party/googletest'
2025-04-21T21:42:41.3729507Z Synchronizing submodule url for 'third_party/ideep'
2025-04-21T21:42:41.3752250Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:41.3792893Z Synchronizing submodule url for 'third_party/ittapi'
2025-04-21T21:42:41.3819242Z Synchronizing submodule url for 'third_party/kineto'
2025-04-21T21:42:41.3844605Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:41.3869894Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:41.3898323Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:41.3925878Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:41.3953513Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:41.3978553Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:41.4008177Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:41.4035776Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:41.4062684Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:41.4090688Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:41.4124005Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:41.4148966Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:41.4182549Z Synchronizing submodule url for 'third_party/kleidiai'
2025-04-21T21:42:41.4208890Z Synchronizing submodule url for 'third_party/mimalloc'
2025-04-21T21:42:41.4235640Z Synchronizing submodule url for 'third_party/nlohmann'
2025-04-21T21:42:41.4263181Z Synchronizing submodule url for 'third_party/onnx'
2025-04-21T21:42:41.4302322Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:41.4333655Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-04-21T21:42:41.4359386Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:41.4385349Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:41.4410127Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:41.4435861Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:41.4462443Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:41.4487814Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:41.4513383Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:41.4538657Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:41.4566418Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:41.4596642Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:41.4644563Z Synchronizing submodule url for 'third_party/pocketfft'
2025-04-21T21:42:41.4671314Z Synchronizing submodule url for 'third_party/protobuf'
2025-04-21T21:42:41.4698429Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:41.4723874Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:41.4754963Z Synchronizing submodule url for 'third_party/psimd'
2025-04-21T21:42:41.4781192Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-04-21T21:42:41.4807651Z Synchronizing submodule url for 'third_party/pybind11'
2025-04-21T21:42:41.4835302Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-04-21T21:42:41.4861601Z Synchronizing submodule url for 'third_party/sleef'
2025-04-21T21:42:41.4888134Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-04-21T21:42:41.4912444Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:41.4938121Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:41.4963234Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:41.4988958Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:41.5012192Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:41.5057799Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-04-21T21:42:41.5650549Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-04-21T21:42:41.5820739Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-04-21T21:42:41.5952441Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-04-21T21:42:41.6344106Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-04-21T21:42:41.7003597Z Submodule path 'third_party/NVTX': checked out 'e170594ac7cf1dac584da473d4ca9301087090c1'
2025-04-21T21:42:41.7572811Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2025-04-21T21:42:43.0074080Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-04-21T21:42:43.0402751Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2025-04-21T21:42:43.4760476Z Submodule path 'third_party/composable_kernel': checked out '8086bbe3a78d931eb96fe12fdc014082e18d18d3'
2025-04-21T21:42:43.5370863Z Submodule path 'third_party/cpp-httplib': checked out '3b6597bba913d51161383657829b7e644e59c006'
2025-04-21T21:42:43.6764827Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-04-21T21:42:43.7270879Z Submodule path 'third_party/cudnn_frontend': checked out '8801fd7b31c2f798732ed1e1dd4711a9cde4217d'
2025-04-21T21:42:44.7350952Z Submodule path 'third_party/cutlass': checked out 'afa1772203677c5118fcd82537a9c8fefbcc7008'
2025-04-21T21:42:45.0800088Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2025-04-21T21:42:45.1750347Z Submodule path 'third_party/fbgemm': checked out 'dbc3157bf256f1339b3fa1fef2be89ac4078be0e'
2025-04-21T21:42:45.2353987Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2025-04-21T21:42:45.3711980Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2025-04-21T21:42:46.0810491Z Submodule path 'third_party/fbgemm/third_party/cutlass': checked out 'fc9ebc645b63f3a6bc80aaefde5c063fb72110d6'
2025-04-21T21:42:46.1432227Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2025-04-21T21:42:46.1580017Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '23f53b025b466d8ec3c45d52290d3442f7fbe6b1'
2025-04-21T21:42:46.3009415Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-04-21T21:42:46.7254842Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-04-21T21:42:47.6842463Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-04-21T21:42:47.9049735Z Submodule path 'third_party/flatbuffers': checked out '01834de25e4bf3975a9a00e816292b1ad0fe184b'
2025-04-21T21:42:47.9516366Z Submodule path 'third_party/fmt': checked out '123913715afeb8a437e6388b4473fcc4753e1c9a'
2025-04-21T21:42:48.0023999Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-04-21T21:42:48.0434184Z Submodule path 'third_party/gloo': checked out 'c61070427610ccd923efe3e7f8b3eca12bbcc31a'
2025-04-21T21:42:48.1045403Z Submodule path 'third_party/googletest': checked out 'b514bdc898e2951020cbdca1304b75f5950d1f59'
2025-04-21T21:42:48.1232091Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-04-21T21:42:48.9447386Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-04-21T21:42:48.9672874Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2025-04-21T21:42:49.1010036Z Submodule path 'third_party/kineto': checked out '5fa4bd8c8fb581a621f72957d0cb11431cc4714f'
2025-04-21T21:42:49.2127631Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-04-21T21:42:49.4525347Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-04-21T21:42:49.4806478Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-04-21T21:42:49.5304118Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-04-21T21:42:49.5506136Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-04-21T21:42:49.5629354Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-04-21T21:42:49.5887485Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-04-21T21:42:49.6434813Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-04-21T21:42:49.7920323Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-04-21T21:42:49.8164466Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-04-21T21:42:49.8643771Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-04-21T21:42:49.9259616Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-04-21T21:42:49.9864195Z Submodule path 'third_party/kleidiai': checked out 'ef685a13cfbe8d418aa2ed34350e21e4938358b6'
2025-04-21T21:42:50.0419896Z Submodule path 'third_party/mimalloc': checked out 'b66e3214d8a104669c2ec05ae91ebc26a8f5ab78'
2025-04-21T21:42:50.2265293Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2025-04-21T21:42:50.9130730Z Submodule path 'third_party/onnx': checked out 'b8baa8446686496da4cc8fda09f2b6fe65c2a02c'
2025-04-21T21:42:50.9652599Z Submodule path 'third_party/onnx/third_party/pybind11': checked out '3e9dfa2866941655c56877882565e7577de6fc7b'
2025-04-21T21:42:51.0914259Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-04-21T21:42:51.1225837Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-04-21T21:42:51.1764228Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-04-21T21:42:51.1939640Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-04-21T21:42:51.3554288Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-04-21T21:42:51.3723015Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-04-21T21:42:51.3937887Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-04-21T21:42:51.4190543Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-04-21T21:42:51.7495563Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-04-21T21:42:51.8112738Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-04-21T21:42:52.7339795Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-04-21T21:42:52.7495640Z Submodule path 'third_party/pocketfft': checked out '9d3ab05a7fffbc71a492bc6a17be034e83e8f0fe'
2025-04-21T21:42:53.1577483Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-04-21T21:42:53.1798062Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-04-21T21:42:53.2581123Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-04-21T21:42:53.2709798Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-04-21T21:42:53.2892259Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-04-21T21:42:53.3445413Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-04-21T21:42:53.3864916Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-04-21T21:42:53.4516490Z Submodule path 'third_party/sleef': checked out '56e1f79cb140fb9326d612d0be06b5250565cade'
2025-04-21T21:42:53.4983323Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2025-04-21T21:42:53.5592563Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-04-21T21:42:53.5834334Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-04-21T21:42:53.6779688Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2025-04-21T21:42:53.7196974Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-04-21T21:42:53.7316001Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-04-21T21:42:53.7375146Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-04-21T21:42:53.7734393Z Entering 'android/libs/fbjni'
2025-04-21T21:42:53.7788766Z Entering 'third_party/FP16'
2025-04-21T21:42:53.7840679Z Entering 'third_party/FXdiv'
2025-04-21T21:42:53.7892052Z Entering 'third_party/NNPACK'
2025-04-21T21:42:53.7945567Z Entering 'third_party/NVTX'
2025-04-21T21:42:53.7997909Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:53.8051970Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:53.8124722Z Entering 'third_party/benchmark'
2025-04-21T21:42:53.8176917Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:53.8237309Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:53.8288835Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:53.8342803Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:53.8396426Z Entering 'third_party/cutlass'
2025-04-21T21:42:53.8459097Z Entering 'third_party/eigen'
2025-04-21T21:42:53.8515802Z Entering 'third_party/fbgemm'
2025-04-21T21:42:53.8569945Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:53.8621032Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:53.8678252Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:53.8735971Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:53.8786233Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:53.8839973Z Entering 'third_party/flash-attention'
2025-04-21T21:42:53.8896017Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:53.8954633Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:53.9016355Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:53.9071707Z Entering 'third_party/fmt'
2025-04-21T21:42:53.9130438Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:53.9182939Z Entering 'third_party/gloo'
2025-04-21T21:42:53.9241582Z Entering 'third_party/googletest'
2025-04-21T21:42:53.9296686Z Entering 'third_party/ideep'
2025-04-21T21:42:53.9349933Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:53.9411041Z Entering 'third_party/ittapi'
2025-04-21T21:42:53.9464154Z Entering 'third_party/kineto'
2025-04-21T21:42:53.9516072Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:53.9565885Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:53.9618371Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:53.9678059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:53.9729964Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:53.9780274Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:53.9836075Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:53.9889537Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:53.9941546Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:53.9993502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:54.0048717Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:54.0098703Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:54.0151469Z Entering 'third_party/kleidiai'
2025-04-21T21:42:54.0203661Z Entering 'third_party/mimalloc'
2025-04-21T21:42:54.0257561Z Entering 'third_party/nlohmann'
2025-04-21T21:42:54.0310336Z Entering 'third_party/onnx'
2025-04-21T21:42:54.0377920Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:54.0434894Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:54.0487763Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:54.0537255Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:54.0586229Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:54.0635084Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:54.0685244Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:54.0734054Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:54.0782872Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:54.0832798Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:54.0885645Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:54.0940462Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:54.1013231Z Entering 'third_party/pocketfft'
2025-04-21T21:42:54.1066050Z Entering 'third_party/protobuf'
2025-04-21T21:42:54.1120892Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:54.1173147Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:54.1230107Z Entering 'third_party/psimd'
2025-04-21T21:42:54.1286206Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:54.1339526Z Entering 'third_party/pybind11'
2025-04-21T21:42:54.1395397Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:54.1447877Z Entering 'third_party/sleef'
2025-04-21T21:42:54.1499521Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:54.1551160Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:54.1601266Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:54.1651358Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:54.1701246Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:54.1751290Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:54.1824567Z ##[endgroup]
2025-04-21T21:42:54.1825005Z ##[group]Persisting credentials for submodules
2025-04-21T21:42:54.1830922Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-04-21T21:42:54.2190492Z Entering 'android/libs/fbjni'
2025-04-21T21:42:54.2237103Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2237585Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2281353Z Entering 'third_party/FP16'
2025-04-21T21:42:54.2327434Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2328266Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2371842Z Entering 'third_party/FXdiv'
2025-04-21T21:42:54.2416859Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2417349Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2463636Z Entering 'third_party/NNPACK'
2025-04-21T21:42:54.2506812Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2507239Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2551171Z Entering 'third_party/NVTX'
2025-04-21T21:42:54.2595292Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2595705Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2639559Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:54.2683523Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2684015Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2728737Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:54.2773629Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2774072Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2833234Z Entering 'third_party/benchmark'
2025-04-21T21:42:54.2877492Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2877921Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2921330Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:54.2965748Z url.https://github.com/.insteadof
2025-04-21T21:42:54.2966170Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3016428Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:54.3060568Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3060937Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3105423Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:54.3149681Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3150067Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3194423Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:54.3238961Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3239309Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3283409Z Entering 'third_party/cutlass'
2025-04-21T21:42:54.3328077Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3328439Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3381782Z Entering 'third_party/eigen'
2025-04-21T21:42:54.3426764Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3427251Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3474023Z Entering 'third_party/fbgemm'
2025-04-21T21:42:54.3518083Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3518418Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3562920Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:54.3607654Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3608089Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3652451Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:54.3700692Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3701177Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3745134Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:54.3787850Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3788320Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3838682Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:54.3881533Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3882037Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3925398Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:54.3967789Z url.https://github.com/.insteadof
2025-04-21T21:42:54.3968155Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4015389Z Entering 'third_party/flash-attention'
2025-04-21T21:42:54.4059655Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4060016Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4104028Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:54.4147042Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4147396Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4196678Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:54.4239745Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4240353Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4295800Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:54.4340954Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4341301Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4387886Z Entering 'third_party/fmt'
2025-04-21T21:42:54.4432167Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4432511Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4475854Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:54.4519674Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4520018Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4564064Z Entering 'third_party/gloo'
2025-04-21T21:42:54.4608113Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4608463Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4653184Z Entering 'third_party/googletest'
2025-04-21T21:42:54.4697259Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4697584Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4755022Z Entering 'third_party/ideep'
2025-04-21T21:42:54.4787079Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4787437Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4830013Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:54.4873050Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4873824Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4931329Z Entering 'third_party/ittapi'
2025-04-21T21:42:54.4976177Z url.https://github.com/.insteadof
2025-04-21T21:42:54.4976667Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5021013Z Entering 'third_party/kineto'
2025-04-21T21:42:54.5066262Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5066788Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5109525Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:54.5152630Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5152969Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5199589Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:54.5243897Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5244226Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5289674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:54.5333262Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5333602Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5377869Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:54.5420975Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5421343Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5469722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:54.5512441Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5512801Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5558647Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:54.5601869Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5602258Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5654002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:54.5697537Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5697866Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5742868Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:54.5790668Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5791009Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5837621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:54.5880985Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5881344Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5928487Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:54.5972006Z url.https://github.com/.insteadof
2025-04-21T21:42:54.5972350Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6019762Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:54.6062570Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6062933Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6105598Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:54.6150909Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6151265Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6197743Z Entering 'third_party/kleidiai'
2025-04-21T21:42:54.6245273Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6245745Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6290276Z Entering 'third_party/mimalloc'
2025-04-21T21:42:54.6335372Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6335720Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6379557Z Entering 'third_party/nlohmann'
2025-04-21T21:42:54.6423894Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6424557Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6470691Z Entering 'third_party/onnx'
2025-04-21T21:42:54.6520188Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6520523Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6579552Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:54.6622481Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6622974Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6673567Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:54.6718616Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6719106Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6764876Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:54.6811590Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6812088Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6855664Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:54.6898515Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6898876Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6943165Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:54.6993588Z url.https://github.com/.insteadof
2025-04-21T21:42:54.6994067Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7037527Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:54.7080015Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7080367Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7125351Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:54.7167604Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7168088Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7210753Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:54.7253949Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7254445Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7297516Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:54.7340781Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7341258Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7386346Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:54.7430494Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7430937Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7476211Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:54.7518405Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7518816Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7566383Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:54.7609437Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7609922Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7676909Z Entering 'third_party/pocketfft'
2025-04-21T21:42:54.7721103Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7721818Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7765224Z Entering 'third_party/protobuf'
2025-04-21T21:42:54.7808873Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7809589Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7855805Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:54.7899431Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7899911Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7943714Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:54.7990604Z url.https://github.com/.insteadof
2025-04-21T21:42:54.7991039Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8044648Z Entering 'third_party/psimd'
2025-04-21T21:42:54.8088164Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8088612Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8132546Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:54.8175688Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8176125Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8218752Z Entering 'third_party/pybind11'
2025-04-21T21:42:54.8262643Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8263002Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8308039Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:54.8352436Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8352765Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8396989Z Entering 'third_party/sleef'
2025-04-21T21:42:54.8440862Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8441221Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8484180Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:54.8529767Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8530123Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8572509Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:54.8615154Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8615492Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8658068Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:54.8699305Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8699628Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8741927Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:54.8784045Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8784373Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8828259Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:54.8869405Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8869717Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8911135Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:54.8954296Z url.https://github.com/.insteadof
2025-04-21T21:42:54.8954640Z url.https://github.com/.insteadof
2025-04-21T21:42:54.9024651Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-04-21T21:42:54.9384440Z Entering 'android/libs/fbjni'
2025-04-21T21:42:54.9446866Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-04-21T21:42:54.9468624Z Entering 'third_party/FP16'
2025-04-21T21:42:54.9531194Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-04-21T21:42:54.9553091Z Entering 'third_party/FXdiv'
2025-04-21T21:42:54.9615778Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-04-21T21:42:54.9637596Z Entering 'third_party/NNPACK'
2025-04-21T21:42:54.9699829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-04-21T21:42:54.9721871Z Entering 'third_party/NVTX'
2025-04-21T21:42:54.9785281Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-04-21T21:42:54.9806994Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:54.9869939Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-04-21T21:42:54.9892132Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:55.0058391Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-04-21T21:42:55.0096825Z Entering 'third_party/benchmark'
2025-04-21T21:42:55.0164600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-04-21T21:42:55.0186141Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:55.0255472Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-04-21T21:42:55.0286074Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:55.0348135Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-04-21T21:42:55.0369958Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:55.0435823Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-04-21T21:42:55.0459276Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:55.0522642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-04-21T21:42:55.0543624Z Entering 'third_party/cutlass'
2025-04-21T21:42:55.0606485Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-04-21T21:42:55.0637626Z Entering 'third_party/eigen'
2025-04-21T21:42:55.0699094Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2025-04-21T21:42:55.0723417Z Entering 'third_party/fbgemm'
2025-04-21T21:42:55.0789972Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-04-21T21:42:55.0811842Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:55.0875546Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2025-04-21T21:42:55.0896742Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:55.0959627Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2025-04-21T21:42:55.0982167Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:55.1048654Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cutlass/config	remote.origin.url
2025-04-21T21:42:55.1077457Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:55.1141944Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.1163037Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:55.1229278Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2025-04-21T21:42:55.1253573Z Entering 'third_party/flash-attention'
2025-04-21T21:42:55.1316280Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-04-21T21:42:55.1337496Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:55.1398812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-04-21T21:42:55.1427109Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:55.1488325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-04-21T21:42:55.1521662Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:55.1588296Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-04-21T21:42:55.1612978Z Entering 'third_party/fmt'
2025-04-21T21:42:55.1675913Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-04-21T21:42:55.1697195Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:55.1760128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-04-21T21:42:55.1782277Z Entering 'third_party/gloo'
2025-04-21T21:42:55.1844980Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-04-21T21:42:55.1866769Z Entering 'third_party/googletest'
2025-04-21T21:42:55.1927655Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.1949289Z Entering 'third_party/ideep'
2025-04-21T21:42:55.2010907Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-04-21T21:42:55.2031613Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:55.2092863Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-04-21T21:42:55.2124510Z Entering 'third_party/ittapi'
2025-04-21T21:42:55.2186273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-04-21T21:42:55.2208139Z Entering 'third_party/kineto'
2025-04-21T21:42:55.2270530Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-04-21T21:42:55.2290678Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:55.2359242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-04-21T21:42:55.2379246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:55.2443253Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-04-21T21:42:55.2470852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:55.2529057Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-04-21T21:42:55.2550040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:55.2611600Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-04-21T21:42:55.2633634Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:55.2700377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-04-21T21:42:55.2719050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:55.2783793Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-04-21T21:42:55.2807775Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:55.2870275Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-04-21T21:42:55.2891216Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:55.2953015Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.2975783Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:55.3036290Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-04-21T21:42:55.3058466Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:55.3119888Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-04-21T21:42:55.3145378Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:55.3209743Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-04-21T21:42:55.3231938Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:55.3298152Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.3323753Z Entering 'third_party/kleidiai'
2025-04-21T21:42:55.3387327Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-04-21T21:42:55.3409876Z Entering 'third_party/mimalloc'
2025-04-21T21:42:55.3472708Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-04-21T21:42:55.3495131Z Entering 'third_party/nlohmann'
2025-04-21T21:42:55.3557142Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-04-21T21:42:55.3579970Z Entering 'third_party/onnx'
2025-04-21T21:42:55.3642187Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-04-21T21:42:55.3676744Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:55.3739411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-04-21T21:42:55.3767101Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:55.3833816Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-04-21T21:42:55.3855672Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:55.3916279Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-04-21T21:42:55.3938526Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:55.3999012Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.4020219Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:55.4081335Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-04-21T21:42:55.4101867Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:55.4170268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-04-21T21:42:55.4192264Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:55.4262059Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-04-21T21:42:55.4282897Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:55.4344330Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-04-21T21:42:55.4366762Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:55.4428531Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-04-21T21:42:55.4447625Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:55.4510605Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-04-21T21:42:55.4535134Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:55.4597770Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-04-21T21:42:55.4621741Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:55.4688136Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-04-21T21:42:55.4731634Z Entering 'third_party/pocketfft'
2025-04-21T21:42:55.4793894Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-04-21T21:42:55.4815041Z Entering 'third_party/protobuf'
2025-04-21T21:42:55.4878197Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-04-21T21:42:55.4900894Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:55.4962437Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-04-21T21:42:55.4983507Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:55.5045242Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.5070792Z Entering 'third_party/psimd'
2025-04-21T21:42:55.5132682Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-04-21T21:42:55.5153923Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:55.5215379Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-04-21T21:42:55.5236735Z Entering 'third_party/pybind11'
2025-04-21T21:42:55.5298430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-04-21T21:42:55.5320481Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:55.5386968Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-04-21T21:42:55.5408572Z Entering 'third_party/sleef'
2025-04-21T21:42:55.5469899Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-04-21T21:42:55.5491447Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:55.5554601Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-04-21T21:42:55.5574292Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:55.5635149Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-04-21T21:42:55.5656860Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:55.5717284Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-04-21T21:42:55.5738692Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:55.5799325Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-04-21T21:42:55.5820384Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:55.5882273Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-04-21T21:42:55.5901519Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:55.5963626Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-04-21T21:42:55.7100135Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-04-21T21:42:55.7464079Z Entering 'android/libs/fbjni'
2025-04-21T21:42:55.7520618Z Entering 'third_party/FP16'
2025-04-21T21:42:55.7574649Z Entering 'third_party/FXdiv'
2025-04-21T21:42:55.7628154Z Entering 'third_party/NNPACK'
2025-04-21T21:42:55.7682015Z Entering 'third_party/NVTX'
2025-04-21T21:42:55.7735742Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:55.7788989Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:55.7858341Z Entering 'third_party/benchmark'
2025-04-21T21:42:55.7911499Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:55.7971553Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:55.8028247Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:55.8094927Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:55.8136611Z Entering 'third_party/cutlass'
2025-04-21T21:42:55.8198977Z Entering 'third_party/eigen'
2025-04-21T21:42:55.8255047Z Entering 'third_party/fbgemm'
2025-04-21T21:42:55.8308338Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:55.8360630Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:55.8411873Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:55.8470125Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:55.8520484Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:55.8575953Z Entering 'third_party/flash-attention'
2025-04-21T21:42:55.8630237Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:55.8692035Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:55.8760575Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:55.8816550Z Entering 'third_party/fmt'
2025-04-21T21:42:55.8870871Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:55.8924960Z Entering 'third_party/gloo'
2025-04-21T21:42:55.8978320Z Entering 'third_party/googletest'
2025-04-21T21:42:55.9031577Z Entering 'third_party/ideep'
2025-04-21T21:42:55.9083560Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:55.9143574Z Entering 'third_party/ittapi'
2025-04-21T21:42:55.9195056Z Entering 'third_party/kineto'
2025-04-21T21:42:55.9246461Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:55.9296489Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:55.9350450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:55.9401776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:55.9453413Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:55.9503624Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:55.9561721Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:55.9611241Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:55.9663134Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:55.9718752Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:55.9772217Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:55.9828494Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:55.9882861Z Entering 'third_party/kleidiai'
2025-04-21T21:42:55.9936888Z Entering 'third_party/mimalloc'
2025-04-21T21:42:55.9990137Z Entering 'third_party/nlohmann'
2025-04-21T21:42:56.0045546Z Entering 'third_party/onnx'
2025-04-21T21:42:56.0112823Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:56.0169813Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:56.0224871Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:56.0276315Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:56.0329745Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:56.0379628Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:56.0436634Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:56.0485939Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:56.0536087Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:56.0585050Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:56.0638498Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:56.0692268Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:56.0771220Z Entering 'third_party/pocketfft'
2025-04-21T21:42:56.0824651Z Entering 'third_party/protobuf'
2025-04-21T21:42:56.0883929Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:56.0934014Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:56.0990295Z Entering 'third_party/psimd'
2025-04-21T21:42:56.1044782Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:56.1098152Z Entering 'third_party/pybind11'
2025-04-21T21:42:56.1150909Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:56.1204070Z Entering 'third_party/sleef'
2025-04-21T21:42:56.1257869Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:56.1310290Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:56.1366694Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:56.1419289Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:56.1470929Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:56.1524873Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:56.1601947Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-04-21T21:42:56.1961860Z Entering 'android/libs/fbjni'
2025-04-21T21:42:56.2012265Z Entering 'third_party/FP16'
2025-04-21T21:42:56.2064253Z Entering 'third_party/FXdiv'
2025-04-21T21:42:56.2115335Z Entering 'third_party/NNPACK'
2025-04-21T21:42:56.2167713Z Entering 'third_party/NVTX'
2025-04-21T21:42:56.2219495Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-21T21:42:56.2278698Z Entering 'third_party/XNNPACK'
2025-04-21T21:42:56.2345469Z Entering 'third_party/benchmark'
2025-04-21T21:42:56.2397412Z Entering 'third_party/composable_kernel'
2025-04-21T21:42:56.2457312Z Entering 'third_party/cpp-httplib'
2025-04-21T21:42:56.2508014Z Entering 'third_party/cpuinfo'
2025-04-21T21:42:56.2560215Z Entering 'third_party/cudnn_frontend'
2025-04-21T21:42:56.2611468Z Entering 'third_party/cutlass'
2025-04-21T21:42:56.2672266Z Entering 'third_party/eigen'
2025-04-21T21:42:56.2725569Z Entering 'third_party/fbgemm'
2025-04-21T21:42:56.2777784Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-21T21:42:56.2829892Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-21T21:42:56.2879715Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-21T21:42:56.2937042Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-21T21:42:56.2987499Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-21T21:42:56.3041154Z Entering 'third_party/flash-attention'
2025-04-21T21:42:56.3093434Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-21T21:42:56.3156127Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-21T21:42:56.3219908Z Entering 'third_party/flatbuffers'
2025-04-21T21:42:56.3274474Z Entering 'third_party/fmt'
2025-04-21T21:42:56.3326792Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-21T21:42:56.3380374Z Entering 'third_party/gloo'
2025-04-21T21:42:56.3434855Z Entering 'third_party/googletest'
2025-04-21T21:42:56.3487445Z Entering 'third_party/ideep'
2025-04-21T21:42:56.3539934Z Entering 'third_party/ideep/mkl-dnn'
2025-04-21T21:42:56.3601528Z Entering 'third_party/ittapi'
2025-04-21T21:42:56.3654294Z Entering 'third_party/kineto'
2025-04-21T21:42:56.3707476Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-21T21:42:56.3758595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-21T21:42:56.3812038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-21T21:42:56.3865636Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-21T21:42:56.3916602Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-21T21:42:56.3966867Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-21T21:42:56.4024262Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-21T21:42:56.4074597Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-21T21:42:56.4125528Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-21T21:42:56.4177084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-21T21:42:56.4230988Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-21T21:42:56.4286063Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-21T21:42:56.4341447Z Entering 'third_party/kleidiai'
2025-04-21T21:42:56.4394851Z Entering 'third_party/mimalloc'
2025-04-21T21:42:56.4448805Z Entering 'third_party/nlohmann'
2025-04-21T21:42:56.4503657Z Entering 'third_party/onnx'
2025-04-21T21:42:56.4571467Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-21T21:42:56.4629079Z Entering 'third_party/opentelemetry-cpp'
2025-04-21T21:42:56.4682038Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-21T21:42:56.4733856Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-21T21:42:56.4784832Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-21T21:42:56.4839061Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-21T21:42:56.4890616Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-21T21:42:56.4940147Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-21T21:42:56.4989791Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-21T21:42:56.5042784Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-21T21:42:56.5094891Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-21T21:42:56.5148507Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-21T21:42:56.5224436Z Entering 'third_party/pocketfft'
2025-04-21T21:42:56.5276369Z Entering 'third_party/protobuf'
2025-04-21T21:42:56.5331261Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-21T21:42:56.5380662Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-21T21:42:56.5434967Z Entering 'third_party/psimd'
2025-04-21T21:42:56.5487390Z Entering 'third_party/pthreadpool'
2025-04-21T21:42:56.5539934Z Entering 'third_party/pybind11'
2025-04-21T21:42:56.5592428Z Entering 'third_party/python-peachpy'
2025-04-21T21:42:56.5645165Z Entering 'third_party/sleef'
2025-04-21T21:42:56.5697661Z Entering 'third_party/tensorpipe'
2025-04-21T21:42:56.5750341Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-21T21:42:56.5800259Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-21T21:42:56.5850828Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-21T21:42:56.5907002Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-21T21:42:56.5956969Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-21T21:42:56.6029907Z ##[endgroup]
2025-04-21T21:42:56.6073293Z [command]/usr/bin/git log -1 --format=%H
2025-04-21T21:42:56.6101972Z b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:42:56.6296499Z Prepare all required actions
2025-04-21T21:42:56.6296993Z Getting action download info
2025-04-21T21:42:56.7201485Z ##[group]Run ./.github/actions/setup-linux
2025-04-21T21:42:56.7201797Z env:
2025-04-21T21:42:56.7202026Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:56.7202290Z ##[endgroup]
2025-04-21T21:42:56.7263647Z ##[group]Run set -euo pipefail
2025-04-21T21:42:56.7264025Z set -euo pipefail
2025-04-21T21:42:56.7264329Z function get_ec2_metadata() {
2025-04-21T21:42:56.7264719Z   # Pulled from instance metadata endpoint for EC2
2025-04-21T21:42:56.7265338Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2025-04-21T21:42:56.7265905Z   category=$1
2025-04-21T21:42:56.7266271Z   # If it is GCP runner (runner name contains gcp), do not run this
2025-04-21T21:42:56.7266693Z   runner_name_str=i-0e08e74978c5f6915
2025-04-21T21:42:56.7267063Z   if [[ -f /.inarc ]]; then
2025-04-21T21:42:56.7267402Z     echo "ARC Runner, no info on ec2 metadata"
2025-04-21T21:42:56.7267790Z   elif [[ $runner_name_str == *"gcp"* ]]; then
2025-04-21T21:42:56.7268244Z     echo "Runner is from Google Cloud Platform, No info on ec2 metadata"
2025-04-21T21:42:56.7268666Z   else
2025-04-21T21:42:56.7269485Z     curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2025-04-21T21:42:56.7270358Z   fi
2025-04-21T21:42:56.7270575Z }
2025-04-21T21:42:56.7270843Z echo "ami-id: $(get_ec2_metadata ami-id)"
2025-04-21T21:42:56.7271259Z echo "instance-id: $(get_ec2_metadata instance-id)"
2025-04-21T21:42:56.7271733Z echo "instance-type: $(get_ec2_metadata instance-type)"
2025-04-21T21:42:56.7272145Z echo "system info $(uname -a)"
2025-04-21T21:42:56.7281743Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:56.7282122Z env:
2025-04-21T21:42:56.7282344Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:56.7282599Z ##[endgroup]
2025-04-21T21:42:56.7439029Z ami-id: ami-071226ecf16aa7d96
2025-04-21T21:42:56.7549434Z instance-id: i-0e08e74978c5f6915
2025-04-21T21:42:56.7660266Z instance-type: g5.4xlarge
2025-04-21T21:42:56.7673687Z system info Linux ip-10-0-23-151.ec2.internal 6.1.130-139.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Mar 11 01:10:58 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
2025-04-21T21:42:56.7706022Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:42:56.7706915Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:42:56.7717316Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:56.7717697Z env:
2025-04-21T21:42:56.7717936Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:56.7718215Z ##[endgroup]
2025-04-21T21:42:56.7787031Z ##[group]Run if systemctl is-active --quiet docker; then
2025-04-21T21:42:56.7787474Z if systemctl is-active --quiet docker; then
2025-04-21T21:42:56.7787858Z     echo "Docker daemon is running...";
2025-04-21T21:42:56.7788181Z else
2025-04-21T21:42:56.7788698Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2025-04-21T21:42:56.7789112Z fi
2025-04-21T21:42:56.7797862Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:56.7798230Z env:
2025-04-21T21:42:56.7798451Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:56.7798714Z ##[endgroup]
2025-04-21T21:42:56.7883012Z Docker daemon is running...
2025-04-21T21:42:56.7930871Z ##[group]Run nick-fields/retry@v3.0.0
2025-04-21T21:42:56.7931173Z with:
2025-04-21T21:42:56.7931382Z   shell: bash
2025-04-21T21:42:56.7931819Z   timeout_minutes: 5
2025-04-21T21:42:56.7932059Z   max_attempts: 3
2025-04-21T21:42:56.7932300Z   retry_wait_seconds: 30
2025-04-21T21:42:56.7934429Z   command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
    --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"

# For LF Runners we need to make sure we also login to Meta's ECR docker registry too.
META_AWS_ACCOUNT_ID=308535385114
if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then
    aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \
        --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
fi

2025-04-21T21:42:56.7936604Z   polling_interval_seconds: 1
2025-04-21T21:42:56.7936877Z   warning_on_retry: true
2025-04-21T21:42:56.7937134Z   continue_on_error: false
2025-04-21T21:42:56.7937386Z env:
2025-04-21T21:42:56.7937596Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:56.7937857Z   AWS_RETRY_MODE: standard
2025-04-21T21:42:56.7938118Z   AWS_MAX_ATTEMPTS: 5
2025-04-21T21:42:56.7938368Z   AWS_DEFAULT_REGION: us-east-1
2025-04-21T21:42:56.7938627Z ##[endgroup]
2025-04-21T21:42:57.9060076Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-04-21T21:42:57.9061019Z Configure a credential helper to remove this warning. See
2025-04-21T21:42:57.9061660Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-04-21T21:42:57.9062047Z 
2025-04-21T21:42:57.9062138Z Login Succeeded
2025-04-21T21:42:58.8799570Z Command completed after 1 attempt(s).
2025-04-21T21:42:58.8880877Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-04-21T21:42:58.8881390Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-04-21T21:42:58.8881845Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-04-21T21:42:58.8891249Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:58.8891617Z env:
2025-04-21T21:42:58.8907182Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:58.8907445Z ##[endgroup]
2025-04-21T21:42:58.9016235Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-04-21T21:42:58.9016782Z # ignore expansion of "docker ps -q" since it could be empty
2025-04-21T21:42:58.9017199Z # shellcheck disable=SC2046
2025-04-21T21:42:58.9017544Z docker stop $(docker ps -q) || true
2025-04-21T21:42:58.9017878Z # Prune all of the docker images
2025-04-21T21:42:58.9018200Z docker system prune -af
2025-04-21T21:42:58.9026523Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:58.9026887Z env:
2025-04-21T21:42:58.9027109Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:58.9027368Z ##[endgroup]
2025-04-21T21:42:58.9317563Z "docker stop" requires at least 1 argument.
2025-04-21T21:42:58.9317974Z See 'docker stop --help'.
2025-04-21T21:42:58.9318149Z 
2025-04-21T21:42:58.9318344Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-04-21T21:42:58.9318615Z 
2025-04-21T21:42:58.9318727Z Stop one or more running containers
2025-04-21T21:42:58.9495775Z Total reclaimed space: 0B
2025-04-21T21:42:58.9547317Z ##[group]Run set +e
2025-04-21T21:42:58.9547597Z set +e
2025-04-21T21:42:58.9547826Z set -x
2025-04-21T21:42:58.9548228Z 
2025-04-21T21:42:58.9548467Z PT_DOMAIN=download.pytorch.org
2025-04-21T21:42:58.9549029Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400,
2025-04-21T21:42:58.9549754Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last
2025-04-21T21:42:58.9550272Z # one is returned at random
2025-04-21T21:42:58.9550651Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1)
2025-04-21T21:42:58.9551023Z 
2025-04-21T21:42:58.9551429Z if [ -z "${RESOLVED_IP}" ]; then
2025-04-21T21:42:58.9551846Z   echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..."
2025-04-21T21:42:58.9552352Z   RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1)
2025-04-21T21:42:58.9552735Z 
2025-04-21T21:42:58.9552974Z   if [ -z "${RESOLVED_IP}" ]; then
2025-04-21T21:42:58.9553455Z     echo "Couldn't resolve ${PT_DOMAIN}, exiting..."
2025-04-21T21:42:58.9553850Z     exit 1
2025-04-21T21:42:58.9554107Z   fi
2025-04-21T21:42:58.9554321Z fi
2025-04-21T21:42:58.9554532Z 
2025-04-21T21:42:58.9554792Z if grep -r "${PT_DOMAIN}" /etc/hosts; then
2025-04-21T21:42:58.9555148Z   # Clean up any old records first
2025-04-21T21:42:58.9555508Z   sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts
2025-04-21T21:42:58.9555833Z fi
2025-04-21T21:42:58.9556046Z 
2025-04-21T21:42:58.9556350Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts
2025-04-21T21:42:58.9556740Z cat /etc/hosts
2025-04-21T21:42:58.9565559Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:58.9565915Z env:
2025-04-21T21:42:58.9566130Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:58.9566384Z ##[endgroup]
2025-04-21T21:42:58.9592669Z + PT_DOMAIN=download.pytorch.org
2025-04-21T21:42:58.9598835Z ++ dig -4 +short download.pytorch.org
2025-04-21T21:42:58.9599212Z ++ tail -n1
2025-04-21T21:42:58.9854645Z + RESOLVED_IP=18.160.10.22
2025-04-21T21:42:58.9855102Z + '[' -z 18.160.10.22 ']'
2025-04-21T21:42:58.9855440Z + grep -r download.pytorch.org /etc/hosts
2025-04-21T21:42:58.9867833Z 18.160.10.22 download.pytorch.org
2025-04-21T21:42:58.9869860Z + sudo sed -i /download.pytorch.org/d /etc/hosts
2025-04-21T21:42:59.1027290Z + echo '18.160.10.22 download.pytorch.org'
2025-04-21T21:42:59.1027983Z + sudo tee -a /etc/hosts
2025-04-21T21:42:59.2983956Z 18.160.10.22 download.pytorch.org
2025-04-21T21:42:59.3006203Z + cat /etc/hosts
2025-04-21T21:42:59.3016359Z 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
2025-04-21T21:42:59.3021247Z ::1         localhost6 localhost6.localdomain6
2025-04-21T21:42:59.3021946Z 18.160.10.22 download.pytorch.org
2025-04-21T21:42:59.3178206Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-04-21T21:42:59.3178662Z with:
2025-04-21T21:42:59.3179483Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3180383Z   docker-build-dir: .ci/docker
2025-04-21T21:42:59.3180693Z   docker-build-script: ./build.sh
2025-04-21T21:42:59.3180995Z   working-directory: .
2025-04-21T21:42:59.3181354Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.3181760Z   force-push: false
2025-04-21T21:42:59.3181994Z env:
2025-04-21T21:42:59.3182220Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:59.3182485Z ##[endgroup]
2025-04-21T21:42:59.3225551Z ##[group]Run set -ex
2025-04-21T21:42:59.3225834Z set -ex
2025-04-21T21:42:59.3226063Z 
2025-04-21T21:42:59.3226485Z # If the docker build directory or the build script doesn't exist, the action will
2025-04-21T21:42:59.3227136Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-04-21T21:42:59.3227854Z # job could then download the pre-built image as usual
2025-04-21T21:42:59.3228400Z if [[ ! -d "${DOCKER_BUILD_DIR}" ]] || [[ ! -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]]; then
2025-04-21T21:42:59.3228906Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3229348Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3229757Z 
2025-04-21T21:42:59.3230115Z   echo "There is no Docker build script in ${REPO_NAME} repo, skipping..."
2025-04-21T21:42:59.3230556Z   exit 0
2025-04-21T21:42:59.3230783Z else
2025-04-21T21:42:59.3231058Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3231381Z fi
2025-04-21T21:42:59.3231588Z 
2025-04-21T21:42:59.3231932Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-04-21T21:42:59.3232524Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-04-21T21:42:59.3233060Z   # use it as it is, but first let's extract the tag
2025-04-21T21:42:59.3233587Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-04-21T21:42:59.3234090Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3234582Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3234972Z else
2025-04-21T21:42:59.3235418Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-04-21T21:42:59.3236018Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3236645Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3237181Z fi
2025-04-21T21:42:59.3247362Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:59.3247746Z env:
2025-04-21T21:42:59.3247961Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:59.3248233Z   REPO_NAME: pytorch
2025-04-21T21:42:59.3249068Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3249963Z   DOCKER_BUILD_DIR: .ci/docker
2025-04-21T21:42:59.3250254Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-04-21T21:42:59.3250637Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.3251031Z   CUSTOM_TAG_PREFIX: 
2025-04-21T21:42:59.3251273Z ##[endgroup]
2025-04-21T21:42:59.3279073Z + [[ ! -d .ci/docker ]]
2025-04-21T21:42:59.3279352Z + [[ ! -f .ci/docker/./build.sh ]]
2025-04-21T21:42:59.3279632Z + echo skip=false
2025-04-21T21:42:59.3281235Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138 == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-04-21T21:42:59.3287225Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3288083Z ++ awk -F '[:,]' '{print $2}'
2025-04-21T21:42:59.3312027Z + DOCKER_TAG=f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3312834Z + echo docker-tag=f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3314388Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3346943Z ##[group]Run set +e
2025-04-21T21:42:59.3347234Z set +e
2025-04-21T21:42:59.3347467Z set -x
2025-04-21T21:42:59.3347685Z 
2025-04-21T21:42:59.3347889Z login() {
2025-04-21T21:42:59.3348356Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-04-21T21:42:59.3349008Z }
2025-04-21T21:42:59.3349213Z 
2025-04-21T21:42:59.3349414Z retry () {
2025-04-21T21:42:59.3349687Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-04-21T21:42:59.3349993Z }
2025-04-21T21:42:59.3350190Z 
2025-04-21T21:42:59.3350423Z retry login "${DOCKER_REGISTRY}"
2025-04-21T21:42:59.3350721Z 
2025-04-21T21:42:59.3350929Z START_TIME=$(date +%s)
2025-04-21T21:42:59.3351219Z # Wait up to 120 minutes
2025-04-21T21:42:59.3351574Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-04-21T21:42:59.3352055Z   # Check if image already exists, if it does then skip building it
2025-04-21T21:42:59.3352534Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-04-21T21:42:59.3352899Z     exit 0
2025-04-21T21:42:59.3353134Z   fi
2025-04-21T21:42:59.3353451Z 
2025-04-21T21:42:59.3353821Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-04-21T21:42:59.3354458Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-04-21T21:42:59.3355096Z   # latter, it will wait for the Docker images to become available before continuing
2025-04-21T21:42:59.3355593Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-04-21T21:42:59.3355985Z     # It's a Docker build job, let's build the image
2025-04-21T21:42:59.3356325Z     break
2025-04-21T21:42:59.3356547Z   else
2025-04-21T21:42:59.3356881Z     # It's a regular build job, wait for the image to become available
2025-04-21T21:42:59.3357282Z     sleep 300
2025-04-21T21:42:59.3357527Z   fi
2025-04-21T21:42:59.3357732Z done
2025-04-21T21:42:59.3357948Z 
2025-04-21T21:42:59.3358291Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-04-21T21:42:59.3358841Z # be empty.  The default action would be to continue rebuild the image
2025-04-21T21:42:59.3359342Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-04-21T21:42:59.3359795Z   # if we're on the base branch then use the parent commit
2025-04-21T21:42:59.3360190Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-04-21T21:42:59.3360491Z else
2025-04-21T21:42:59.3360812Z   # otherwise we're on a PR, so use the most recent base commit
2025-04-21T21:42:59.3361277Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-04-21T21:42:59.3361623Z fi
2025-04-21T21:42:59.3361832Z 
2025-04-21T21:42:59.3362061Z if [[ -z "${MERGE_BASE}" ]]; then
2025-04-21T21:42:59.3362561Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3362885Z 
2025-04-21T21:42:59.3363332Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-04-21T21:42:59.3363882Z   exit 0
2025-04-21T21:42:59.3364118Z fi
2025-04-21T21:42:59.3364346Z 
2025-04-21T21:42:59.3364640Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-04-21T21:42:59.3365290Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-04-21T21:42:59.3365852Z   exit 1
2025-04-21T21:42:59.3366070Z fi
2025-04-21T21:42:59.3366270Z 
2025-04-21T21:42:59.3366620Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-04-21T21:42:59.3367264Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-04-21T21:42:59.3367838Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-04-21T21:42:59.3368476Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-04-21T21:42:59.3369299Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-04-21T21:42:59.3369746Z fi
2025-04-21T21:42:59.3369952Z 
2025-04-21T21:42:59.3370208Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-04-21T21:42:59.3378312Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:42:59.3378680Z env:
2025-04-21T21:42:59.3378898Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:42:59.3379193Z   DOCKER_BUILD_DIR: .ci/docker
2025-04-21T21:42:59.3379540Z   BASE_REVISION: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:42:59.3380483Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3381401Z   DOCKER_TAG: f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:42:59.3381847Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.3382233Z   DOCKER_PUSH: 
2025-04-21T21:42:59.3382453Z ##[endgroup]
2025-04-21T21:42:59.3408672Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.3409126Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.3411232Z + aws ecr get-login-password --region us-east-1
2025-04-21T21:42:59.3413273Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:42:59.8581213Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-04-21T21:42:59.8582013Z Configure a credential helper to remove this warning. See
2025-04-21T21:42:59.8582586Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-04-21T21:42:59.8582988Z 
2025-04-21T21:42:59.8584273Z Login Succeeded
2025-04-21T21:42:59.8609581Z ++ date +%s
2025-04-21T21:42:59.8621784Z + START_TIME=1745271779
2025-04-21T21:42:59.8625938Z ++ date +%s
2025-04-21T21:42:59.8637624Z + [[ 1745264579 -lt 1745271779 ]]
2025-04-21T21:42:59.8638816Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:00.0838029Z {
2025-04-21T21:43:00.0838872Z 	"schemaVersion": 2,
2025-04-21T21:43:00.0839621Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-04-21T21:43:00.0840227Z 	"config": {
2025-04-21T21:43:00.0840678Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-04-21T21:43:00.0841214Z 		"size": 46736,
2025-04-21T21:43:00.0841645Z 		"digest": "sha256:7b29eacbf93f5bdd9e6b666f27edb34c0d2f289aa58ceaf3bdfcf4e06f505e55"
2025-04-21T21:43:00.0842114Z 	},
2025-04-21T21:43:00.0842314Z 	"layers": [
2025-04-21T21:43:00.0842839Z 		{
2025-04-21T21:43:00.0843166Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0843700Z 			"size": 28583948,
2025-04-21T21:43:00.0844258Z 			"digest": "sha256:86e5016c269355b382c9cabab4f6646d56d75914f20d545289970436dae431b1"
2025-04-21T21:43:00.0844715Z 		},
2025-04-21T21:43:00.0844984Z 		{
2025-04-21T21:43:00.0845443Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0845899Z 			"size": 7964619,
2025-04-21T21:43:00.0846394Z 			"digest": "sha256:49e139a3d6c2f1801aa0cea1eb34e57c5314065b679325df205026fb175383b8"
2025-04-21T21:43:00.0847028Z 		},
2025-04-21T21:43:00.0847302Z 		{
2025-04-21T21:43:00.0847716Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0848121Z 			"size": 57379226,
2025-04-21T21:43:00.0848536Z 			"digest": "sha256:a14844c8c51f98ebee2b5eda8bff8742dd804385e1dbbeb56928946113000293"
2025-04-21T21:43:00.0848983Z 		},
2025-04-21T21:43:00.0849172Z 		{
2025-04-21T21:43:00.0849495Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0849891Z 			"size": 187,
2025-04-21T21:43:00.0850501Z 			"digest": "sha256:18fb524087fbed3307d23afde734ef1df452f55e7a22e28b56ce1e6eb9c6b3d9"
2025-04-21T21:43:00.0850956Z 		},
2025-04-21T21:43:00.0851147Z 		{
2025-04-21T21:43:00.0851463Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0851867Z 			"size": 6885,
2025-04-21T21:43:00.0852272Z 			"digest": "sha256:efd686a7b2c8bb052a3bb919973c4bee33c2bbcff08fd431362b90c9469fbc4d"
2025-04-21T21:43:00.0852722Z 		},
2025-04-21T21:43:00.0852916Z 		{
2025-04-21T21:43:00.0853235Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0853638Z 			"size": 1438282733,
2025-04-21T21:43:00.0854041Z 			"digest": "sha256:52c648e213347a29001a45b2c4f834cc60621a3d6ab68fa06ebd2075f6487ec5"
2025-04-21T21:43:00.0854487Z 		},
2025-04-21T21:43:00.0854696Z 		{
2025-04-21T21:43:00.0855050Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0855454Z 			"size": 62694,
2025-04-21T21:43:00.0855858Z 			"digest": "sha256:56e384e4e5aa7ebddf31686fc6ae62ef19fdc5311bbea8186563fc2aeca2fb52"
2025-04-21T21:43:00.0856324Z 		},
2025-04-21T21:43:00.0856511Z 		{
2025-04-21T21:43:00.0856830Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0857228Z 			"size": 1684,
2025-04-21T21:43:00.0857630Z 			"digest": "sha256:fb71b792ec6c458c338cace5ce42f653e4ae4b24442a49b7c47af3a38342b5bc"
2025-04-21T21:43:00.0858088Z 		},
2025-04-21T21:43:00.0858273Z 		{
2025-04-21T21:43:00.0858592Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0858999Z 			"size": 1523,
2025-04-21T21:43:00.0859386Z 			"digest": "sha256:5509576f2693abf94a50a70a59bbf8b519f20705f147d495ddde85981c8189fa"
2025-04-21T21:43:00.0859835Z 		},
2025-04-21T21:43:00.0860022Z 		{
2025-04-21T21:43:00.0860337Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0860743Z 			"size": 2515216700,
2025-04-21T21:43:00.0861157Z 			"digest": "sha256:1e6c6f2d245956001435a5df27fd4defea08affa7f6c22252464f758447db401"
2025-04-21T21:43:00.0861620Z 		},
2025-04-21T21:43:00.0861806Z 		{
2025-04-21T21:43:00.0862125Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0862531Z 			"size": 86619,
2025-04-21T21:43:00.0862919Z 			"digest": "sha256:a1fe8922734d371e0c0fd5b04183758331e08f45553720ecff4b062824e0c8e1"
2025-04-21T21:43:00.0863363Z 		},
2025-04-21T21:43:00.0863551Z 		{
2025-04-21T21:43:00.0863863Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0864265Z 			"size": 1893,
2025-04-21T21:43:00.0864658Z 			"digest": "sha256:84e5433913c60a912877721ebf3a4d115b8bd55fb1c5a298b58fdc1f3ef945b1"
2025-04-21T21:43:00.0865101Z 		},
2025-04-21T21:43:00.0865288Z 		{
2025-04-21T21:43:00.0865606Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0866103Z 			"size": 248991124,
2025-04-21T21:43:00.0866519Z 			"digest": "sha256:c68e78d48b95f6272003f9de9b80c623c4b361ed6e860801cbe1bd2db4d97263"
2025-04-21T21:43:00.0866997Z 		},
2025-04-21T21:43:00.0867266Z 		{
2025-04-21T21:43:00.0867701Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0868261Z 			"size": 703,
2025-04-21T21:43:00.0868706Z 			"digest": "sha256:b14b320c459cd13b3ef4a06575f1b29513d8b0e6c1672ea68ae5b34a977cd975"
2025-04-21T21:43:00.0869155Z 		},
2025-04-21T21:43:00.0869346Z 		{
2025-04-21T21:43:00.0869665Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0870110Z 			"size": 1262,
2025-04-21T21:43:00.0870670Z 			"digest": "sha256:0d929bfaefee4bb307c42773c6f04675af8b94398a51b1c8575447099bbf8607"
2025-04-21T21:43:00.0871208Z 		},
2025-04-21T21:43:00.0871394Z 		{
2025-04-21T21:43:00.0871721Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0872128Z 			"size": 484,
2025-04-21T21:43:00.0872539Z 			"digest": "sha256:a5a8ea5a3eabb6655fad5c54e28c86000dca6ac1c3a5a485c718f87b60e28711"
2025-04-21T21:43:00.0873006Z 		},
2025-04-21T21:43:00.0873458Z 		{
2025-04-21T21:43:00.0873782Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0874181Z 			"size": 91731730,
2025-04-21T21:43:00.0874591Z 			"digest": "sha256:281f13ef6929e0c7c8259f29b3dc71103ec70018f2711ec9f8a8f04093099171"
2025-04-21T21:43:00.0875050Z 		},
2025-04-21T21:43:00.0875241Z 		{
2025-04-21T21:43:00.0875565Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0875973Z 			"size": 3767,
2025-04-21T21:43:00.0876361Z 			"digest": "sha256:a8695f1c94c34953b85e2090d713494e7fc84adc148495937b7a7c3deed10275"
2025-04-21T21:43:00.0876812Z 		},
2025-04-21T21:43:00.0877017Z 		{
2025-04-21T21:43:00.0877333Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0877740Z 			"size": 1868,
2025-04-21T21:43:00.0878148Z 			"digest": "sha256:c72f323be8018695bb0b260acf986dd615ac7ce067ad92b0eb1ac2e72925a39a"
2025-04-21T21:43:00.0878604Z 		},
2025-04-21T21:43:00.0878786Z 		{
2025-04-21T21:43:00.0879104Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0879514Z 			"size": 701,
2025-04-21T21:43:00.0879905Z 			"digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03"
2025-04-21T21:43:00.0880365Z 		},
2025-04-21T21:43:00.0880551Z 		{
2025-04-21T21:43:00.0880861Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0881264Z 			"size": 477,
2025-04-21T21:43:00.0881674Z 			"digest": "sha256:a9bdca78eb50cb709cb6b39f9f2bcd879bc2bb990b944eca5bc0b41d8ac75114"
2025-04-21T21:43:00.0882134Z 		},
2025-04-21T21:43:00.0882327Z 		{
2025-04-21T21:43:00.0882642Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0883040Z 			"size": 3018885224,
2025-04-21T21:43:00.0883448Z 			"digest": "sha256:87b13320574046b541e9fdaa0186880e7082d07d823639a45f77e9da1e325fda"
2025-04-21T21:43:00.0883890Z 		},
2025-04-21T21:43:00.0884086Z 		{
2025-04-21T21:43:00.0884450Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0884856Z 			"size": 380,
2025-04-21T21:43:00.0885249Z 			"digest": "sha256:1a1f3a6b6e2805950beb3918166ea519e8237d7b6493d050bce0ecb65b3750e6"
2025-04-21T21:43:00.0885696Z 		},
2025-04-21T21:43:00.0885883Z 		{
2025-04-21T21:43:00.0886202Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0886602Z 			"size": 12918,
2025-04-21T21:43:00.0887002Z 			"digest": "sha256:1e726e5b69dff0c7b1285406bf76144f96e624de579ac59e252da98586616399"
2025-04-21T21:43:00.0887455Z 		},
2025-04-21T21:43:00.0887655Z 		{
2025-04-21T21:43:00.0887977Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0888379Z 			"size": 791,
2025-04-21T21:43:00.0888786Z 			"digest": "sha256:e55ee61cc4aaf6da32e72efeef162eda299bb6a632e8d8dcb487521c151204a0"
2025-04-21T21:43:00.0889360Z 		},
2025-04-21T21:43:00.0889556Z 		{
2025-04-21T21:43:00.0889877Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0912326Z 			"size": 106,
2025-04-21T21:43:00.0912786Z 			"digest": "sha256:d2c9e5f6d068cc6a329295a862e67a4110accba3b0c35efaab96121dc83e2c79"
2025-04-21T21:43:00.0913324Z 		},
2025-04-21T21:43:00.0913531Z 		{
2025-04-21T21:43:00.0913839Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0914249Z 			"size": 503,
2025-04-21T21:43:00.0914659Z 			"digest": "sha256:a9e23e3c0a187a40d8af9500ce12468894283cc71e0ba2fb1fcb6624dc40ddbf"
2025-04-21T21:43:00.0915106Z 		},
2025-04-21T21:43:00.0915284Z 		{
2025-04-21T21:43:00.0915595Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0915999Z 			"size": 121477387,
2025-04-21T21:43:00.0916391Z 			"digest": "sha256:956077b157237c043dfffd54a6aee7e4f7d3e03584e329d427ab36bcf285b595"
2025-04-21T21:43:00.0916833Z 		},
2025-04-21T21:43:00.0917021Z 		{
2025-04-21T21:43:00.0917331Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0917710Z 			"size": 109,
2025-04-21T21:43:00.0918222Z 			"digest": "sha256:b840d58dab14ffab9220e7e5dba7751d9809c049d77a61b93924e4947527bb65"
2025-04-21T21:43:00.0918666Z 		},
2025-04-21T21:43:00.0918839Z 		{
2025-04-21T21:43:00.0919147Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0919537Z 			"size": 1472,
2025-04-21T21:43:00.0919919Z 			"digest": "sha256:135c1f91f3df5257432cc2d37486ab3cf2ff585d3a2ec4ac2304cb0d25e5fac1"
2025-04-21T21:43:00.0920362Z 		},
2025-04-21T21:43:00.0920541Z 		{
2025-04-21T21:43:00.0920843Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0921239Z 			"size": 429635236,
2025-04-21T21:43:00.0922095Z 			"digest": "sha256:2fba85b4232347228dbfdef797160ace15ad71face213ee558c1180161dfe7c6"
2025-04-21T21:43:00.0922546Z 		},
2025-04-21T21:43:00.0922721Z 		{
2025-04-21T21:43:00.0923039Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0923434Z 			"size": 162,
2025-04-21T21:43:00.0923816Z 			"digest": "sha256:7e377528c596bece391a67c4dfc9bab633557a7bd77e7a650595f6f4ef162b1a"
2025-04-21T21:43:00.0924308Z 		},
2025-04-21T21:43:00.0924504Z 		{
2025-04-21T21:43:00.0924808Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0925198Z 			"size": 801,
2025-04-21T21:43:00.0925591Z 			"digest": "sha256:7c1fe59ec87a4d7e8d2d9c14b3aa7254a4dbeb2566473fceaa3d6ef086a29e5a"
2025-04-21T21:43:00.0926038Z 		},
2025-04-21T21:43:00.0926225Z 		{
2025-04-21T21:43:00.0926532Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0926917Z 			"size": 35860969,
2025-04-21T21:43:00.0927327Z 			"digest": "sha256:bb83f375ed7d52c4af28b418a77af53bdb24511aad990c9e8f017c3b6145d401"
2025-04-21T21:43:00.0927778Z 		},
2025-04-21T21:43:00.0927960Z 		{
2025-04-21T21:43:00.0928270Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0928668Z 			"size": 104,
2025-04-21T21:43:00.0929049Z 			"digest": "sha256:d1ee3dcc43c51d791d948668680e601c3394068ce014a7395b2565facd8e1554"
2025-04-21T21:43:00.0929487Z 		},
2025-04-21T21:43:00.0929676Z 		{
2025-04-21T21:43:00.0929987Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0930377Z 			"size": 424,
2025-04-21T21:43:00.0930760Z 			"digest": "sha256:54687ddf5aab346564b45b9340d55679e17969e1a9290a068a71acd75e445a68"
2025-04-21T21:43:00.0931204Z 		},
2025-04-21T21:43:00.0931385Z 		{
2025-04-21T21:43:00.0931697Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0932096Z 			"size": 20262168,
2025-04-21T21:43:00.0932490Z 			"digest": "sha256:6b673e167d0fd859f7f81837936c87feb0f3540e9d83ccb685e01c2d2b1a105c"
2025-04-21T21:43:00.0932935Z 		},
2025-04-21T21:43:00.0933118Z 		{
2025-04-21T21:43:00.0933429Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0933993Z 			"size": 643,
2025-04-21T21:43:00.0934380Z 			"digest": "sha256:a4f475d6d4d727b2776f8de8fb347830ba056f3485b53102d7502bf63a1c08b4"
2025-04-21T21:43:00.0934826Z 		},
2025-04-21T21:43:00.0935001Z 		{
2025-04-21T21:43:00.0935309Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0935698Z 			"size": 701,
2025-04-21T21:43:00.0936078Z 			"digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03"
2025-04-21T21:43:00.0936521Z 		},
2025-04-21T21:43:00.0936703Z 		{
2025-04-21T21:43:00.0937005Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0937397Z 			"size": 141,
2025-04-21T21:43:00.0937784Z 			"digest": "sha256:48e60e8ea738df82c6a874ed4b7e587451261d7675d166a68e4b882ce5a932e7"
2025-04-21T21:43:00.0938221Z 		},
2025-04-21T21:43:00.0938393Z 		{
2025-04-21T21:43:00.0938702Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0939091Z 			"size": 134,
2025-04-21T21:43:00.0939482Z 			"digest": "sha256:d52c79949221ec29bacb5c73e1630c76a77c93a6dcd0a36570ef858cb7ff2491"
2025-04-21T21:43:00.0939925Z 		},
2025-04-21T21:43:00.0940107Z 		{
2025-04-21T21:43:00.0940535Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0940926Z 			"size": 5221737836,
2025-04-21T21:43:00.0941318Z 			"digest": "sha256:846d8aa2652360437687998e7786934ac4a2680471da233060f0e42acca36db8"
2025-04-21T21:43:00.0941736Z 		},
2025-04-21T21:43:00.0941918Z 		{
2025-04-21T21:43:00.0942231Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0942616Z 			"size": 194,
2025-04-21T21:43:00.0942994Z 			"digest": "sha256:595e9954171617ada6037b7869515bf0e2d1dff54b868e3ffba1f652ef8c56f5"
2025-04-21T21:43:00.0943435Z 		},
2025-04-21T21:43:00.0943626Z 		{
2025-04-21T21:43:00.0943938Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0944332Z 			"size": 563,
2025-04-21T21:43:00.0944740Z 			"digest": "sha256:013dfd09e68f59d76a79ddcc8b47d99f10ad62529abcd1725885de25818509ae"
2025-04-21T21:43:00.0945178Z 		},
2025-04-21T21:43:00.0945370Z 		{
2025-04-21T21:43:00.0945690Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0946086Z 			"size": 43164291,
2025-04-21T21:43:00.0946501Z 			"digest": "sha256:4402edece585e892e3dcd04507c4e9315cd4537bacd16ed748aebfec5e8af5e8"
2025-04-21T21:43:00.0946956Z 		},
2025-04-21T21:43:00.0947131Z 		{
2025-04-21T21:43:00.0947445Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0947847Z 			"size": 106,
2025-04-21T21:43:00.0948236Z 			"digest": "sha256:82aa68866e535166ba9028ae8d5fe7bf3d13ea4c69a43469aaefdffccc00b8a9"
2025-04-21T21:43:00.0948686Z 		},
2025-04-21T21:43:00.0948867Z 		{
2025-04-21T21:43:00.0949184Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0949570Z 			"size": 307403395,
2025-04-21T21:43:00.0949973Z 			"digest": "sha256:e624d917de213d9b06921538cf6d41f55585fc181f15c727920ca878e0d432c1"
2025-04-21T21:43:00.0950404Z 		},
2025-04-21T21:43:00.0950581Z 		{
2025-04-21T21:43:00.0950891Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0951294Z 			"size": 5761507135,
2025-04-21T21:43:00.0951700Z 			"digest": "sha256:79d2c9f506eccd71c934a8639db0b5fb46a4c03b1e4c68535c972001a0b3a1f6"
2025-04-21T21:43:00.0952162Z 		},
2025-04-21T21:43:00.0952346Z 		{
2025-04-21T21:43:00.0952648Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0953041Z 			"size": 127,
2025-04-21T21:43:00.0953480Z 			"digest": "sha256:3b25d713acb3675cc21cfaf3e73036b34d21dd039d0339ebd5874e41f29d4c73"
2025-04-21T21:43:00.0953926Z 		},
2025-04-21T21:43:00.0954100Z 		{
2025-04-21T21:43:00.0954411Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0954810Z 			"size": 1012,
2025-04-21T21:43:00.0955245Z 			"digest": "sha256:c0ad696e18d34210a240d661b9a6994f1fcf548fb2eeaeccef011fd9fbb01d79"
2025-04-21T21:43:00.0955782Z 		},
2025-04-21T21:43:00.0955966Z 		{
2025-04-21T21:43:00.0956273Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0956670Z 			"size": 701,
2025-04-21T21:43:00.0957064Z 			"digest": "sha256:4afa91cd0662787b5dfd97c4724a659e2b0dad7e1c058307ee96067d064b7e03"
2025-04-21T21:43:00.0957501Z 		},
2025-04-21T21:43:00.0957687Z 		{
2025-04-21T21:43:00.0958006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0958406Z 			"size": 135,
2025-04-21T21:43:00.0958789Z 			"digest": "sha256:767566b693177188e1a3b60bd320a787e121ffd7dd9f0cd79f75f94f17d2197b"
2025-04-21T21:43:00.0959232Z 		},
2025-04-21T21:43:00.0959417Z 		{
2025-04-21T21:43:00.0959724Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0960121Z 			"size": 32,
2025-04-21T21:43:00.0960521Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-21T21:43:00.0960974Z 		},
2025-04-21T21:43:00.0961170Z 		{
2025-04-21T21:43:00.0961490Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0961886Z 			"size": 159,
2025-04-21T21:43:00.0962366Z 			"digest": "sha256:d83ab53d6ab47f0836a1d834f483ea68f6bb74717f9b5f435958be607a1e25de"
2025-04-21T21:43:00.0962819Z 		},
2025-04-21T21:43:00.0962999Z 		{
2025-04-21T21:43:00.0963324Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0963724Z 			"size": 1898,
2025-04-21T21:43:00.0964137Z 			"digest": "sha256:e9ae24bb3b5b9eaace2576d0bc4ce0ccdea8f446743644ae2f5d99fa4d2704ed"
2025-04-21T21:43:00.0964588Z 		},
2025-04-21T21:43:00.0964772Z 		{
2025-04-21T21:43:00.0965087Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0965485Z 			"size": 237571532,
2025-04-21T21:43:00.0965887Z 			"digest": "sha256:8c11aa345fff3b36368b1fc7fc8268149038691d5e7a11054e8c029d0b9a6675"
2025-04-21T21:43:00.0966331Z 		},
2025-04-21T21:43:00.0966504Z 		{
2025-04-21T21:43:00.0966824Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0967216Z 			"size": 162,
2025-04-21T21:43:00.0967608Z 			"digest": "sha256:cc9ed3f492faeb8df7ef7ca746a2c85d1189181e3a2516a19c0126bd224ad5d9"
2025-04-21T21:43:00.0968063Z 		},
2025-04-21T21:43:00.0968244Z 		{
2025-04-21T21:43:00.0968551Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0968941Z 			"size": 7944,
2025-04-21T21:43:00.0969332Z 			"digest": "sha256:6133da88e6692508c12cae6643582782b9a424aa73892cf6e4c0befebe516269"
2025-04-21T21:43:00.0969767Z 		},
2025-04-21T21:43:00.0969941Z 		{
2025-04-21T21:43:00.0970253Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0970643Z 			"size": 8071,
2025-04-21T21:43:00.0971022Z 			"digest": "sha256:f55162ff6427c5c71e1737cb77e4c9913c679981ec16b241d92084fe621b74c9"
2025-04-21T21:43:00.0971457Z 		},
2025-04-21T21:43:00.0971639Z 		{
2025-04-21T21:43:00.0971949Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0972346Z 			"size": 304,
2025-04-21T21:43:00.0972738Z 			"digest": "sha256:f2fb53f519b30fceaf1192ac5221e9c1130032af58e6dd7312a7d82f6be9e66e"
2025-04-21T21:43:00.0973182Z 		},
2025-04-21T21:43:00.0973365Z 		{
2025-04-21T21:43:00.0973676Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0974080Z 			"size": 7631978,
2025-04-21T21:43:00.0974472Z 			"digest": "sha256:d93b86c61192d758035dbbef6102f4c29339a3bc439c06bde06e3e7efb73c071"
2025-04-21T21:43:00.0974913Z 		},
2025-04-21T21:43:00.0975096Z 		{
2025-04-21T21:43:00.0975403Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0975799Z 			"size": 108,
2025-04-21T21:43:00.0976198Z 			"digest": "sha256:2d75b00aef3ef33cd0a6aca03e1b81df01b8d5c874850e5ee0a580fdaeffafda"
2025-04-21T21:43:00.0976652Z 		},
2025-04-21T21:43:00.0976836Z 		{
2025-04-21T21:43:00.0977157Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0977644Z 			"size": 54145661,
2025-04-21T21:43:00.0978064Z 			"digest": "sha256:165cb697180f8d5f9e2677a69ad5431ab294be16a39bc423f0d8593513c041f0"
2025-04-21T21:43:00.0978520Z 		},
2025-04-21T21:43:00.0978701Z 		{
2025-04-21T21:43:00.0979021Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0979414Z 			"size": 495,
2025-04-21T21:43:00.0979794Z 			"digest": "sha256:952482b7353e3317938e0c2a2c86c92dc2301bec5f1683ae1fc4f72ef2e8ac4b"
2025-04-21T21:43:00.0980224Z 		},
2025-04-21T21:43:00.0980401Z 		{
2025-04-21T21:43:00.0980712Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0981100Z 			"size": 1179582084,
2025-04-21T21:43:00.0981507Z 			"digest": "sha256:f4fb942e12736de733008a26bdd886b81205f8a80595e32db7584540ac193776"
2025-04-21T21:43:00.0981954Z 		},
2025-04-21T21:43:00.0982131Z 		{
2025-04-21T21:43:00.0982447Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0982856Z 			"size": 106,
2025-04-21T21:43:00.0983258Z 			"digest": "sha256:22d7ad62e40c8f69aaedce849745730ac752c1b564e4a7d39bbae1f12e7f03c6"
2025-04-21T21:43:00.0983709Z 		},
2025-04-21T21:43:00.0983977Z 		{
2025-04-21T21:43:00.0984293Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0984687Z 			"size": 612,
2025-04-21T21:43:00.0985078Z 			"digest": "sha256:340ddaf5579e22644b795f56a17c3697533215e26d5761d5cca8bae92205e078"
2025-04-21T21:43:00.0985523Z 		},
2025-04-21T21:43:00.0985704Z 		{
2025-04-21T21:43:00.0986023Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0986423Z 			"size": 317359363,
2025-04-21T21:43:00.0986824Z 			"digest": "sha256:abb8b598f42085f2a06e7d8095a6ddb1407e5611c878d0ec1c605c6b62565e39"
2025-04-21T21:43:00.0987274Z 		},
2025-04-21T21:43:00.0987463Z 		{
2025-04-21T21:43:00.0987775Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0988167Z 			"size": 111,
2025-04-21T21:43:00.0988559Z 			"digest": "sha256:410b8ad7da03358202afe62f6f95a90ffc13b6c289c59f77d52078e0cfb3b427"
2025-04-21T21:43:00.0988998Z 		},
2025-04-21T21:43:00.0989172Z 		{
2025-04-21T21:43:00.0989480Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0989876Z 			"size": 475,
2025-04-21T21:43:00.0990258Z 			"digest": "sha256:79d2e9034f2b1b76f47e712bb3ab767c7b0a8aa2a9bdfc69526ea55345939d3c"
2025-04-21T21:43:00.0990700Z 		},
2025-04-21T21:43:00.0990882Z 		{
2025-04-21T21:43:00.0991188Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0991579Z 			"size": 180,
2025-04-21T21:43:00.0991956Z 			"digest": "sha256:c2128146ebc3a8a388e3ae687d3a8509470f980106006e2c07f01878a39cb1e2"
2025-04-21T21:43:00.0992389Z 		},
2025-04-21T21:43:00.0992575Z 		{
2025-04-21T21:43:00.0992886Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0993330Z 			"size": 529690496,
2025-04-21T21:43:00.0993743Z 			"digest": "sha256:4b1e8ee3de3882b3d9b74a17aa0895307d6b7f5aa4bde3fe3338dc7599d38fcf"
2025-04-21T21:43:00.0994187Z 		},
2025-04-21T21:43:00.0994366Z 		{
2025-04-21T21:43:00.0994672Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0995068Z 			"size": 183,
2025-04-21T21:43:00.0995461Z 			"digest": "sha256:02d23db28b3d929d4f9be01edc8371656c3215d2b4c08df0f24fb897e8b9c4eb"
2025-04-21T21:43:00.0995896Z 		},
2025-04-21T21:43:00.0996084Z 		{
2025-04-21T21:43:00.0996405Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0996788Z 			"size": 529,
2025-04-21T21:43:00.0997182Z 			"digest": "sha256:24d4ae1408767f7ceda609b9b1eef8ff3b3f7ee876a8a92ba1de0800db9328d1"
2025-04-21T21:43:00.0997639Z 		},
2025-04-21T21:43:00.0997815Z 		{
2025-04-21T21:43:00.0998132Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.0998526Z 			"size": 26134,
2025-04-21T21:43:00.0998918Z 			"digest": "sha256:2604f3be2ad176777da15e3755dea31789b0e584891d00680961ee3095a8b9e0"
2025-04-21T21:43:00.0999459Z 		},
2025-04-21T21:43:00.0999643Z 		{
2025-04-21T21:43:00.0999962Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.1000352Z 			"size": 106,
2025-04-21T21:43:00.1000742Z 			"digest": "sha256:ba564700c169e37253f9d67c6224e478ab8ee66a1906b540eaa06a77e3c5b75a"
2025-04-21T21:43:00.1001180Z 		},
2025-04-21T21:43:00.1001355Z 		{
2025-04-21T21:43:00.1001672Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.1002065Z 			"size": 32,
2025-04-21T21:43:00.1002448Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-21T21:43:00.1002892Z 		},
2025-04-21T21:43:00.1003075Z 		{
2025-04-21T21:43:00.1003380Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.1003769Z 			"size": 32,
2025-04-21T21:43:00.1004163Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-21T21:43:00.1004607Z 		},
2025-04-21T21:43:00.1004788Z 		{
2025-04-21T21:43:00.1005101Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.1005491Z 			"size": 32,
2025-04-21T21:43:00.1005959Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-21T21:43:00.1006401Z 		},
2025-04-21T21:43:00.1006584Z 		{
2025-04-21T21:43:00.1006894Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-04-21T21:43:00.1007283Z 			"size": 32,
2025-04-21T21:43:00.1007672Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-04-21T21:43:00.1008108Z 		}
2025-04-21T21:43:00.1008289Z 	]
2025-04-21T21:43:00.1008467Z }
2025-04-21T21:43:00.1008659Z + exit 0
2025-04-21T21:43:00.1045640Z ##[group]Run set -eux
2025-04-21T21:43:00.1045906Z set -eux
2025-04-21T21:43:00.1046734Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-04-21T21:43:00.1058512Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:43:00.1058875Z env:
2025-04-21T21:43:00.1059096Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:43:00.1059352Z ##[endgroup]
2025-04-21T21:43:00.1090887Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-04-21T21:43:00.1091358Z + jq --raw-output .SecretString
2025-04-21T21:43:00.1092919Z + jq -r .docker_hub_readonly_token
2025-04-21T21:43:00.1094578Z + docker login --username pytorchbot --password-stdin
2025-04-21T21:43:00.6900228Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-04-21T21:43:00.6900831Z Configure a credential helper to remove this warning. See
2025-04-21T21:43:00.6901384Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-04-21T21:43:00.6901832Z 
2025-04-21T21:43:00.6902728Z Login Succeeded
2025-04-21T21:43:00.7001092Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*/}
2025-04-21T21:43:00.7001459Z tag=${ECR_DOCKER_IMAGE##*/}
2025-04-21T21:43:00.7001841Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}"
2025-04-21T21:43:00.7010725Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:43:00.7011092Z env:
2025-04-21T21:43:00.7011305Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:43:00.7012153Z   ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:00.7013019Z ##[endgroup]
2025-04-21T21:43:00.7046034Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks-f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:00.7093698Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-04-21T21:43:00.7094122Z with:
2025-04-21T21:43:00.7094899Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:00.7095853Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:43:00.7096248Z env:
2025-04-21T21:43:00.7096463Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:43:00.7096719Z ##[endgroup]
2025-04-21T21:43:00.7125599Z ##[group]Run set -x
2025-04-21T21:43:00.7125863Z set -x
2025-04-21T21:43:00.7126088Z set +e
2025-04-21T21:43:00.7126308Z 
2025-04-21T21:43:00.7126514Z login() {
2025-04-21T21:43:00.7126978Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-04-21T21:43:00.7127488Z }
2025-04-21T21:43:00.7127696Z 
2025-04-21T21:43:00.7127944Z retry () {
2025-04-21T21:43:00.7128211Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-04-21T21:43:00.7128525Z }
2025-04-21T21:43:00.7128732Z 
2025-04-21T21:43:00.7128965Z retry login "${DOCKER_REGISTRY}"
2025-04-21T21:43:00.7129257Z 
2025-04-21T21:43:00.7129725Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-04-21T21:43:00.7130525Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-04-21T21:43:00.7130887Z 
2025-04-21T21:43:00.7131097Z set -e
2025-04-21T21:43:00.7131435Z # ignore output since only exit code is used for conditional
2025-04-21T21:43:00.7131918Z # only pull docker image if it's not available locally
2025-04-21T21:43:00.7132445Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-04-21T21:43:00.7132942Z   retry docker pull "${DOCKER_IMAGE}"
2025-04-21T21:43:00.7133253Z fi
2025-04-21T21:43:00.7141295Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:43:00.7141666Z env:
2025-04-21T21:43:00.7141887Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:43:00.7142712Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:00.7143677Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:43:00.7144059Z ##[endgroup]
2025-04-21T21:43:00.7169962Z + set +e
2025-04-21T21:43:00.7170599Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:43:00.7171042Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:43:00.7173671Z + aws ecr get-login-password --region us-east-1
2025-04-21T21:43:00.7175620Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-04-21T21:43:01.2590008Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2025-04-21T21:43:01.2590614Z Configure a credential helper to remove this warning. See
2025-04-21T21:43:01.2591258Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2025-04-21T21:43:01.2591714Z 
2025-04-21T21:43:01.2591815Z Login Succeeded
2025-04-21T21:43:01.2622164Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:01.2623160Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-04-21T21:43:01.5082971Z + IMAGE_SIZE=20670.273096084595
2025-04-21T21:43:01.5083343Z + echo 'Compressed size of image in MB: 20670.273096084595'
2025-04-21T21:43:01.5083690Z + set -e
2025-04-21T21:43:01.5083950Z Compressed size of image in MB: 20670.273096084595
2025-04-21T21:43:01.5085173Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:01.5237050Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:01.5238502Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:43:02.0639899Z f583c1656c56372ad0e1c57f3463a017fbf03138: Pulling from pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks
2025-04-21T21:43:02.0713601Z 86e5016c2693: Pulling fs layer
2025-04-21T21:43:02.0714459Z 49e139a3d6c2: Pulling fs layer
2025-04-21T21:43:02.0715004Z a14844c8c51f: Pulling fs layer
2025-04-21T21:43:02.0715340Z 18fb524087fb: Pulling fs layer
2025-04-21T21:43:02.0715657Z efd686a7b2c8: Pulling fs layer
2025-04-21T21:43:02.0716095Z 52c648e21334: Pulling fs layer
2025-04-21T21:43:02.0716512Z 56e384e4e5aa: Pulling fs layer
2025-04-21T21:43:02.0716903Z fb71b792ec6c: Pulling fs layer
2025-04-21T21:43:02.0717279Z 5509576f2693: Pulling fs layer
2025-04-21T21:43:02.0717654Z 1e6c6f2d2459: Pulling fs layer
2025-04-21T21:43:02.0718047Z a1fe8922734d: Pulling fs layer
2025-04-21T21:43:02.0718448Z 84e5433913c6: Pulling fs layer
2025-04-21T21:43:02.0718896Z c68e78d48b95: Pulling fs layer
2025-04-21T21:43:02.0719684Z b14b320c459c: Pulling fs layer
2025-04-21T21:43:02.0720036Z 52c648e21334: Waiting
2025-04-21T21:43:02.0720377Z 0d929bfaefee: Pulling fs layer
2025-04-21T21:43:02.0720799Z a5a8ea5a3eab: Pulling fs layer
2025-04-21T21:43:02.0721230Z 281f13ef6929: Pulling fs layer
2025-04-21T21:43:02.0721942Z a1fe8922734d: Waiting
2025-04-21T21:43:02.0722324Z a8695f1c94c3: Pulling fs layer
2025-04-21T21:43:02.0722725Z c72f323be801: Pulling fs layer
2025-04-21T21:43:02.0723012Z 84e5433913c6: Waiting
2025-04-21T21:43:02.0723249Z 5509576f2693: Waiting
2025-04-21T21:43:02.0723483Z 56e384e4e5aa: Waiting
2025-04-21T21:43:02.0723722Z 1e6c6f2d2459: Waiting
2025-04-21T21:43:02.0723958Z 18fb524087fb: Waiting
2025-04-21T21:43:02.0724185Z c68e78d48b95: Waiting
2025-04-21T21:43:02.0724420Z a5a8ea5a3eab: Waiting
2025-04-21T21:43:02.0724659Z efd686a7b2c8: Waiting
2025-04-21T21:43:02.0724909Z 4afa91cd0662: Pulling fs layer
2025-04-21T21:43:02.0725225Z c72f323be801: Waiting
2025-04-21T21:43:02.0725514Z a9bdca78eb50: Pulling fs layer
2025-04-21T21:43:02.0725876Z fb71b792ec6c: Waiting
2025-04-21T21:43:02.0726118Z 0d929bfaefee: Waiting
2025-04-21T21:43:02.0726356Z 4afa91cd0662: Waiting
2025-04-21T21:43:02.0726692Z a8695f1c94c3: Waiting
2025-04-21T21:43:02.0726948Z 87b133205740: Pulling fs layer
2025-04-21T21:43:02.0727225Z 1a1f3a6b6e28: Pulling fs layer
2025-04-21T21:43:02.0727516Z 1e726e5b69df: Pulling fs layer
2025-04-21T21:43:02.0738015Z a9bdca78eb50: Waiting
2025-04-21T21:43:02.0738302Z e55ee61cc4aa: Pulling fs layer
2025-04-21T21:43:02.0738590Z d2c9e5f6d068: Pulling fs layer
2025-04-21T21:43:02.0738876Z a9e23e3c0a18: Pulling fs layer
2025-04-21T21:43:02.0739147Z e55ee61cc4aa: Waiting
2025-04-21T21:43:02.0739392Z a9e23e3c0a18: Waiting
2025-04-21T21:43:02.0739624Z 281f13ef6929: Waiting
2025-04-21T21:43:02.0739876Z 956077b15723: Pulling fs layer
2025-04-21T21:43:02.0740156Z b840d58dab14: Pulling fs layer
2025-04-21T21:43:02.0740429Z 135c1f91f3df: Pulling fs layer
2025-04-21T21:43:02.0740706Z 2fba85b42323: Pulling fs layer
2025-04-21T21:43:02.0740985Z 7e377528c596: Pulling fs layer
2025-04-21T21:43:02.0741263Z 7c1fe59ec87a: Pulling fs layer
2025-04-21T21:43:02.0741536Z bb83f375ed7d: Pulling fs layer
2025-04-21T21:43:02.0741818Z d1ee3dcc43c5: Pulling fs layer
2025-04-21T21:43:02.0742086Z 7e377528c596: Waiting
2025-04-21T21:43:02.0742327Z 54687ddf5aab: Pulling fs layer
2025-04-21T21:43:02.0742633Z 6b673e167d0f: Pulling fs layer
2025-04-21T21:43:02.0742906Z 2fba85b42323: Waiting
2025-04-21T21:43:02.0743155Z a4f475d6d4d7: Pulling fs layer
2025-04-21T21:43:02.0743411Z 87b133205740: Waiting
2025-04-21T21:43:02.0743912Z 7c1fe59ec87a: Waiting
2025-04-21T21:43:02.0744157Z 1e726e5b69df: Waiting
2025-04-21T21:43:02.0744399Z 48e60e8ea738: Pulling fs layer
2025-04-21T21:43:02.0744661Z 54687ddf5aab: Waiting
2025-04-21T21:43:02.0744897Z 6b673e167d0f: Waiting
2025-04-21T21:43:02.0745134Z d52c79949221: Pulling fs layer
2025-04-21T21:43:02.0745410Z 846d8aa26523: Pulling fs layer
2025-04-21T21:43:02.0745675Z a4f475d6d4d7: Waiting
2025-04-21T21:43:02.0745919Z 595e99541716: Pulling fs layer
2025-04-21T21:43:02.0746194Z 013dfd09e68f: Pulling fs layer
2025-04-21T21:43:02.0746471Z 4402edece585: Pulling fs layer
2025-04-21T21:43:02.0746744Z 82aa68866e53: Pulling fs layer
2025-04-21T21:43:02.0747008Z b840d58dab14: Waiting
2025-04-21T21:43:02.0747259Z e624d917de21: Pulling fs layer
2025-04-21T21:43:02.0747524Z b14b320c459c: Waiting
2025-04-21T21:43:02.0747758Z 135c1f91f3df: Waiting
2025-04-21T21:43:02.0747994Z 846d8aa26523: Waiting
2025-04-21T21:43:02.0748242Z 79d2c9f506ec: Pulling fs layer
2025-04-21T21:43:02.0748508Z 4402edece585: Waiting
2025-04-21T21:43:02.0748759Z 3b25d713acb3: Pulling fs layer
2025-04-21T21:43:02.0749038Z c0ad696e18d3: Pulling fs layer
2025-04-21T21:43:02.0749291Z 48e60e8ea738: Waiting
2025-04-21T21:43:02.0749538Z 013dfd09e68f: Waiting
2025-04-21T21:43:02.0749783Z 767566b69317: Pulling fs layer
2025-04-21T21:43:02.0750038Z 79d2c9f506ec: Waiting
2025-04-21T21:43:02.0750275Z d1ee3dcc43c5: Waiting
2025-04-21T21:43:02.0750526Z 4f4fb700ef54: Pulling fs layer
2025-04-21T21:43:02.0750949Z d83ab53d6ab4: Pulling fs layer
2025-04-21T21:43:02.0751231Z e9ae24bb3b5b: Pulling fs layer
2025-04-21T21:43:02.0751501Z 4f4fb700ef54: Waiting
2025-04-21T21:43:02.0751745Z 8c11aa345fff: Pulling fs layer
2025-04-21T21:43:02.0752032Z cc9ed3f492fa: Pulling fs layer
2025-04-21T21:43:02.0752300Z d83ab53d6ab4: Waiting
2025-04-21T21:43:02.0752532Z 8c11aa345fff: Waiting
2025-04-21T21:43:02.0752782Z 6133da88e669: Pulling fs layer
2025-04-21T21:43:02.0753058Z f55162ff6427: Pulling fs layer
2025-04-21T21:43:02.0753417Z 3b25d713acb3: Waiting
2025-04-21T21:43:02.0753655Z 6133da88e669: Waiting
2025-04-21T21:43:02.0753911Z f2fb53f519b3: Pulling fs layer
2025-04-21T21:43:02.0754185Z f55162ff6427: Waiting
2025-04-21T21:43:02.0754425Z d93b86c61192: Pulling fs layer
2025-04-21T21:43:02.0754698Z cc9ed3f492fa: Waiting
2025-04-21T21:43:02.0754954Z 2d75b00aef3e: Pulling fs layer
2025-04-21T21:43:02.0755227Z 165cb697180f: Pulling fs layer
2025-04-21T21:43:02.0755493Z bb83f375ed7d: Waiting
2025-04-21T21:43:02.0755748Z 952482b7353e: Pulling fs layer
2025-04-21T21:43:02.0756008Z 2d75b00aef3e: Waiting
2025-04-21T21:43:02.0756247Z 952482b7353e: Waiting
2025-04-21T21:43:02.0756493Z f4fb942e1273: Pulling fs layer
2025-04-21T21:43:02.0756761Z 22d7ad62e40c: Pulling fs layer
2025-04-21T21:43:02.0757030Z f4fb942e1273: Waiting
2025-04-21T21:43:02.0757280Z 340ddaf5579e: Pulling fs layer
2025-04-21T21:43:02.0757538Z 165cb697180f: Waiting
2025-04-21T21:43:02.0757777Z 22d7ad62e40c: Waiting
2025-04-21T21:43:02.0758036Z abb8b598f420: Pulling fs layer
2025-04-21T21:43:02.0758319Z 410b8ad7da03: Pulling fs layer
2025-04-21T21:43:02.0758599Z 79d2e9034f2b: Pulling fs layer
2025-04-21T21:43:02.0758884Z c2128146ebc3: Pulling fs layer
2025-04-21T21:43:02.0759151Z abb8b598f420: Waiting
2025-04-21T21:43:02.0759389Z 410b8ad7da03: Waiting
2025-04-21T21:43:02.0759643Z 4b1e8ee3de38: Pulling fs layer
2025-04-21T21:43:02.0759933Z 02d23db28b3d: Pulling fs layer
2025-04-21T21:43:02.0760203Z 24d4ae140876: Pulling fs layer
2025-04-21T21:43:02.0760484Z 2604f3be2ad1: Pulling fs layer
2025-04-21T21:43:02.0760754Z 02d23db28b3d: Waiting
2025-04-21T21:43:02.0760983Z 24d4ae140876: Waiting
2025-04-21T21:43:02.0761222Z 79d2e9034f2b: Waiting
2025-04-21T21:43:02.0761468Z ba564700c169: Pulling fs layer
2025-04-21T21:43:02.0761728Z 2604f3be2ad1: Waiting
2025-04-21T21:43:02.0761972Z 4b1e8ee3de38: Waiting
2025-04-21T21:43:02.0762210Z ba564700c169: Waiting
2025-04-21T21:43:02.2240098Z 49e139a3d6c2: Verifying Checksum
2025-04-21T21:43:02.2240578Z 49e139a3d6c2: Download complete
2025-04-21T21:43:02.3120318Z 18fb524087fb: Verifying Checksum
2025-04-21T21:43:02.3120867Z 18fb524087fb: Download complete
2025-04-21T21:43:02.3836132Z efd686a7b2c8: Verifying Checksum
2025-04-21T21:43:02.3836512Z efd686a7b2c8: Download complete
2025-04-21T21:43:02.4297742Z 86e5016c2693: Verifying Checksum
2025-04-21T21:43:02.4298060Z 86e5016c2693: Download complete
2025-04-21T21:43:02.5249422Z 56e384e4e5aa: Verifying Checksum
2025-04-21T21:43:02.6182736Z fb71b792ec6c: Verifying Checksum
2025-04-21T21:43:02.6183082Z fb71b792ec6c: Download complete
2025-04-21T21:43:02.6992965Z 5509576f2693: Verifying Checksum
2025-04-21T21:43:02.6994319Z 5509576f2693: Download complete
2025-04-21T21:43:02.7241403Z a14844c8c51f: Verifying Checksum
2025-04-21T21:43:02.7241806Z a14844c8c51f: Download complete
2025-04-21T21:43:02.8040012Z a1fe8922734d: Verifying Checksum
2025-04-21T21:43:02.8040493Z a1fe8922734d: Download complete
2025-04-21T21:43:02.8727417Z 84e5433913c6: Verifying Checksum
2025-04-21T21:43:02.8727872Z 84e5433913c6: Download complete
2025-04-21T21:43:03.6860493Z 86e5016c2693: Pull complete
2025-04-21T21:43:04.1543682Z 49e139a3d6c2: Pull complete
2025-04-21T21:43:05.1301871Z a14844c8c51f: Pull complete
2025-04-21T21:43:05.2158774Z 18fb524087fb: Pull complete
2025-04-21T21:43:05.2763458Z efd686a7b2c8: Pull complete
2025-04-21T21:43:05.4095157Z c68e78d48b95: Verifying Checksum
2025-04-21T21:43:05.4095558Z c68e78d48b95: Download complete
2025-04-21T21:43:05.5289895Z b14b320c459c: Verifying Checksum
2025-04-21T21:43:05.5290696Z b14b320c459c: Download complete
2025-04-21T21:43:05.6015379Z 0d929bfaefee: Download complete
2025-04-21T21:43:05.6965503Z a5a8ea5a3eab: Verifying Checksum
2025-04-21T21:43:05.6965819Z a5a8ea5a3eab: Download complete
2025-04-21T21:43:06.7295628Z 281f13ef6929: Verifying Checksum
2025-04-21T21:43:06.7295980Z 281f13ef6929: Download complete
2025-04-21T21:43:06.7944601Z a8695f1c94c3: Download complete
2025-04-21T21:43:06.8890970Z c72f323be801: Verifying Checksum
2025-04-21T21:43:06.9802466Z c72f323be801: Download complete
2025-04-21T21:43:06.9802838Z 4afa91cd0662: Verifying Checksum
2025-04-21T21:43:06.9803217Z 4afa91cd0662: Download complete
2025-04-21T21:43:07.0730967Z a9bdca78eb50: Verifying Checksum
2025-04-21T21:43:07.0731471Z a9bdca78eb50: Download complete
2025-04-21T21:43:17.3022857Z 52c648e21334: Verifying Checksum
2025-04-21T21:43:17.3023210Z 52c648e21334: Download complete
2025-04-21T21:43:17.4003881Z 1a1f3a6b6e28: Verifying Checksum
2025-04-21T21:43:17.4004263Z 1a1f3a6b6e28: Download complete
2025-04-21T21:43:17.4973676Z 1e726e5b69df: Verifying Checksum
2025-04-21T21:43:17.4974009Z 1e726e5b69df: Download complete
2025-04-21T21:43:17.6100348Z e55ee61cc4aa: Verifying Checksum
2025-04-21T21:43:17.6100706Z e55ee61cc4aa: Download complete
2025-04-21T21:43:17.6931759Z d2c9e5f6d068: Download complete
2025-04-21T21:43:17.8403071Z a9e23e3c0a18: Download complete
2025-04-21T21:43:20.2143101Z 956077b15723: Verifying Checksum
2025-04-21T21:43:20.2143487Z 956077b15723: Download complete
2025-04-21T21:43:20.3310270Z b840d58dab14: Download complete
2025-04-21T21:43:20.4243237Z 135c1f91f3df: Verifying Checksum
2025-04-21T21:43:20.4243659Z 135c1f91f3df: Download complete
2025-04-21T21:43:27.2114238Z 2fba85b42323: Verifying Checksum
2025-04-21T21:43:27.2114586Z 2fba85b42323: Download complete
2025-04-21T21:43:27.3117885Z 7e377528c596: Download complete
2025-04-21T21:43:27.4043982Z 7c1fe59ec87a: Verifying Checksum
2025-04-21T21:43:27.4044339Z 7c1fe59ec87a: Download complete
2025-04-21T21:43:28.4182501Z bb83f375ed7d: Verifying Checksum
2025-04-21T21:43:28.4182864Z bb83f375ed7d: Download complete
2025-04-21T21:43:28.5231640Z d1ee3dcc43c5: Verifying Checksum
2025-04-21T21:43:28.5231982Z d1ee3dcc43c5: Download complete
2025-04-21T21:43:28.6031195Z 54687ddf5aab: Verifying Checksum
2025-04-21T21:43:28.6031704Z 54687ddf5aab: Download complete
2025-04-21T21:43:28.9659212Z 6b673e167d0f: Verifying Checksum
2025-04-21T21:43:28.9659635Z 6b673e167d0f: Download complete
2025-04-21T21:43:28.9690720Z 1e6c6f2d2459: Verifying Checksum
2025-04-21T21:43:28.9691425Z 1e6c6f2d2459: Download complete
2025-04-21T21:43:29.0549086Z a4f475d6d4d7: Verifying Checksum
2025-04-21T21:43:29.0549515Z a4f475d6d4d7: Download complete
2025-04-21T21:43:29.0628836Z 48e60e8ea738: Verifying Checksum
2025-04-21T21:43:29.0629145Z 48e60e8ea738: Download complete
2025-04-21T21:43:29.1520889Z d52c79949221: Verifying Checksum
2025-04-21T21:43:29.1521282Z d52c79949221: Download complete
2025-04-21T21:43:29.2428314Z 595e99541716: Verifying Checksum
2025-04-21T21:43:29.2429766Z 595e99541716: Download complete
2025-04-21T21:43:29.3184985Z 013dfd09e68f: Verifying Checksum
2025-04-21T21:43:29.3185457Z 013dfd09e68f: Download complete
2025-04-21T21:43:29.5844689Z 52c648e21334: Pull complete
2025-04-21T21:43:29.7880831Z 56e384e4e5aa: Pull complete
2025-04-21T21:43:30.0015207Z fb71b792ec6c: Pull complete
2025-04-21T21:43:30.0172707Z 4402edece585: Verifying Checksum
2025-04-21T21:43:30.0173138Z 4402edece585: Download complete
2025-04-21T21:43:30.1064453Z 82aa68866e53: Verifying Checksum
2025-04-21T21:43:30.1065332Z 82aa68866e53: Download complete
2025-04-21T21:43:30.1811810Z 5509576f2693: Pull complete
2025-04-21T21:43:35.3442120Z e624d917de21: Verifying Checksum
2025-04-21T21:43:35.3442570Z e624d917de21: Download complete
2025-04-21T21:43:53.0880570Z 87b133205740: Verifying Checksum
2025-04-21T21:43:53.0881039Z 87b133205740: Download complete
2025-04-21T21:43:53.1854845Z 3b25d713acb3: Download complete
2025-04-21T21:43:53.3046173Z c0ad696e18d3: Verifying Checksum
2025-04-21T21:43:53.3046994Z c0ad696e18d3: Download complete
2025-04-21T21:43:53.3874112Z 767566b69317: Verifying Checksum
2025-04-21T21:43:53.3874565Z 767566b69317: Download complete
2025-04-21T21:43:53.3941788Z 4f4fb700ef54: Download complete
2025-04-21T21:43:53.4949420Z d83ab53d6ab4: Verifying Checksum
2025-04-21T21:43:53.4949901Z d83ab53d6ab4: Download complete
2025-04-21T21:43:53.5631834Z e9ae24bb3b5b: Verifying Checksum
2025-04-21T21:43:53.5632263Z e9ae24bb3b5b: Download complete
2025-04-21T21:43:59.4029257Z 8c11aa345fff: Verifying Checksum
2025-04-21T21:43:59.4029748Z 8c11aa345fff: Download complete
2025-04-21T21:43:59.4957807Z cc9ed3f492fa: Download complete
2025-04-21T21:43:59.5814492Z 6133da88e669: Verifying Checksum
2025-04-21T21:43:59.5815395Z 6133da88e669: Download complete
2025-04-21T21:43:59.6613198Z f55162ff6427: Verifying Checksum
2025-04-21T21:43:59.6613626Z f55162ff6427: Download complete
2025-04-21T21:43:59.7390451Z f2fb53f519b3: Download complete
2025-04-21T21:44:00.1327132Z d93b86c61192: Download complete
2025-04-21T21:44:00.2283375Z 2d75b00aef3e: Download complete
2025-04-21T21:44:02.3227621Z 165cb697180f: Verifying Checksum
2025-04-21T21:44:02.3228255Z 165cb697180f: Download complete
2025-04-21T21:44:02.4133938Z 952482b7353e: Download complete
2025-04-21T21:44:21.6764204Z 1e6c6f2d2459: Pull complete
2025-04-21T21:44:21.8657725Z a1fe8922734d: Pull complete
2025-04-21T21:44:22.0531431Z 84e5433913c6: Pull complete
2025-04-21T21:44:32.9144066Z f4fb942e1273: Verifying Checksum
2025-04-21T21:44:32.9144400Z f4fb942e1273: Download complete
2025-04-21T21:44:33.0153223Z 22d7ad62e40c: Verifying Checksum
2025-04-21T21:44:33.0155656Z 22d7ad62e40c: Download complete
2025-04-21T21:44:33.1239287Z 340ddaf5579e: Download complete
2025-04-21T21:44:41.3538104Z abb8b598f420: Verifying Checksum
2025-04-21T21:44:41.3538477Z abb8b598f420: Download complete
2025-04-21T21:44:41.4887640Z 410b8ad7da03: Verifying Checksum
2025-04-21T21:44:41.4887976Z 410b8ad7da03: Download complete
2025-04-21T21:44:41.5645870Z 79d2e9034f2b: Download complete
2025-04-21T21:44:41.6674790Z c2128146ebc3: Verifying Checksum
2025-04-21T21:44:41.6675106Z c2128146ebc3: Download complete
2025-04-21T21:44:55.1352236Z 4b1e8ee3de38: Verifying Checksum
2025-04-21T21:44:55.1352588Z 4b1e8ee3de38: Download complete
2025-04-21T21:44:55.2556316Z 02d23db28b3d: Verifying Checksum
2025-04-21T21:44:55.2556660Z 02d23db28b3d: Download complete
2025-04-21T21:44:55.3756674Z 24d4ae140876: Download complete
2025-04-21T21:44:55.5228657Z 2604f3be2ad1: Verifying Checksum
2025-04-21T21:44:55.5229419Z 2604f3be2ad1: Download complete
2025-04-21T21:44:55.6706108Z ba564700c169: Download complete
2025-04-21T21:45:11.0313017Z c68e78d48b95: Pull complete
2025-04-21T21:45:11.2642336Z b14b320c459c: Pull complete
2025-04-21T21:45:11.5096704Z 0d929bfaefee: Pull complete
2025-04-21T21:45:11.7367339Z a5a8ea5a3eab: Pull complete
2025-04-21T21:45:16.8850956Z 846d8aa26523: Verifying Checksum
2025-04-21T21:45:16.8851926Z 846d8aa26523: Download complete
2025-04-21T21:45:22.6752767Z 281f13ef6929: Pull complete
2025-04-21T21:45:22.7117175Z a8695f1c94c3: Pull complete
2025-04-21T21:45:22.7749340Z c72f323be801: Pull complete
2025-04-21T21:45:22.8124943Z 4afa91cd0662: Pull complete
2025-04-21T21:45:22.8453998Z a9bdca78eb50: Pull complete
2025-04-21T21:45:33.9789533Z 79d2c9f506ec: Verifying Checksum
2025-04-21T21:45:33.9790007Z 79d2c9f506ec: Download complete
2025-04-21T21:46:27.1258521Z 87b133205740: Pull complete
2025-04-21T21:46:27.3567683Z 1a1f3a6b6e28: Pull complete
2025-04-21T21:46:27.6026037Z 1e726e5b69df: Pull complete
2025-04-21T21:46:27.8438896Z e55ee61cc4aa: Pull complete
2025-04-21T21:46:28.0810683Z d2c9e5f6d068: Pull complete
2025-04-21T21:46:28.2979343Z a9e23e3c0a18: Pull complete
2025-04-21T21:46:30.7394565Z 956077b15723: Pull complete
2025-04-21T21:46:30.8769427Z b840d58dab14: Pull complete
2025-04-21T21:46:31.0151791Z 135c1f91f3df: Pull complete
2025-04-21T21:46:39.2979731Z 2fba85b42323: Pull complete
2025-04-21T21:46:39.4937076Z 7e377528c596: Pull complete
2025-04-21T21:46:39.6854403Z 7c1fe59ec87a: Pull complete
2025-04-21T21:46:40.4429954Z bb83f375ed7d: Pull complete
2025-04-21T21:46:40.6645876Z d1ee3dcc43c5: Pull complete
2025-04-21T21:46:40.8723056Z 54687ddf5aab: Pull complete
2025-04-21T21:46:41.3263451Z 6b673e167d0f: Pull complete
2025-04-21T21:46:41.4259720Z a4f475d6d4d7: Pull complete
2025-04-21T21:46:41.6984139Z 48e60e8ea738: Pull complete
2025-04-21T21:46:41.8600483Z d52c79949221: Pull complete
2025-04-21T21:47:20.6417435Z 846d8aa26523: Pull complete
2025-04-21T21:47:20.8766609Z 595e99541716: Pull complete
2025-04-21T21:47:21.1124177Z 013dfd09e68f: Pull complete
2025-04-21T21:47:23.6745352Z 4402edece585: Pull complete
2025-04-21T21:47:23.8997176Z 82aa68866e53: Pull complete
2025-04-21T21:47:25.2702950Z e624d917de21: Pull complete
2025-04-21T21:48:55.0330064Z 79d2c9f506ec: Pull complete
2025-04-21T21:48:55.1499276Z 3b25d713acb3: Pull complete
2025-04-21T21:48:55.2235621Z c0ad696e18d3: Pull complete
2025-04-21T21:48:55.2659197Z 767566b69317: Pull complete
2025-04-21T21:48:55.2871494Z 4f4fb700ef54: Pull complete
2025-04-21T21:48:55.3094760Z d83ab53d6ab4: Pull complete
2025-04-21T21:48:55.3313795Z e9ae24bb3b5b: Pull complete
2025-04-21T21:49:02.3518400Z 8c11aa345fff: Pull complete
2025-04-21T21:49:02.5735497Z cc9ed3f492fa: Pull complete
2025-04-21T21:49:02.8087531Z 6133da88e669: Pull complete
2025-04-21T21:49:03.0462765Z f55162ff6427: Pull complete
2025-04-21T21:49:03.2949757Z f2fb53f519b3: Pull complete
2025-04-21T21:49:04.6543223Z d93b86c61192: Pull complete
2025-04-21T21:49:04.9012285Z 2d75b00aef3e: Pull complete
2025-04-21T21:49:06.6289139Z 165cb697180f: Pull complete
2025-04-21T21:49:06.8641960Z 952482b7353e: Pull complete
2025-04-21T21:49:17.4221318Z f4fb942e1273: Pull complete
2025-04-21T21:49:17.6557270Z 22d7ad62e40c: Pull complete
2025-04-21T21:49:17.8797800Z 340ddaf5579e: Pull complete
2025-04-21T21:49:20.9118414Z abb8b598f420: Pull complete
2025-04-21T21:49:21.1330330Z 410b8ad7da03: Pull complete
2025-04-21T21:49:21.3537278Z 79d2e9034f2b: Pull complete
2025-04-21T21:49:21.5738415Z c2128146ebc3: Pull complete
2025-04-21T21:49:25.8906673Z 4b1e8ee3de38: Pull complete
2025-04-21T21:49:26.1259416Z 02d23db28b3d: Pull complete
2025-04-21T21:49:26.3671185Z 24d4ae140876: Pull complete
2025-04-21T21:49:26.6084528Z 2604f3be2ad1: Pull complete
2025-04-21T21:49:26.8359748Z ba564700c169: Pull complete
2025-04-21T21:49:27.9350103Z Digest: sha256:745e08885687216ce5adf3196dea7f49a9467042491210addb990693d7fef90a
2025-04-21T21:49:27.9812537Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:49:28.0022920Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:49:28.0081905Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:49:28.0082824Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-04-21T21:49:28.0093921Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:49:28.0094316Z env:
2025-04-21T21:49:28.0094555Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:49:28.0094809Z ##[endgroup]
2025-04-21T21:49:28.0360456Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2025-04-21T21:49:28.0360872Z with:
2025-04-21T21:49:28.0361089Z   driver-version: 570.133.07
2025-04-21T21:49:28.0361347Z env:
2025-04-21T21:49:28.0361565Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:49:28.0361817Z ##[endgroup]
2025-04-21T21:49:28.0461545Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2025-04-21T21:49:28.0461955Z with:
2025-04-21T21:49:28.0462167Z   timeout_minutes: 10
2025-04-21T21:49:28.0462413Z   max_attempts: 3
2025-04-21T21:49:28.0486845Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        if [[ "${DISTRIBUTION}" == "amzn2023" ]] ; then
          YUM_REPO_URL="https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo"
        else
          # Amazon Linux 2
          YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"
        fi

        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        # Install nvidia-driver package if not installed
        status="$(dpkg-query -W --showformat='${db:Status-Status}' nvidia-docker2 2>&1)"
        if [ ! $? = 0 ] || [ ! "$status" = installed ]; then
          sudo apt-get install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
          sudo systemctl restart docker
        fi
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"

                # Turn off persistent mode so that the installation script can unload the kernel module
                sudo killall nvidia-persistenced || true
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e

            nvidia-smi
            # NB: Annoyingly, nvidia-smi command returns successfully with return code 0 even in
            # the case where the driver has already crashed as it still can get the driver version
            # and some basic information like the bus ID.  However, the rest of the information
            # would be missing (ERR!), for example:
            #
            # +-----------------------------------------------------------------------------+
            # | NVIDIA-SMI 525.89.02    Driver Version: 525.89.02    CUDA Version: 12.0     |
            # |-------------------------------+----------------------+----------------------+
            # | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
            # | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
            # |                               |                      |               MIG M. |
            # |===============================+======================+======================|
            # |   0  ERR!                Off  | 00000000:00:1E.0 Off |                 ERR! |
            # |ERR!  ERR! ERR!    ERR! / ERR! |   4184MiB / 23028MiB |    ERR!      Default |
            # |                               |                      |                 ERR! |
            # +-------------------------------+----------------------+----------------------+
            #
            # +-----------------------------------------------------------------------------+
            # | Processes:                                                                  |
            # |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
            # |        ID   ID                                                   Usage      |
            # |=============================================================================|
            # +-----------------------------------------------------------------------------+
            #
            # This should be reported as a failure instead as it will guarantee to fail when
            # Docker tries to run with --gpus all
            #
            # So, the correct check here is to query one of the missing piece of info like
            # GPU name, so that the command can fail accordingly
            nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all -e NVIDIA_DRIVER_CAPABILITIES=all" >> "${GITHUB_ENV}"

# Fix https://github.com/NVIDIA/nvidia-docker/issues/1648 on runners with
# more than one GPUs. This just needs to be run once. The command fails
# on subsequent runs and complains that the mode is already on, but that's
# ok
sudo nvidia-persistenced || true
# This should show persistence mode ON
nvidia-smi

2025-04-21T21:49:28.0511494Z   retry_wait_seconds: 10
2025-04-21T21:49:28.0511766Z   polling_interval_seconds: 1
2025-04-21T21:49:28.0512046Z   warning_on_retry: true
2025-04-21T21:49:28.0512315Z   continue_on_error: false
2025-04-21T21:49:28.0512568Z env:
2025-04-21T21:49:28.0512779Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:49:28.0513041Z   DRIVER_VERSION: 570.133.07
2025-04-21T21:49:28.0513342Z ##[endgroup]
2025-04-21T21:49:28.1377446Z == Installing nvidia driver NVIDIA-Linux-x86_64-570.133.07.run ==
2025-04-21T21:49:28.1378905Z + pre_install_nvidia_driver_amzn2
2025-04-21T21:49:28.1379432Z + sudo yum remove -y nvidia-driver-latest-dkms
2025-04-21T21:49:28.9888910Z No match for argument: nvidia-driver-latest-dkms
2025-04-21T21:49:28.9889524Z No packages marked for removal.
2025-04-21T21:49:29.0440310Z Dependencies resolved.
2025-04-21T21:49:29.0449987Z Nothing to do.
2025-04-21T21:49:29.0450543Z Complete!
2025-04-21T21:49:29.1061584Z + install_nvidia_driver_common
2025-04-21T21:49:29.1065348Z + echo 'Before installing NVIDIA driver'
2025-04-21T21:49:29.1065776Z + lspci
2025-04-21T21:49:29.1067208Z Before installing NVIDIA driver
2025-04-21T21:49:29.1190708Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2025-04-21T21:49:29.1191388Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2025-04-21T21:49:29.1192019Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2025-04-21T21:49:29.1192802Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2025-04-21T21:49:29.1193601Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2025-04-21T21:49:29.1194344Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2025-04-21T21:49:29.1195439Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2025-04-21T21:49:29.1196150Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2025-04-21T21:49:29.1196723Z + lsmod
2025-04-21T21:49:29.1321957Z Module                  Size  Used by
2025-04-21T21:49:29.1322283Z veth                   36864  0
2025-04-21T21:49:29.1322562Z nvidia_modeset       1716224  0
2025-04-21T21:49:29.1322928Z video                  65536  1 nvidia_modeset
2025-04-21T21:49:29.1323259Z wmi                    36864  1 video
2025-04-21T21:49:29.1323539Z nvidia_uvm           1884160  0
2025-04-21T21:49:29.1323843Z nvidia              11583488  7 nvidia_uvm,nvidia_modeset
2025-04-21T21:49:29.1324477Z drm                   602112  1 nvidia
2025-04-21T21:49:29.1324802Z drm_panel_orientation_quirks    32768  1 drm
2025-04-21T21:49:29.1325234Z backlight              24576  3 video,drm,nvidia_modeset
2025-04-21T21:49:29.1325612Z i2c_core              110592  2 nvidia,drm
2025-04-21T21:49:29.1325911Z xt_conntrack           16384  1
2025-04-21T21:49:29.1326193Z nft_chain_nat          16384  3
2025-04-21T21:49:29.1326456Z xt_MASQUERADE          20480  1
2025-04-21T21:49:29.1326763Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2025-04-21T21:49:29.1327114Z nf_conntrack_netlink    57344  0
2025-04-21T21:49:29.1327518Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2025-04-21T21:49:29.1327980Z nf_defrag_ipv6         24576  1 nf_conntrack
2025-04-21T21:49:29.1328308Z nf_defrag_ipv4         16384  1 nf_conntrack
2025-04-21T21:49:29.1328614Z xfrm_user              57344  1
2025-04-21T21:49:29.1328995Z xfrm_algo              16384  1 xfrm_user
2025-04-21T21:49:29.1329297Z xt_addrtype            16384  2
2025-04-21T21:49:29.1329565Z nft_compat             20480  4
2025-04-21T21:49:29.1329875Z nf_tables             311296  57 nft_compat,nft_chain_nat
2025-04-21T21:49:29.1330301Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2025-04-21T21:49:29.1330699Z br_netfilter           36864  0
2025-04-21T21:49:29.1330982Z bridge                323584  1 br_netfilter
2025-04-21T21:49:29.1331288Z stp                    16384  1 bridge
2025-04-21T21:49:29.1331584Z llc                    16384  2 bridge,stp
2025-04-21T21:49:29.1331869Z overlay               167936  0
2025-04-21T21:49:29.1332132Z tls                   135168  0
2025-04-21T21:49:29.1332394Z nls_ascii              16384  1
2025-04-21T21:49:29.1332659Z nls_cp437              20480  1
2025-04-21T21:49:29.1332908Z vfat                   24576  1
2025-04-21T21:49:29.1333166Z fat                    86016  1 vfat
2025-04-21T21:49:29.1333446Z sunrpc                696320  1
2025-04-21T21:49:29.1333699Z ena                   180224  0
2025-04-21T21:49:29.1333953Z i8042                  45056  0
2025-04-21T21:49:29.1334218Z serio                  28672  3 i8042
2025-04-21T21:49:29.1334495Z ghash_clmulni_intel    16384  0
2025-04-21T21:49:29.1334767Z button                 24576  0
2025-04-21T21:49:29.1335079Z sch_fq_codel           20480  17
2025-04-21T21:49:29.1335347Z dm_mod                188416  0
2025-04-21T21:49:29.1335608Z dax                    45056  1 dm_mod
2025-04-21T21:49:29.1335886Z loop                   36864  0
2025-04-21T21:49:29.1336136Z fuse                  163840  1
2025-04-21T21:49:29.1336392Z configfs               57344  1
2025-04-21T21:49:29.1336651Z dmi_sysfs              20480  0
2025-04-21T21:49:29.1336903Z crc32_pclmul           16384  0
2025-04-21T21:49:29.1337165Z crc32c_intel           24576  0
2025-04-21T21:49:29.1337424Z efivarfs               24576  1
2025-04-21T21:49:29.1337675Z + modinfo nvidia
2025-04-21T21:49:29.1345365Z filename:       /lib/modules/6.1.130-139.222.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2025-04-21T21:49:29.1345951Z import_ns:      DMA_BUF
2025-04-21T21:49:29.1346233Z alias:          char-major-195-*
2025-04-21T21:49:29.1346508Z version:        570.133.07
2025-04-21T21:49:29.1346756Z supported:      external
2025-04-21T21:49:29.1347010Z license:        Dual MIT/GPL
2025-04-21T21:49:29.1347496Z firmware:       nvidia/570.133.07/gsp_tu10x.bin
2025-04-21T21:49:29.1347853Z firmware:       nvidia/570.133.07/gsp_ga10x.bin
2025-04-21T21:49:29.1348188Z srcversion:     49515739FD8F721A3F2F714
2025-04-21T21:49:29.1348519Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2025-04-21T21:49:29.1348863Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2025-04-21T21:49:29.1349217Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2025-04-21T21:49:29.1349539Z depends:        i2c-core,drm
2025-04-21T21:49:29.1349791Z retpoline:      Y
2025-04-21T21:49:29.1350019Z name:           nvidia
2025-04-21T21:49:29.1350478Z vermagic:       6.1.130-139.222.amzn2023.x86_64 SMP preempt mod_unload modversions 
2025-04-21T21:49:29.1350970Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2025-04-21T21:49:29.1351425Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2025-04-21T21:49:29.1351861Z parm:           NVreg_ResmanDebugLevel:int
2025-04-21T21:49:29.1352185Z parm:           NVreg_RmLogonRC:int
2025-04-21T21:49:29.1352490Z parm:           NVreg_ModifyDeviceFiles:int
2025-04-21T21:49:29.1352817Z parm:           NVreg_DeviceFileUID:int
2025-04-21T21:49:29.1353132Z parm:           NVreg_DeviceFileGID:int
2025-04-21T21:49:29.1353534Z parm:           NVreg_DeviceFileMode:int
2025-04-21T21:49:29.1353911Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2025-04-21T21:49:29.1354316Z parm:           NVreg_UsePageAttributeTable:int
2025-04-21T21:49:29.1354658Z parm:           NVreg_EnablePCIeGen3:int
2025-04-21T21:49:29.1354973Z parm:           NVreg_EnableMSI:int
2025-04-21T21:49:29.1355297Z parm:           NVreg_EnableStreamMemOPs:int
2025-04-21T21:49:29.1355673Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2025-04-21T21:49:29.1356077Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2025-04-21T21:49:29.1356474Z parm:           NVreg_EnableS0ixPowerManagement:int
2025-04-21T21:49:29.1356908Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2025-04-21T21:49:29.1357324Z parm:           NVreg_DynamicPowerManagement:int
2025-04-21T21:49:29.1357757Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2025-04-21T21:49:29.1358185Z parm:           NVreg_EnableGpuFirmware:int
2025-04-21T21:49:29.1358529Z parm:           NVreg_EnableGpuFirmwareLogs:int
2025-04-21T21:49:29.1358913Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2025-04-21T21:49:29.1359300Z parm:           NVreg_EnableUserNUMAManagement:int
2025-04-21T21:49:29.1359653Z parm:           NVreg_MemoryPoolSize:int
2025-04-21T21:49:29.1359982Z parm:           NVreg_KMallocHeapMaxSize:int
2025-04-21T21:49:29.1360341Z parm:           NVreg_VMallocHeapMaxSize:int
2025-04-21T21:49:29.1360676Z parm:           NVreg_IgnoreMMIOCheck:int
2025-04-21T21:49:29.1360990Z parm:           NVreg_NvLinkDisable:int
2025-04-21T21:49:29.1361352Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2025-04-21T21:49:29.1361725Z parm:           NVreg_RegisterPCIDriver:int
2025-04-21T21:49:29.1362061Z parm:           NVreg_EnableResizableBar:int
2025-04-21T21:49:29.1362407Z parm:           NVreg_EnableDbgBreakpoint:int
2025-04-21T21:49:29.1362767Z parm:           NVreg_EnableNonblockingOpen:int
2025-04-21T21:49:29.1363111Z parm:           NVreg_RegistryDwords:charp
2025-04-21T21:49:29.1363466Z parm:           NVreg_RegistryDwordsPerDevice:charp
2025-04-21T21:49:29.1363809Z parm:           NVreg_RmMsg:charp
2025-04-21T21:49:29.1364098Z parm:           NVreg_GpuBlacklist:charp
2025-04-21T21:49:29.1364434Z parm:           NVreg_TemporaryFilePath:charp
2025-04-21T21:49:29.1364769Z parm:           NVreg_ExcludedGpus:charp
2025-04-21T21:49:29.1365100Z parm:           NVreg_DmaRemapPeerMmio:int
2025-04-21T21:49:29.1365430Z parm:           NVreg_RmNvlinkBandwidth:charp
2025-04-21T21:49:29.1365803Z parm:           NVreg_RmNvlinkBandwidthLinkCount:int
2025-04-21T21:49:29.1366163Z parm:           NVreg_ImexChannelCount:int
2025-04-21T21:49:29.1366502Z parm:           NVreg_CreateImexChannel0:int
2025-04-21T21:49:29.1366960Z parm:           NVreg_GrdmaPciTopoCheckOverride:int
2025-04-21T21:49:29.1367313Z parm:           rm_firmware_active:charp
2025-04-21T21:49:29.1367600Z + HAS_NVIDIA_DRIVER=0
2025-04-21T21:49:29.1367855Z ++ command -v nvidia-smi
2025-04-21T21:49:29.1368116Z + '[' -x /usr/bin/nvidia-smi ']'
2025-04-21T21:49:29.1368375Z + set +e
2025-04-21T21:49:29.1368684Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2025-04-21T21:49:29.1603324Z + INSTALLED_DRIVER_VERSION=570.133.07
2025-04-21T21:49:29.1603634Z + NVIDIA_SMI_STATUS=0
2025-04-21T21:49:29.1604182Z + '[' 0 -ne 0 ']'
2025-04-21T21:49:29.1604639Z + '[' 570.133.07 '!=' 570.133.07 ']'
2025-04-21T21:49:29.1604975Z + HAS_NVIDIA_DRIVER=1
2025-04-21T21:49:29.1605434Z + echo 'NVIDIA driver (570.133.07) has already been installed. Skipping NVIDIA driver installation'
2025-04-21T21:49:29.1605939Z + set -e
2025-04-21T21:49:29.1606139Z + '[' 1 -eq 0 ']'
2025-04-21T21:49:29.1606561Z NVIDIA driver (570.133.07) has already been installed. Skipping NVIDIA driver installation
2025-04-21T21:49:29.1607059Z + post_install_nvidia_driver_common
2025-04-21T21:49:29.1610165Z + sudo modprobe nvidia
2025-04-21T21:49:29.3128046Z + echo 'After installing NVIDIA driver'
2025-04-21T21:49:29.3128368Z + lspci
2025-04-21T21:49:29.3128586Z After installing NVIDIA driver
2025-04-21T21:49:29.3245704Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma]
2025-04-21T21:49:29.3246224Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2025-04-21T21:49:29.3246778Z 00:01.3 Non-VGA unclassified device: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
2025-04-21T21:49:29.3247335Z 00:03.0 VGA compatible controller: Amazon.com, Inc. Device 1111
2025-04-21T21:49:29.3247835Z 00:04.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe EBS Controller
2025-04-21T21:49:29.3248385Z 00:05.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2025-04-21T21:49:29.3248897Z 00:1e.0 3D controller: NVIDIA Corporation GA102GL [A10G] (rev a1)
2025-04-21T21:49:29.3249394Z 00:1f.0 Non-Volatile memory controller: Amazon.com, Inc. NVMe SSD Controller
2025-04-21T21:49:29.3249812Z + lsmod
2025-04-21T21:49:29.3281326Z Module                  Size  Used by
2025-04-21T21:49:29.3281624Z veth                   36864  0
2025-04-21T21:49:29.3281899Z nvidia_modeset       1716224  0
2025-04-21T21:49:29.3282200Z video                  65536  1 nvidia_modeset
2025-04-21T21:49:29.3282519Z wmi                    36864  1 video
2025-04-21T21:49:29.3282796Z nvidia_uvm           1884160  0
2025-04-21T21:49:29.3283107Z nvidia              11583488  7 nvidia_uvm,nvidia_modeset
2025-04-21T21:49:29.3283450Z drm                   602112  1 nvidia
2025-04-21T21:49:29.3283757Z drm_panel_orientation_quirks    32768  1 drm
2025-04-21T21:49:29.3284126Z backlight              24576  3 video,drm,nvidia_modeset
2025-04-21T21:49:29.3284481Z i2c_core              110592  2 nvidia,drm
2025-04-21T21:49:29.3284771Z xt_conntrack           16384  1
2025-04-21T21:49:29.3285044Z nft_chain_nat          16384  3
2025-04-21T21:49:29.3285314Z xt_MASQUERADE          20480  1
2025-04-21T21:49:29.3285616Z nf_nat                 57344  2 nft_chain_nat,xt_MASQUERADE
2025-04-21T21:49:29.3285959Z nf_conntrack_netlink    57344  0
2025-04-21T21:49:29.3286372Z nf_conntrack          184320  4 xt_conntrack,nf_nat,nf_conntrack_netlink,xt_MASQUERADE
2025-04-21T21:49:29.3286822Z nf_defrag_ipv6         24576  1 nf_conntrack
2025-04-21T21:49:29.3287145Z nf_defrag_ipv4         16384  1 nf_conntrack
2025-04-21T21:49:29.3287444Z xfrm_user              57344  1
2025-04-21T21:49:29.3287717Z xfrm_algo              16384  1 xfrm_user
2025-04-21T21:49:29.3288016Z xt_addrtype            16384  2
2025-04-21T21:49:29.3288281Z nft_compat             20480  4
2025-04-21T21:49:29.3288593Z nf_tables             311296  57 nft_compat,nft_chain_nat
2025-04-21T21:49:29.3289013Z nfnetlink              20480  4 nft_compat,nf_conntrack_netlink,nf_tables
2025-04-21T21:49:29.3289400Z br_netfilter           36864  0
2025-04-21T21:49:29.3289937Z bridge                323584  1 br_netfilter
2025-04-21T21:49:29.3290243Z stp                    16384  1 bridge
2025-04-21T21:49:29.3290535Z llc                    16384  2 bridge,stp
2025-04-21T21:49:29.3290830Z overlay               167936  0
2025-04-21T21:49:29.3291093Z tls                   135168  0
2025-04-21T21:49:29.3291353Z nls_ascii              16384  1
2025-04-21T21:49:29.3291610Z nls_cp437              20480  1
2025-04-21T21:49:29.3291868Z vfat                   24576  1
2025-04-21T21:49:29.3292121Z fat                    86016  1 vfat
2025-04-21T21:49:29.3292396Z sunrpc                696320  1
2025-04-21T21:49:29.3292782Z ena                   180224  0
2025-04-21T21:49:29.3293027Z i8042                  45056  0
2025-04-21T21:49:29.3293292Z serio                  28672  3 i8042
2025-04-21T21:49:29.3293584Z ghash_clmulni_intel    16384  0
2025-04-21T21:49:29.3293841Z button                 24576  0
2025-04-21T21:49:29.3294106Z sch_fq_codel           20480  17
2025-04-21T21:49:29.3294382Z dm_mod                188416  0
2025-04-21T21:49:29.3294642Z dax                    45056  1 dm_mod
2025-04-21T21:49:29.3294962Z loop                   36864  0
2025-04-21T21:49:29.3295235Z fuse                  163840  1
2025-04-21T21:49:29.3295484Z configfs               57344  1
2025-04-21T21:49:29.3295745Z dmi_sysfs              20480  0
2025-04-21T21:49:29.3296005Z crc32_pclmul           16384  0
2025-04-21T21:49:29.3296262Z crc32c_intel           24576  0
2025-04-21T21:49:29.3296514Z efivarfs               24576  1
2025-04-21T21:49:29.3296771Z + modinfo nvidia
2025-04-21T21:49:29.3297708Z filename:       /lib/modules/6.1.130-139.222.amzn2023.x86_64/kernel/drivers/video/nvidia.ko
2025-04-21T21:49:29.3298173Z import_ns:      DMA_BUF
2025-04-21T21:49:29.3298445Z alias:          char-major-195-*
2025-04-21T21:49:29.3298717Z version:        570.133.07
2025-04-21T21:49:29.3298970Z supported:      external
2025-04-21T21:49:29.3299218Z license:        Dual MIT/GPL
2025-04-21T21:49:29.3299519Z firmware:       nvidia/570.133.07/gsp_tu10x.bin
2025-04-21T21:49:29.3299870Z firmware:       nvidia/570.133.07/gsp_ga10x.bin
2025-04-21T21:49:29.3300194Z srcversion:     49515739FD8F721A3F2F714
2025-04-21T21:49:29.3300526Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2025-04-21T21:49:29.3300875Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2025-04-21T21:49:29.3301217Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2025-04-21T21:49:29.3301540Z depends:        i2c-core,drm
2025-04-21T21:49:29.3301802Z retpoline:      Y
2025-04-21T21:49:29.3302028Z name:           nvidia
2025-04-21T21:49:29.3302391Z vermagic:       6.1.130-139.222.amzn2023.x86_64 SMP preempt mod_unload modversions 
2025-04-21T21:49:29.3302884Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2025-04-21T21:49:29.3303342Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2025-04-21T21:49:29.3303765Z parm:           NVreg_ResmanDebugLevel:int
2025-04-21T21:49:29.3304092Z parm:           NVreg_RmLogonRC:int
2025-04-21T21:49:29.3304400Z parm:           NVreg_ModifyDeviceFiles:int
2025-04-21T21:49:29.3304716Z parm:           NVreg_DeviceFileUID:int
2025-04-21T21:49:29.3305053Z parm:           NVreg_DeviceFileGID:int
2025-04-21T21:49:29.3305389Z parm:           NVreg_DeviceFileMode:int
2025-04-21T21:49:29.3305764Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2025-04-21T21:49:29.3306159Z parm:           NVreg_UsePageAttributeTable:int
2025-04-21T21:49:29.3306508Z parm:           NVreg_EnablePCIeGen3:int
2025-04-21T21:49:29.3306819Z parm:           NVreg_EnableMSI:int
2025-04-21T21:49:29.3307132Z parm:           NVreg_EnableStreamMemOPs:int
2025-04-21T21:49:29.3307515Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2025-04-21T21:49:29.3307927Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2025-04-21T21:49:29.3308317Z parm:           NVreg_EnableS0ixPowerManagement:int
2025-04-21T21:49:29.3308748Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2025-04-21T21:49:29.3310416Z parm:           NVreg_DynamicPowerManagement:int
2025-04-21T21:49:29.3310863Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2025-04-21T21:49:29.3311282Z parm:           NVreg_EnableGpuFirmware:int
2025-04-21T21:49:29.3311632Z parm:           NVreg_EnableGpuFirmwareLogs:int
2025-04-21T21:49:29.3312014Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2025-04-21T21:49:29.3312395Z parm:           NVreg_EnableUserNUMAManagement:int
2025-04-21T21:49:29.3312749Z parm:           NVreg_MemoryPoolSize:int
2025-04-21T21:49:29.3313085Z parm:           NVreg_KMallocHeapMaxSize:int
2025-04-21T21:49:29.3313571Z parm:           NVreg_VMallocHeapMaxSize:int
2025-04-21T21:49:29.3313903Z parm:           NVreg_IgnoreMMIOCheck:int
2025-04-21T21:49:29.3314220Z parm:           NVreg_NvLinkDisable:int
2025-04-21T21:49:29.3314569Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2025-04-21T21:49:29.3314942Z parm:           NVreg_RegisterPCIDriver:int
2025-04-21T21:49:29.3315286Z parm:           NVreg_EnableResizableBar:int
2025-04-21T21:49:29.3315628Z parm:           NVreg_EnableDbgBreakpoint:int
2025-04-21T21:49:29.3315976Z parm:           NVreg_EnableNonblockingOpen:int
2025-04-21T21:49:29.3316325Z parm:           NVreg_RegistryDwords:charp
2025-04-21T21:49:29.3316674Z parm:           NVreg_RegistryDwordsPerDevice:charp
2025-04-21T21:49:29.3317009Z parm:           NVreg_RmMsg:charp
2025-04-21T21:49:29.3317305Z parm:           NVreg_GpuBlacklist:charp
2025-04-21T21:49:29.3317639Z parm:           NVreg_TemporaryFilePath:charp
2025-04-21T21:49:29.3317967Z parm:           NVreg_ExcludedGpus:charp
2025-04-21T21:49:29.3318301Z parm:           NVreg_DmaRemapPeerMmio:int
2025-04-21T21:49:29.3318645Z parm:           NVreg_RmNvlinkBandwidth:charp
2025-04-21T21:49:29.3319012Z parm:           NVreg_RmNvlinkBandwidthLinkCount:int
2025-04-21T21:49:29.3319369Z parm:           NVreg_ImexChannelCount:int
2025-04-21T21:49:29.3319706Z parm:           NVreg_CreateImexChannel0:int
2025-04-21T21:49:29.3320069Z parm:           NVreg_GrdmaPciTopoCheckOverride:int
2025-04-21T21:49:29.3320414Z parm:           rm_firmware_active:charp
2025-04-21T21:49:29.3320702Z + set +e
2025-04-21T21:49:29.3320897Z + nvidia-smi
2025-04-21T21:49:29.3474748Z Mon Apr 21 21:49:29 2025       
2025-04-21T21:49:29.3475468Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:49:29.3476474Z | NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
2025-04-21T21:49:29.3477457Z |-----------------------------------------+------------------------+----------------------+
2025-04-21T21:49:29.3478453Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2025-04-21T21:49:29.3479527Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2025-04-21T21:49:29.3480419Z |                                         |                        |               MIG M. |
2025-04-21T21:49:29.3481114Z |=========================================+========================+======================|
2025-04-21T21:49:29.3629883Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2025-04-21T21:49:29.3630358Z |  0%   30C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2025-04-21T21:49:29.3630768Z |                                         |                        |                  N/A |
2025-04-21T21:49:29.3631184Z +-----------------------------------------+------------------------+----------------------+
2025-04-21T21:49:29.3634560Z                                                                                          
2025-04-21T21:49:29.3634980Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:49:29.3635424Z | Processes:                                                                              |
2025-04-21T21:49:29.3636066Z |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
2025-04-21T21:49:29.3636498Z |        ID   ID                                                               Usage      |
2025-04-21T21:49:29.3636861Z |=========================================================================================|
2025-04-21T21:49:29.3639640Z |  No running processes found                                                             |
2025-04-21T21:49:29.3640131Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:49:29.6394882Z + nvidia-smi --query-gpu=gpu_name --format=csv,noheader --id=0
2025-04-21T21:49:29.6558647Z NVIDIA A10G
2025-04-21T21:49:29.6600223Z + NVIDIA_SMI_STATUS=0
2025-04-21T21:49:29.6600676Z + '[' 0 -eq 0 ']'
2025-04-21T21:49:29.6600953Z + echo 'INFO: Ignoring allowed status 0'
2025-04-21T21:49:29.6601251Z + set -e
2025-04-21T21:49:29.6601474Z INFO: Ignoring allowed status 0
2025-04-21T21:49:29.6609875Z == Installing nvidia container toolkit for amzn2023 ==
2025-04-21T21:49:29.6613835Z + sudo yum install -y yum-utils
2025-04-21T21:49:30.1775891Z Last metadata expiration check: 1:26:51 ago on Mon Apr 21 20:22:39 2025.
2025-04-21T21:49:30.2031125Z Package dnf-utils-4.3.0-13.amzn2023.0.5.noarch is already installed.
2025-04-21T21:49:30.2433993Z Dependencies resolved.
2025-04-21T21:49:30.2619116Z Nothing to do.
2025-04-21T21:49:30.2619575Z Complete!
2025-04-21T21:49:30.3267296Z + [[ amzn2023 == \a\m\z\n\2\0\2\3 ]]
2025-04-21T21:49:30.3268038Z + YUM_REPO_URL=https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2025-04-21T21:49:30.3268964Z + sudo yum-config-manager --add-repo https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2025-04-21T21:49:30.6045460Z Adding repo from: https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo
2025-04-21T21:49:30.6611513Z + sudo yum install -y nvidia-docker2 nvidia-container-toolkit-1.16.2
2025-04-21T21:49:31.1781984Z nvidia-container-toolkit                         14 kB/s | 833  B     00:00    
2025-04-21T21:49:31.2035585Z Package nvidia-docker2-2.14.0-1.noarch is already installed.
2025-04-21T21:49:31.2041145Z Package nvidia-container-toolkit-1.16.2-1.x86_64 is already installed.
2025-04-21T21:49:31.2434584Z Dependencies resolved.
2025-04-21T21:49:31.2638890Z Nothing to do.
2025-04-21T21:49:31.2639144Z Complete!
2025-04-21T21:49:31.4028677Z + sudo systemctl restart docker
2025-04-21T21:50:15.7087941Z nvidia-persistenced failed to initialize. Check syslog for more details.
2025-04-21T21:50:15.7286910Z Mon Apr 21 21:50:15 2025       
2025-04-21T21:50:15.7287909Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:50:15.7288531Z | NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
2025-04-21T21:50:15.7289045Z |-----------------------------------------+------------------------+----------------------+
2025-04-21T21:50:15.7289582Z | GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
2025-04-21T21:50:15.7290131Z | Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
2025-04-21T21:50:15.7290590Z |                                         |                        |               MIG M. |
2025-04-21T21:50:15.7290942Z |=========================================+========================+======================|
2025-04-21T21:50:15.7421889Z |   0  NVIDIA A10G                    On  |   00000000:00:1E.0 Off |                    0 |
2025-04-21T21:50:15.7422516Z |  0%   30C    P8             15W /  300W |       0MiB /  23028MiB |      0%      Default |
2025-04-21T21:50:15.7422987Z |                                         |                        |                  N/A |
2025-04-21T21:50:15.7423415Z +-----------------------------------------+------------------------+----------------------+
2025-04-21T21:50:15.7426262Z                                                                                          
2025-04-21T21:50:15.7427189Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:50:15.7427649Z | Processes:                                                                              |
2025-04-21T21:50:15.7428110Z |  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
2025-04-21T21:50:15.7428545Z |        ID   ID                                                               Usage      |
2025-04-21T21:50:15.7428904Z |=========================================================================================|
2025-04-21T21:50:15.7432828Z |  No running processes found                                                             |
2025-04-21T21:50:15.7433410Z +-----------------------------------------------------------------------------------------+
2025-04-21T21:50:16.1647715Z Command completed after 1 attempt(s).
2025-04-21T21:50:16.1741434Z Prepare all required actions
2025-04-21T21:50:16.1770589Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-04-21T21:50:16.1770923Z with:
2025-04-21T21:50:16.1771356Z   github-token: ***
2025-04-21T21:50:16.1771588Z env:
2025-04-21T21:50:16.1771797Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:16.1772129Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:16.1772479Z ##[endgroup]
2025-04-21T21:50:16.1790963Z ##[group]Run set -eux
2025-04-21T21:50:16.1791229Z set -eux
2025-04-21T21:50:16.1791659Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-04-21T21:50:16.1806302Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:16.1806681Z env:
2025-04-21T21:50:16.1806899Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:16.1807222Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:16.1807704Z   GITHUB_TOKEN: ***
2025-04-21T21:50:16.1807937Z ##[endgroup]
2025-04-21T21:50:16.1841735Z + python3 .github/scripts/get_workflow_job_id.py 14580756816 i-0e08e74978c5f6915
2025-04-21T21:50:17.2891584Z setting job-id=40900324329
2025-04-21T21:50:17.2892152Z setting job-name=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:17.3009038Z ##[group]Run python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7
2025-04-21T21:50:17.3009756Z python3 -m pip install psutil==5.9.1 nvidia-ml-py==11.525.84 dataclasses_json==0.6.7
2025-04-21T21:50:17.3010330Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2025-04-21T21:50:17.3010795Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2025-04-21T21:50:17.3020365Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:17.3020734Z env:
2025-04-21T21:50:17.3020945Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:17.3021277Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:17.3021971Z   JOB_ID: 40900324329
2025-04-21T21:50:17.3022473Z   JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:17.3022986Z   WORKFLOW_NAME: inductor
2025-04-21T21:50:17.3023254Z   WORKFLOW_RUN_ID: 14580756816
2025-04-21T21:50:17.3023557Z ##[endgroup]
2025-04-21T21:50:17.5349648Z Defaulting to user installation because normal site-packages is not writeable
2025-04-21T21:50:17.5520278Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.9/site-packages (5.9.1)
2025-04-21T21:50:17.5525014Z Requirement already satisfied: nvidia-ml-py==11.525.84 in /home/ec2-user/.local/lib/python3.9/site-packages (11.525.84)
2025-04-21T21:50:17.5529792Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7)
2025-04-21T21:50:17.5632737Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0)
2025-04-21T21:50:17.5634748Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1)
2025-04-21T21:50:17.5720614Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0)
2025-04-21T21:50:17.5748569Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.13.2)
2025-04-21T21:50:17.5755283Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.0.0)
2025-04-21T21:50:17.7640115Z Prepare all required actions
2025-04-21T21:50:17.7640495Z Getting action download info
2025-04-21T21:50:17.9108126Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-04-21T21:50:18.1299697Z Download action repository 'actions/download-artifact@v4' (SHA:95815c38cf2ff2164869cbab79da8d1f422bc89e)
2025-04-21T21:50:18.3791826Z ##[group]Run ./.github/actions/download-build-artifacts
2025-04-21T21:50:18.3792199Z with:
2025-04-21T21:50:18.3792452Z   name: linux-focal-cuda12.6-py3.10-gcc9-sm86
2025-04-21T21:50:18.3792793Z   s3-bucket: gha-artifacts
2025-04-21T21:50:18.3793060Z env:
2025-04-21T21:50:18.3793267Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:18.3793725Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:18.3794130Z ##[endgroup]
2025-04-21T21:50:18.3825676Z ##[group]Run seemethere/download-artifact-s3@v4
2025-04-21T21:50:18.3826021Z with:
2025-04-21T21:50:18.3826319Z   name: linux-focal-cuda12.6-py3.10-gcc9-sm86
2025-04-21T21:50:18.3826656Z   s3-bucket: gha-artifacts
2025-04-21T21:50:18.3826927Z   region: us-east-1
2025-04-21T21:50:18.3827156Z env:
2025-04-21T21:50:18.3827366Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:18.3827699Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:18.3828063Z ##[endgroup]
2025-04-21T21:50:18.8574462Z (node:107306) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-04-21T21:50:18.8574948Z 
2025-04-21T21:50:18.8575144Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-04-21T21:50:18.8575674Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-04-21T21:50:18.8576286Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-04-21T21:50:18.9660840Z Found 1 objects with prefix pytorch/pytorch/14580756816/linux-focal-cuda12.6-py3.10-gcc9-sm86/
2025-04-21T21:50:18.9662346Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-04-21T21:50:37.5225943Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-04-21T21:50:37.5231376Z Artifact download has finished successfully
2025-04-21T21:50:37.5737963Z ##[group]Run unzip -o artifacts.zip
2025-04-21T21:50:37.5738299Z unzip -o artifacts.zip
2025-04-21T21:50:37.5750142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:37.5750566Z env:
2025-04-21T21:50:37.5750812Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:37.5751154Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:37.5751498Z ##[endgroup]
2025-04-21T21:50:37.5796007Z Archive:  artifacts.zip
2025-04-21T21:50:37.5797560Z    creating: dist/
2025-04-21T21:50:39.7573787Z   inflating: dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl  
2025-04-21T21:50:39.7708254Z   inflating: dist/.ninja_log         
2025-04-21T21:50:39.7708702Z    creating: build/custom_test_artifacts/
2025-04-21T21:50:39.7709145Z    creating: build/custom_test_artifacts/custom-op-build/
2025-04-21T21:50:39.7709622Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-04-21T21:50:39.7710185Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-04-21T21:50:39.7718634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-21T21:50:39.7719322Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/
2025-04-21T21:50:39.7719955Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-21T21:50:39.7720642Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-21T21:50:39.7721309Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-21T21:50:39.7723326Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-21T21:50:39.7725020Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-21T21:50:39.7725753Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-21T21:50:39.7726460Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-21T21:50:39.7727147Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-21T21:50:39.7729917Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-21T21:50:39.7731400Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-21T21:50:39.7732670Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-21T21:50:39.7734674Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-21T21:50:39.7736908Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-21T21:50:39.7737671Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/
2025-04-21T21:50:39.7738364Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/
2025-04-21T21:50:39.7779744Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2025-04-21T21:50:39.7821011Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2025-04-21T21:50:39.7822161Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2025-04-21T21:50:39.7869419Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2025-04-21T21:50:39.7870584Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2025-04-21T21:50:39.7871686Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2025-04-21T21:50:39.7872705Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2025-04-21T21:50:39.7873730Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2025-04-21T21:50:39.7874678Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2025-04-21T21:50:39.7875634Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2025-04-21T21:50:39.7876602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2025-04-21T21:50:39.7877953Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2025-04-21T21:50:39.7878855Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2025-04-21T21:50:39.7879723Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c  
2025-04-21T21:50:39.7880579Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin  
2025-04-21T21:50:39.7882589Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2025-04-21T21:50:39.7883432Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o  
2025-04-21T21:50:39.7886452Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2025-04-21T21:50:39.7964928Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out  
2025-04-21T21:50:39.7965711Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake  
2025-04-21T21:50:39.8040906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin  
2025-04-21T21:50:39.8041672Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-04-21T21:50:39.8042298Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-04-21T21:50:39.8042950Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-04-21T21:50:39.8043598Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-04-21T21:50:39.8044309Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-04-21T21:50:39.8045141Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-04-21T21:50:39.8045921Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-04-21T21:50:39.8046646Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-04-21T21:50:39.8047397Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-04-21T21:50:39.8048164Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-04-21T21:50:39.8048935Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-04-21T21:50:39.8049804Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-04-21T21:50:39.8050880Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-04-21T21:50:39.8072341Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-04-21T21:50:39.8271373Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-04-21T21:50:39.8272074Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-04-21T21:50:39.8272833Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-04-21T21:50:39.8273729Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-04-21T21:50:39.8274553Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-04-21T21:50:39.8275321Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-04-21T21:50:39.8276121Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-04-21T21:50:39.8277049Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-04-21T21:50:39.8277874Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-04-21T21:50:39.8278822Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-04-21T21:50:39.8279792Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-04-21T21:50:39.8302142Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-04-21T21:50:39.8385347Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-04-21T21:50:39.8386536Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-21T21:50:39.8387586Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-04-21T21:50:39.8388900Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-04-21T21:50:39.8389729Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-04-21T21:50:39.8390584Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-04-21T21:50:39.8391414Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2025-04-21T21:50:39.8394051Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-04-21T21:50:39.8394928Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-04-21T21:50:39.8395792Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-04-21T21:50:39.8560302Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-04-21T21:50:39.8616714Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-04-21T21:50:39.8617439Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-04-21T21:50:39.8618080Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-04-21T21:50:39.8618630Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-04-21T21:50:39.8626317Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-21T21:50:39.8627193Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/
2025-04-21T21:50:39.8628051Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-21T21:50:39.8628915Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-21T21:50:39.8629550Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-21T21:50:39.8630940Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-21T21:50:39.8632689Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-21T21:50:39.8633803Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-21T21:50:39.8634751Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-21T21:50:39.8635575Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-21T21:50:39.8638015Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-21T21:50:39.8639225Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-21T21:50:39.8640462Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-21T21:50:39.8642563Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-21T21:50:39.8644381Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-21T21:50:39.8645441Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/
2025-04-21T21:50:39.8646220Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/
2025-04-21T21:50:39.8687658Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2025-04-21T21:50:39.8728907Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2025-04-21T21:50:39.8730088Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2025-04-21T21:50:39.8776899Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2025-04-21T21:50:39.8778363Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2025-04-21T21:50:39.8779867Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2025-04-21T21:50:39.8781257Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2025-04-21T21:50:39.8782617Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2025-04-21T21:50:39.8783871Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2025-04-21T21:50:39.8785077Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2025-04-21T21:50:39.8786394Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2025-04-21T21:50:39.8787693Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2025-04-21T21:50:39.8788904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2025-04-21T21:50:39.8790096Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c  
2025-04-21T21:50:39.8791139Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin  
2025-04-21T21:50:39.8791986Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2025-04-21T21:50:39.8793072Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o  
2025-04-21T21:50:39.8794114Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2025-04-21T21:50:39.8868451Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out  
2025-04-21T21:50:39.8869491Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake  
2025-04-21T21:50:39.8944279Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin  
2025-04-21T21:50:39.8945216Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-04-21T21:50:39.8945800Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-04-21T21:50:39.8946633Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-04-21T21:50:39.8947522Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-04-21T21:50:39.8948242Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-04-21T21:50:39.8949069Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-04-21T21:50:39.8949862Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-04-21T21:50:39.8950599Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-04-21T21:50:39.8951352Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-04-21T21:50:39.8952119Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-04-21T21:50:39.8952899Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-04-21T21:50:39.8953795Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-04-21T21:50:39.8954553Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-04-21T21:50:39.8975575Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-04-21T21:50:39.9040089Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-04-21T21:50:39.9040907Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-21T21:50:39.9041657Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-04-21T21:50:39.9042335Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-04-21T21:50:39.9043302Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-04-21T21:50:39.9045070Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-04-21T21:50:39.9045687Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2025-04-21T21:50:39.9048605Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-04-21T21:50:39.9049395Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-04-21T21:50:39.9050442Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-04-21T21:50:39.9089158Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-04-21T21:50:39.9089669Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-04-21T21:50:39.9090182Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-04-21T21:50:39.9090788Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-04-21T21:50:39.9098559Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-04-21T21:50:39.9099239Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/
2025-04-21T21:50:39.9099915Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeSystem.cmake  
2025-04-21T21:50:39.9100791Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/
2025-04-21T21:50:39.9101528Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/tmp/
2025-04-21T21:50:39.9103540Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/CMakeCCompilerId.c  
2025-04-21T21:50:39.9105268Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdC/a.out  
2025-04-21T21:50:39.9106053Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCCompiler.cmake  
2025-04-21T21:50:39.9106808Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/
2025-04-21T21:50:39.9107543Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/tmp/
2025-04-21T21:50:39.9110000Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-04-21T21:50:39.9111659Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCXX/a.out  
2025-04-21T21:50:39.9112812Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCXXCompiler.cmake  
2025-04-21T21:50:39.9114842Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_C.bin  
2025-04-21T21:50:39.9116919Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CXX.bin  
2025-04-21T21:50:39.9117750Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/
2025-04-21T21:50:39.9118482Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/
2025-04-21T21:50:39.9159975Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2025-04-21T21:50:39.9201500Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2025-04-21T21:50:39.9202710Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2025-04-21T21:50:39.9249657Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2025-04-21T21:50:39.9250690Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2025-04-21T21:50:39.9252016Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2025-04-21T21:50:39.9253202Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2025-04-21T21:50:39.9256292Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2025-04-21T21:50:39.9257936Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2025-04-21T21:50:39.9259369Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2025-04-21T21:50:39.9260786Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2025-04-21T21:50:39.9262178Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2025-04-21T21:50:39.9263541Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2025-04-21T21:50:39.9264817Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.reg.c  
2025-04-21T21:50:39.9266084Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin  
2025-04-21T21:50:39.9267361Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2025-04-21T21:50:39.9268594Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/tmp/a_dlink.o  
2025-04-21T21:50:39.9269850Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2025-04-21T21:50:39.9341385Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CompilerIdCUDA/a.out  
2025-04-21T21:50:39.9342358Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeCUDACompiler.cmake  
2025-04-21T21:50:39.9417180Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.31.2/CMakeDetermineCompilerABI_CUDA.bin  
2025-04-21T21:50:39.9418774Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-04-21T21:50:39.9420063Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-04-21T21:50:39.9421379Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-04-21T21:50:39.9422877Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-04-21T21:50:39.9423707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-04-21T21:50:39.9424593Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-04-21T21:50:39.9425456Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-04-21T21:50:39.9426262Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-04-21T21:50:39.9427096Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-04-21T21:50:39.9428196Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-04-21T21:50:39.9429034Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-04-21T21:50:39.9429876Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-04-21T21:50:39.9430704Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-04-21T21:50:39.9431792Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-04-21T21:50:39.9555258Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-04-21T21:50:39.9556098Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-04-21T21:50:39.9556941Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-04-21T21:50:39.9557882Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-04-21T21:50:39.9558782Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-04-21T21:50:39.9559630Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-04-21T21:50:39.9560506Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-04-21T21:50:39.9561389Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-04-21T21:50:39.9562264Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-04-21T21:50:39.9563145Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-04-21T21:50:39.9564015Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-04-21T21:50:39.9585986Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-04-21T21:50:39.9641776Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-04-21T21:50:39.9642973Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-04-21T21:50:39.9643798Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-04-21T21:50:39.9644529Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-04-21T21:50:39.9645222Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-04-21T21:50:39.9646979Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-04-21T21:50:39.9647647Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2025-04-21T21:50:39.9650487Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-04-21T21:50:39.9651353Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-04-21T21:50:39.9652340Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-04-21T21:50:39.9753239Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-04-21T21:50:39.9792016Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-04-21T21:50:39.9792488Z    creating: build/lib/
2025-04-21T21:50:39.9881373Z   inflating: build/lib/libprotobuf-lite.a  
2025-04-21T21:50:40.0336439Z   inflating: build/lib/libprotobuf.a  
2025-04-21T21:50:40.0840226Z   inflating: build/lib/libprotoc.a   
2025-04-21T21:50:40.0850424Z   inflating: build/lib/libpthreadpool.a  
2025-04-21T21:50:40.0858635Z   inflating: build/lib/libcpuinfo.a  
2025-04-21T21:50:40.0866640Z   inflating: build/lib/libcpuinfo_internals.a  
2025-04-21T21:50:40.0867563Z   inflating: build/lib/libclog.a     
2025-04-21T21:50:40.0887206Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-04-21T21:50:40.0889913Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-04-21T21:50:40.0908243Z   inflating: build/lib/libnnpack.a   
2025-04-21T21:50:40.1094584Z   inflating: build/lib/libmicrokernels-prod.a  
2025-04-21T21:50:40.1899039Z   inflating: build/lib/libmicrokernels-all.a  
2025-04-21T21:50:40.1990818Z   inflating: build/lib/libXNNPACK.a  
2025-04-21T21:50:40.2059766Z   inflating: build/lib/libgtest.a    
2025-04-21T21:50:40.2077634Z   inflating: build/lib/libgmock.a    
2025-04-21T21:50:40.2078329Z   inflating: build/lib/libgmock_main.a  
2025-04-21T21:50:40.2079461Z   inflating: build/lib/libgtest_main.a  
2025-04-21T21:50:40.2155234Z   inflating: build/lib/libbenchmark.a  
2025-04-21T21:50:40.2155837Z   inflating: build/lib/libbenchmark_main.a  
2025-04-21T21:50:40.2220381Z   inflating: build/lib/libasmjit.a   
2025-04-21T21:50:40.3477573Z   inflating: build/lib/libfbgemm.a   
2025-04-21T21:50:40.3486146Z   inflating: build/lib/libittnotify.a  
2025-04-21T21:50:40.3514498Z   inflating: build/lib/libtensorpipe_uv.a  
2025-04-21T21:50:40.4089703Z   inflating: build/lib/libtensorpipe.a  
2025-04-21T21:50:40.4345851Z   inflating: build/lib/libtensorpipe_cuda.a  
2025-04-21T21:50:40.4481138Z   inflating: build/lib/libgloo.a     
2025-04-21T21:50:40.4870711Z   inflating: build/lib/libgloo_cuda.a  
2025-04-21T21:50:40.4913267Z   inflating: build/lib/libonnx_proto.a  
2025-04-21T21:50:40.5644249Z   inflating: build/lib/libonnx.a     
2025-04-21T21:50:41.6199845Z   inflating: build/lib/libdnnl.a     
2025-04-21T21:50:41.6218248Z   inflating: build/lib/libfmt.a      
2025-04-21T21:50:41.6686240Z   inflating: build/lib/libkineto.a   
2025-04-21T21:50:41.6797000Z   inflating: build/lib/libc10.so     
2025-04-21T21:50:41.6853971Z   inflating: build/lib/libc10_cuda.so  
2025-04-21T21:50:41.6856397Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-04-21T21:50:41.6857166Z   inflating: build/lib/libtorch_global_deps.so  
2025-04-21T21:50:44.2424529Z   inflating: build/lib/libtorch_cpu.so  
2025-04-21T21:50:46.2748793Z   inflating: build/lib/libtorch_cuda.so  
2025-04-21T21:50:46.2753719Z   inflating: build/lib/libunbox_lib.a  
2025-04-21T21:50:46.2754624Z   inflating: build/lib/libtorch.so   
2025-04-21T21:50:47.1519996Z   inflating: build/lib/libtorch_cuda_linalg.so  
2025-04-21T21:50:47.1590118Z   inflating: build/lib/libtorchbind_test.so  
2025-04-21T21:50:47.1610424Z   inflating: build/lib/libjitbackend_test.so  
2025-04-21T21:50:47.1635467Z   inflating: build/lib/libbackend_with_compiler.so  
2025-04-21T21:50:47.1660454Z   inflating: build/lib/libaoti_custom_ops.so  
2025-04-21T21:50:47.1663714Z   inflating: build/lib/libc10d_cuda_test.so  
2025-04-21T21:50:47.1667391Z   inflating: build/lib/libshm.so     
2025-04-21T21:50:47.3742145Z   inflating: build/lib/libtorch_python.so  
2025-04-21T21:50:47.3776686Z   inflating: build/lib/libnnapi_backend.so  
2025-04-21T21:50:47.3777008Z    creating: build/bin/
2025-04-21T21:50:47.4225865Z   inflating: build/bin/protoc-3.13.0.0  
2025-04-21T21:50:47.4672181Z   inflating: build/bin/protoc        
2025-04-21T21:50:47.4726187Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-04-21T21:50:47.4781158Z   inflating: build/bin/c10_DeviceGuard_test  
2025-04-21T21:50:47.4837615Z   inflating: build/bin/c10_Device_test  
2025-04-21T21:50:47.4900530Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-04-21T21:50:47.4957769Z   inflating: build/bin/c10_Scalar_test  
2025-04-21T21:50:47.5009577Z   inflating: build/bin/c10_StreamGuard_test  
2025-04-21T21:50:47.5064283Z   inflating: build/bin/c10_SymInt_test  
2025-04-21T21:50:47.5121221Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-04-21T21:50:47.5181076Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-04-21T21:50:47.5240638Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-04-21T21:50:47.5314944Z   inflating: build/bin/c10_cow_test  
2025-04-21T21:50:47.5367988Z   inflating: build/bin/c10_ArrayRef_test  
2025-04-21T21:50:47.5424615Z   inflating: build/bin/c10_Bitset_test  
2025-04-21T21:50:47.5476501Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-04-21T21:50:47.5529805Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-04-21T21:50:47.5583810Z   inflating: build/bin/c10_Half_test  
2025-04-21T21:50:47.5643845Z   inflating: build/bin/c10_LeftRight_test  
2025-04-21T21:50:47.5702803Z   inflating: build/bin/c10_Metaprogramming_test  
2025-04-21T21:50:47.5756128Z   inflating: build/bin/c10_Synchronized_test  
2025-04-21T21:50:47.5812663Z   inflating: build/bin/c10_NetworkFlow_test  
2025-04-21T21:50:47.5871598Z   inflating: build/bin/c10_ThreadLocal_test  
2025-04-21T21:50:47.5927462Z   inflating: build/bin/c10_TypeIndex_test  
2025-04-21T21:50:47.5982628Z   inflating: build/bin/c10_TypeList_test  
2025-04-21T21:50:47.6034496Z   inflating: build/bin/c10_TypeTraits_test  
2025-04-21T21:50:47.6089548Z   inflating: build/bin/c10_accumulate_test  
2025-04-21T21:50:47.6149979Z   inflating: build/bin/c10_bfloat16_test  
2025-04-21T21:50:47.6203316Z   inflating: build/bin/c10_bit_cast_test  
2025-04-21T21:50:47.6264849Z   inflating: build/bin/c10_complex_math_test  
2025-04-21T21:50:47.6323577Z   inflating: build/bin/c10_complex_test  
2025-04-21T21:50:47.6377098Z   inflating: build/bin/c10_error_test  
2025-04-21T21:50:47.6432861Z   inflating: build/bin/c10_exception_test  
2025-04-21T21:50:47.6495324Z   inflating: build/bin/c10_flags_test  
2025-04-21T21:50:47.6658568Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-04-21T21:50:47.6711923Z   inflating: build/bin/c10_generic_math_test  
2025-04-21T21:50:47.6767664Z   inflating: build/bin/c10_irange_test  
2025-04-21T21:50:47.6824510Z   inflating: build/bin/c10_lazy_test  
2025-04-21T21:50:47.6885235Z   inflating: build/bin/c10_logging_test  
2025-04-21T21:50:47.6964969Z   inflating: build/bin/c10_optional_test  
2025-04-21T21:50:47.7031172Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-04-21T21:50:47.7087938Z   inflating: build/bin/c10_registry_test  
2025-04-21T21:50:47.7247868Z   inflating: build/bin/c10_small_vector_test  
2025-04-21T21:50:47.7303567Z   inflating: build/bin/c10_ssize_test  
2025-04-21T21:50:47.7358787Z   inflating: build/bin/c10_string_util_test  
2025-04-21T21:50:47.7410986Z   inflating: build/bin/c10_string_view_test  
2025-04-21T21:50:47.7464962Z   inflating: build/bin/c10_tempfile_test  
2025-04-21T21:50:47.7524550Z   inflating: build/bin/c10_typeid_test  
2025-04-21T21:50:47.7573260Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-04-21T21:50:47.7629678Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2025-04-21T21:50:47.7685824Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2025-04-21T21:50:47.7742046Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2025-04-21T21:50:47.7798084Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2025-04-21T21:50:47.7853967Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-04-21T21:50:47.7911298Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2025-04-21T21:50:47.7966971Z   inflating: build/bin/c10_cuda_CUDATest  
2025-04-21T21:50:47.8023955Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2025-04-21T21:50:47.8448038Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-04-21T21:50:47.8877477Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-04-21T21:50:47.9323310Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-04-21T21:50:47.9380915Z   inflating: build/bin/test_edge_op_registration  
2025-04-21T21:50:47.9458370Z   inflating: build/bin/Dict_test     
2025-04-21T21:50:47.9514574Z   inflating: build/bin/Dimname_test  
2025-04-21T21:50:47.9582354Z   inflating: build/bin/MaybeOwned_test  
2025-04-21T21:50:47.9643796Z   inflating: build/bin/NamedTensor_test  
2025-04-21T21:50:47.9706143Z   inflating: build/bin/apply_utils_test  
2025-04-21T21:50:47.9768978Z   inflating: build/bin/atest         
2025-04-21T21:50:47.9835732Z   inflating: build/bin/basic         
2025-04-21T21:50:47.9894049Z   inflating: build/bin/broadcast_test  
2025-04-21T21:50:47.9948797Z   inflating: build/bin/cpu_allocator_test  
2025-04-21T21:50:48.0010746Z   inflating: build/bin/cpu_generator_test  
2025-04-21T21:50:48.0068540Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-04-21T21:50:48.0164134Z   inflating: build/bin/cpu_rng_test  
2025-04-21T21:50:48.0217936Z   inflating: build/bin/dlconvertor_test  
2025-04-21T21:50:48.0279582Z   inflating: build/bin/extension_backend_test  
2025-04-21T21:50:48.0338637Z   inflating: build/bin/half_test     
2025-04-21T21:50:48.0439798Z   inflating: build/bin/ivalue_test   
2025-04-21T21:50:48.0492998Z   inflating: build/bin/lazy_tensor_test  
2025-04-21T21:50:48.0550562Z   inflating: build/bin/math_kernel_test  
2025-04-21T21:50:48.0608409Z   inflating: build/bin/memory_format_test  
2025-04-21T21:50:48.0665368Z   inflating: build/bin/memory_overlapping_test  
2025-04-21T21:50:48.0721428Z   inflating: build/bin/mobile_memory_cleanup  
2025-04-21T21:50:48.0781480Z   inflating: build/bin/native_test   
2025-04-21T21:50:48.0836145Z   inflating: build/bin/operator_name_test  
2025-04-21T21:50:48.0891001Z   inflating: build/bin/operators_test  
2025-04-21T21:50:48.0947624Z   inflating: build/bin/packedtensoraccessor_test  
2025-04-21T21:50:48.1017721Z   inflating: build/bin/pow_test      
2025-04-21T21:50:48.1078928Z   inflating: build/bin/quantized_test  
2025-04-21T21:50:48.1132572Z   inflating: build/bin/reduce_ops_test  
2025-04-21T21:50:48.1187200Z   inflating: build/bin/reportMemoryUsage_test  
2025-04-21T21:50:48.1247758Z   inflating: build/bin/scalar_tensor_test  
2025-04-21T21:50:48.1310450Z   inflating: build/bin/scalar_test   
2025-04-21T21:50:48.1365940Z   inflating: build/bin/StorageUtils_test  
2025-04-21T21:50:48.1422356Z   inflating: build/bin/stride_properties_test  
2025-04-21T21:50:48.1505979Z   inflating: build/bin/tensor_iterator_test  
2025-04-21T21:50:48.1563660Z   inflating: build/bin/test_parallel  
2025-04-21T21:50:48.1566898Z   inflating: build/bin/thread_init_test  
2025-04-21T21:50:48.1626379Z   inflating: build/bin/type_ptr_test  
2025-04-21T21:50:48.1689545Z   inflating: build/bin/type_test     
2025-04-21T21:50:48.1745483Z   inflating: build/bin/undefined_tensor_test  
2025-04-21T21:50:48.1749608Z   inflating: build/bin/verify_api_visibility  
2025-04-21T21:50:48.1823754Z   inflating: build/bin/legacy_vmap_test  
2025-04-21T21:50:48.1878503Z   inflating: build/bin/weakref_test  
2025-04-21T21:50:48.1933633Z   inflating: build/bin/wrapdim_test  
2025-04-21T21:50:48.1989272Z   inflating: build/bin/xla_tensor_test  
2025-04-21T21:50:48.2052306Z   inflating: build/bin/IListRef_test  
2025-04-21T21:50:48.2164409Z   inflating: build/bin/List_test     
2025-04-21T21:50:48.2234113Z   inflating: build/bin/KernelFunction_test  
2025-04-21T21:50:48.2360216Z   inflating: build/bin/kernel_function_legacy_test  
2025-04-21T21:50:48.2461636Z   inflating: build/bin/kernel_function_test  
2025-04-21T21:50:48.2595155Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-04-21T21:50:48.2702937Z   inflating: build/bin/kernel_lambda_test  
2025-04-21T21:50:48.2767789Z   inflating: build/bin/kernel_stackbased_test  
2025-04-21T21:50:48.2867864Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-04-21T21:50:48.2922901Z   inflating: build/bin/CppSignature_test  
2025-04-21T21:50:48.2981904Z   inflating: build/bin/backend_fallback_test  
2025-04-21T21:50:48.3034312Z   inflating: build/bin/op_allowlist_test  
2025-04-21T21:50:48.3342644Z   inflating: build/bin/op_registration_test  
2025-04-21T21:50:48.3411730Z   inflating: build/bin/inline_container_test  
2025-04-21T21:50:48.3468355Z   inflating: build/bin/cuda_allocator_test  
2025-04-21T21:50:48.3524519Z   inflating: build/bin/cuda_apply_test  
2025-04-21T21:50:48.3587560Z   inflating: build/bin/cuda_atomic_ops_test  
2025-04-21T21:50:48.3648730Z   inflating: build/bin/cuda_caching_host_allocator_test  
2025-04-21T21:50:48.3722962Z   inflating: build/bin/cuda_complex_math_test  
2025-04-21T21:50:48.3786070Z   inflating: build/bin/cuda_complex_test  
2025-04-21T21:50:48.3852192Z   inflating: build/bin/cuda_cub_test  
2025-04-21T21:50:48.3905760Z   inflating: build/bin/cuda_device_test  
2025-04-21T21:50:48.3974955Z   inflating: build/bin/cuda_distributions_test  
2025-04-21T21:50:48.4029616Z   inflating: build/bin/cuda_dlconvertor_test  
2025-04-21T21:50:48.4090095Z   inflating: build/bin/cuda_generator_test  
2025-04-21T21:50:48.4143970Z   inflating: build/bin/cuda_half_test  
2025-04-21T21:50:48.4198678Z   inflating: build/bin/cuda_integer_divider_test  
2025-04-21T21:50:48.4252663Z   inflating: build/bin/cuda_optional_test  
2025-04-21T21:50:48.4308124Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2025-04-21T21:50:48.4364493Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2025-04-21T21:50:48.4417643Z   inflating: build/bin/cuda_allocatorTraceTracker_test  
2025-04-21T21:50:48.4482335Z   inflating: build/bin/cuda_stream_test  
2025-04-21T21:50:48.4538877Z   inflating: build/bin/cuda_vectorized_test  
2025-04-21T21:50:48.4592040Z   inflating: build/bin/cuda_cudnn_test  
2025-04-21T21:50:48.5176474Z   inflating: build/bin/test_jit      
2025-04-21T21:50:48.6019213Z   inflating: build/bin/test_tensorexpr  
2025-04-21T21:50:48.6033628Z   inflating: build/bin/tutorial_tensorexpr  
2025-04-21T21:50:48.6089658Z   inflating: build/bin/BackoffTest   
2025-04-21T21:50:48.6146498Z   inflating: build/bin/FileStoreTest  
2025-04-21T21:50:48.6205855Z   inflating: build/bin/TCPStoreTest  
2025-04-21T21:50:48.6263165Z   inflating: build/bin/HashStoreTest  
2025-04-21T21:50:48.6333723Z   inflating: build/bin/ProcessGroupGlooTest  
2025-04-21T21:50:48.6394613Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2025-04-21T21:50:48.6463067Z   inflating: build/bin/ProcessGroupNCCLTest  
2025-04-21T21:50:48.6476970Z   inflating: build/bin/ProcessGroupMPITest  
2025-04-21T21:50:48.6541995Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2025-04-21T21:50:48.6545510Z   inflating: build/bin/example_allreduce  
2025-04-21T21:50:48.6603872Z   inflating: build/bin/test_dist_autograd  
2025-04-21T21:50:48.6676539Z   inflating: build/bin/test_cpp_rpc  
2025-04-21T21:50:48.7880815Z   inflating: build/bin/test_api      
2025-04-21T21:50:48.7883649Z   inflating: build/bin/parallel_benchmark  
2025-04-21T21:50:48.7954165Z   inflating: build/bin/test_mobile_nnc  
2025-04-21T21:50:48.7963419Z   inflating: build/bin/aot_model_compiler_test  
2025-04-21T21:50:48.8318958Z   inflating: build/bin/test_lazy     
2025-04-21T21:50:48.8327132Z   inflating: build/bin/torch_shm_manager  
2025-04-21T21:50:48.8327501Z    creating: .additional_ci_files/
2025-04-21T21:50:48.8444270Z   inflating: .additional_ci_files/test-times.json  
2025-04-21T21:50:48.8894089Z   inflating: .additional_ci_files/test-class-times.json  
2025-04-21T21:50:48.8930794Z ##[group]Run rm artifacts.zip
2025-04-21T21:50:48.8931104Z rm artifacts.zip
2025-04-21T21:50:48.8940075Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:48.8940437Z env:
2025-04-21T21:50:48.8940651Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:48.8940980Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:48.8941332Z ##[endgroup]
2025-04-21T21:50:49.0096932Z ##[group]Run df -H
2025-04-21T21:50:49.0097189Z df -H
2025-04-21T21:50:49.0105924Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:49.0106285Z env:
2025-04-21T21:50:49.0106503Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.0106833Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.0107181Z ##[endgroup]
2025-04-21T21:50:49.0174619Z Filesystem        Size  Used Avail Use% Mounted on
2025-04-21T21:50:49.0175206Z devtmpfs          4.2M     0  4.2M   0% /dev
2025-04-21T21:50:49.0175683Z tmpfs              34G     0   34G   0% /dev/shm
2025-04-21T21:50:49.0176200Z tmpfs              14G  553k   14G   1% /run
2025-04-21T21:50:49.0176665Z /dev/nvme0n1p1    161G   59G  103G  37% /
2025-04-21T21:50:49.0177134Z tmpfs              34G   25k   34G   1% /tmp
2025-04-21T21:50:49.0177618Z /dev/nvme0n1p128   11M  1.4M  9.2M  13% /boot/efi
2025-04-21T21:50:49.0178244Z tmpfs             6.7G     0  6.7G   0% /run/user/0
2025-04-21T21:50:49.0212052Z Prepare all required actions
2025-04-21T21:50:49.0212415Z Getting action download info
2025-04-21T21:50:49.1351338Z ##[group]Run ./.github/actions/download-td-artifacts
2025-04-21T21:50:49.1351685Z with:
2025-04-21T21:50:49.1351892Z env:
2025-04-21T21:50:49.1352100Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.1352428Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.1352799Z ##[endgroup]
2025-04-21T21:50:49.1383848Z ##[group]Run seemethere/download-artifact-s3@v4
2025-04-21T21:50:49.1384180Z with:
2025-04-21T21:50:49.1384384Z   name: td_results
2025-04-21T21:50:49.1384612Z   s3-bucket: gha-artifacts
2025-04-21T21:50:49.1384870Z   region: us-east-1
2025-04-21T21:50:49.1385089Z env:
2025-04-21T21:50:49.1385292Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.1385616Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.1385964Z ##[endgroup]
2025-04-21T21:50:49.6053306Z (node:107329) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-04-21T21:50:49.6053798Z 
2025-04-21T21:50:49.6053996Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-04-21T21:50:49.6054511Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-04-21T21:50:49.6055051Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-04-21T21:50:49.7010908Z Found 0 objects with prefix pytorch/pytorch/14580756816/td_results/
2025-04-21T21:50:49.7016632Z Artifact download has finished successfully
2025-04-21T21:50:49.7360921Z ##[group]Run mkdir -p .additional_ci_files
2025-04-21T21:50:49.7361291Z mkdir -p .additional_ci_files
2025-04-21T21:50:49.7361722Z mv td_results.json .additional_ci_files/td_results.json || true
2025-04-21T21:50:49.7370595Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:49.7370965Z env:
2025-04-21T21:50:49.7371186Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.7371527Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.7371885Z ##[endgroup]
2025-04-21T21:50:49.7440173Z mv: cannot stat 'td_results.json': No such file or directory
2025-04-21T21:50:49.7477360Z ##[group]Run .github/scripts/parse_ref.py
2025-04-21T21:50:49.7477738Z .github/scripts/parse_ref.py
2025-04-21T21:50:49.7485722Z shell: /usr/bin/bash -e {0}
2025-04-21T21:50:49.7486003Z env:
2025-04-21T21:50:49.7486238Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.7486572Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.7486935Z ##[endgroup]
2025-04-21T21:50:49.7820273Z Prepare all required actions
2025-04-21T21:50:49.7820951Z Getting action download info
2025-04-21T21:50:49.8831431Z ##[group]Run ./.github/actions/filter-test-configs
2025-04-21T21:50:49.8831759Z with:
2025-04-21T21:50:49.8832132Z   github-token: ***
2025-04-21T21:50:49.8833838Z   test-matrix: {"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}
2025-04-21T21:50:49.8835946Z   job-name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:49.8836448Z env:
2025-04-21T21:50:49.8836829Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.8837149Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.8837492Z ##[endgroup]
2025-04-21T21:50:49.8874119Z ##[group]Run nick-fields/retry@v3.0.0
2025-04-21T21:50:49.8874423Z with:
2025-04-21T21:50:49.8874621Z   shell: bash
2025-04-21T21:50:49.8874845Z   timeout_minutes: 10
2025-04-21T21:50:49.8875090Z   max_attempts: 5
2025-04-21T21:50:49.8875329Z   retry_wait_seconds: 30
2025-04-21T21:50:49.8876084Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2025-04-21T21:50:49.8876931Z   polling_interval_seconds: 1
2025-04-21T21:50:49.8877215Z   warning_on_retry: true
2025-04-21T21:50:49.8877473Z   continue_on_error: false
2025-04-21T21:50:49.8877736Z env:
2025-04-21T21:50:49.8877951Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:49.8878286Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:49.8878773Z   GITHUB_TOKEN: ***
2025-04-21T21:50:49.8879006Z ##[endgroup]
2025-04-21T21:50:49.9866215Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2025-04-21T21:50:50.2156358Z Defaulting to user installation because normal site-packages is not writeable
2025-04-21T21:50:50.2329407Z Requirement already satisfied: requests==2.27.1 in /home/ec2-user/.local/lib/python3.9/site-packages (2.27.1)
2025-04-21T21:50:50.2332808Z Requirement already satisfied: pyyaml==6.0.1 in /home/ec2-user/.local/lib/python3.9/site-packages (6.0.1)
2025-04-21T21:50:50.2441513Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10)
2025-04-21T21:50:50.2449604Z Requirement already satisfied: charset-normalizer~=2.0.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2.0.12)
2025-04-21T21:50:50.2458942Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10)
2025-04-21T21:50:50.2462505Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.9/site-packages (from requests==2.27.1) (2025.1.31)
2025-04-21T21:50:50.9652164Z Command completed after 1 attempt(s).
2025-04-21T21:50:50.9725389Z ##[group]Run set -x
2025-04-21T21:50:50.9725809Z set -x
2025-04-21T21:50:50.9726161Z 
2025-04-21T21:50:50.9726719Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-04-21T21:50:50.9727530Z # in runner workspace
2025-04-21T21:50:50.9728127Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-04-21T21:50:50.9739808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:50.9740365Z env:
2025-04-21T21:50:50.9740688Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:50.9741212Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:50.9741755Z ##[endgroup]
2025-04-21T21:50:50.9770654Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-04-21T21:50:51.0006256Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-04-21T21:50:51.0006657Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-04-21T21:50:51.0006979Z echo "Job name: ${JOB_NAME}"
2025-04-21T21:50:51.0007261Z 
2025-04-21T21:50:51.0007628Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-04-21T21:50:51.0008085Z # in runner workspace
2025-04-21T21:50:51.0008497Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-04-21T21:50:51.0008962Z   --workflow "${GITHUB_WORKFLOW}" \
2025-04-21T21:50:51.0009290Z   --job-name "${JOB_NAME}" \
2025-04-21T21:50:51.0011027Z   --test-matrix "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}" \
2025-04-21T21:50:51.0013003Z   --selected-test-configs "" \
2025-04-21T21:50:51.0013336Z   --pr-number "${PR_NUMBER}" \
2025-04-21T21:50:51.0013642Z   --tag "${TAG}" \
2025-04-21T21:50:51.0013930Z   --event-name "${EVENT_NAME}" \
2025-04-21T21:50:51.0014244Z   --schedule "${SCHEDULE}" \
2025-04-21T21:50:51.0014547Z   --branch "${HEAD_BRANCH}"
2025-04-21T21:50:51.0023545Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:51.0023906Z env:
2025-04-21T21:50:51.0024118Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:51.0024448Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:51.0024982Z   GITHUB_TOKEN: ***
2025-04-21T21:50:51.0025422Z   JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:51.0025921Z   PR_NUMBER: 
2025-04-21T21:50:51.0026151Z   TAG: ciflow/inductor/148893
2025-04-21T21:50:51.0026410Z   EVENT_NAME: push
2025-04-21T21:50:51.0026633Z   SCHEDULE: 
2025-04-21T21:50:51.0026840Z   HEAD_BRANCH: 
2025-04-21T21:50:51.0027047Z ##[endgroup]
2025-04-21T21:50:51.0055071Z Workflow: inductor
2025-04-21T21:50:51.0055560Z Job name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:51.2124533Z INFO:root:Found no test-config label on the PR, so all test configs are included
2025-04-21T21:50:51.3282198Z ##[group]Run echo "Filtered matrix:"
2025-04-21T21:50:51.3282557Z echo "Filtered matrix:"
2025-04-21T21:50:51.3284324Z echo "{"include": [{"config": "inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}, {"config": "inductor_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.g5.4xlarge.nvidia.gpu"}]}"
2025-04-21T21:50:51.3286072Z 
2025-04-21T21:50:51.3286285Z echo
2025-04-21T21:50:51.3286565Z echo "Is the current job unstable? False"
2025-04-21T21:50:51.3286895Z 
2025-04-21T21:50:51.3287096Z echo
2025-04-21T21:50:51.3287403Z echo "Is keep-going label set? False"
2025-04-21T21:50:51.3287724Z 
2025-04-21T21:50:51.3287926Z echo
2025-04-21T21:50:51.3288171Z echo "Renabled issues? "
2025-04-21T21:50:51.3296905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:51.3297467Z env:
2025-04-21T21:50:51.3297683Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:51.3298014Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:51.3298370Z ##[endgroup]
2025-04-21T21:50:51.3326793Z Filtered matrix:
2025-04-21T21:50:51.3329630Z {include: [{config: inductor_huggingface, shard: 1, num_shards: 1, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_timm, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 1, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}, {config: inductor_torchbench, shard: 2, num_shards: 2, runner: linux.g5.4xlarge.nvidia.gpu}]}
2025-04-21T21:50:51.3332223Z 
2025-04-21T21:50:51.3332408Z Is the current job unstable? False
2025-04-21T21:50:51.3333064Z 
2025-04-21T21:50:51.3333237Z Is keep-going label set? False
2025-04-21T21:50:51.3333518Z 
2025-04-21T21:50:51.3333664Z Renabled issues? 
2025-04-21T21:50:51.3372150Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-04-21T21:50:51.3372664Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-04-21T21:50:51.3380865Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-21T21:50:51.3381236Z env:
2025-04-21T21:50:51.3381451Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:51.3381780Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:51.3382138Z   JOB_TIMEOUT: 240
2025-04-21T21:50:51.3382375Z ##[endgroup]
2025-04-21T21:50:51.3453587Z ##[group]Run set -x
2025-04-21T21:50:51.3453928Z set -x
2025-04-21T21:50:51.3454162Z 
2025-04-21T21:50:51.3454419Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-04-21T21:50:51.3454824Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-04-21T21:50:51.3455236Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-04-21T21:50:51.3455608Z   TEST_COMMAND=.ci/onnx/test.sh
2025-04-21T21:50:51.3455906Z else
2025-04-21T21:50:51.3456163Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-04-21T21:50:51.3456473Z fi
2025-04-21T21:50:51.3456679Z 
2025-04-21T21:50:51.3456947Z # Leaving 1GB for the runner and other things
2025-04-21T21:50:51.3457562Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo)
2025-04-21T21:50:51.3458385Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap
2025-04-21T21:50:51.3459078Z # comes from https://github.com/pytorch/test-infra/pull/6058
2025-04-21T21:50:51.3459603Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3))
2025-04-21T21:50:51.3460015Z 
2025-04-21T21:50:51.3460291Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then
2025-04-21T21:50:51.3460634Z   SHM_OPTS=
2025-04-21T21:50:51.3460888Z   JENKINS_USER=
2025-04-21T21:50:51.3461237Z   # ensure that docker container cleanly exits in 12 hours
2025-04-21T21:50:51.3461709Z   # if for some reason cleanup action doesn't stop container
2025-04-21T21:50:51.3462099Z   # when job is cancelled
2025-04-21T21:50:51.3462408Z   DOCKER_SHELL_CMD="sleep 12h"
2025-04-21T21:50:51.3462704Z 
2025-04-21T21:50:51.3463062Z   # since some steps are skipped on s390x, if they are necessary, run them here
2025-04-21T21:50:51.3463600Z   env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-04-21T21:50:51.3464042Z   env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2025-04-21T21:50:51.3464393Z else
2025-04-21T21:50:51.3464644Z   SHM_OPTS="--shm-size=${SHM_SIZE}"
2025-04-21T21:50:51.3464986Z   JENKINS_USER="--user jenkins"
2025-04-21T21:50:51.3465302Z   DOCKER_SHELL_CMD=
2025-04-21T21:50:51.3465559Z fi
2025-04-21T21:50:51.3465770Z 
2025-04-21T21:50:51.3466101Z # detached container should get cleaned up by teardown_ec2_linux
2025-04-21T21:50:51.3466623Z # TODO: Stop building test binaries as part of the build phase
2025-04-21T21:50:51.3467207Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice
2025-04-21T21:50:51.3467736Z # shellcheck disable=SC2086,SC2090
2025-04-21T21:50:51.3468079Z container_name=$(docker run \
2025-04-21T21:50:51.3468385Z   ${GPU_FLAG:-} \
2025-04-21T21:50:51.3468693Z   ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \
2025-04-21T21:50:51.3469042Z   -e BUILD_ENVIRONMENT \
2025-04-21T21:50:51.3469335Z   -e PR_NUMBER \
2025-04-21T21:50:51.3469612Z   -e GITHUB_ACTIONS \
2025-04-21T21:50:51.3469907Z   -e GITHUB_REPOSITORY \
2025-04-21T21:50:51.3470375Z   -e GITHUB_WORKFLOW \
2025-04-21T21:50:51.3470658Z   -e GITHUB_JOB \
2025-04-21T21:50:51.3470933Z   -e GITHUB_RUN_ID \
2025-04-21T21:50:51.3471216Z   -e GITHUB_RUN_NUMBER \
2025-04-21T21:50:51.3471509Z   -e GITHUB_RUN_ATTEMPT \
2025-04-21T21:50:51.3471800Z   -e JOB_ID \
2025-04-21T21:50:51.3472059Z   -e JOB_NAME \
2025-04-21T21:50:51.3472317Z   -e BASE_SHA \
2025-04-21T21:50:51.3472581Z   -e BRANCH \
2025-04-21T21:50:51.3472834Z   -e SHA1 \
2025-04-21T21:50:51.3473088Z   -e AWS_DEFAULT_REGION \
2025-04-21T21:50:51.3473610Z   -e IN_WHEEL_TEST \
2025-04-21T21:50:51.3473896Z   -e SHARD_NUMBER \
2025-04-21T21:50:51.3474181Z   -e TEST_CONFIG \
2025-04-21T21:50:51.3474453Z   -e NUM_TEST_SHARDS \
2025-04-21T21:50:51.3474750Z   -e REENABLED_ISSUES \
2025-04-21T21:50:51.3475055Z   -e CONTINUE_THROUGH_ERROR \
2025-04-21T21:50:51.3475375Z   -e VERBOSE_TEST_LOGS \
2025-04-21T21:50:51.3475669Z   -e TEST_SHOWLOCALS \
2025-04-21T21:50:51.3475959Z   -e NO_TEST_TIMEOUT \
2025-04-21T21:50:51.3476230Z   -e NO_TD \
2025-04-21T21:50:51.3476491Z   -e TD_DISTRIBUTED \
2025-04-21T21:50:51.3476774Z   -e PR_LABELS \
2025-04-21T21:50:51.3477074Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-04-21T21:50:51.3477428Z   -e SCCACHE_BUCKET \
2025-04-21T21:50:51.3477738Z   -e SCCACHE_REGION \
2025-04-21T21:50:51.3478020Z   -e XLA_CUDA \
2025-04-21T21:50:51.3478304Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2025-04-21T21:50:51.3478675Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-04-21T21:50:51.3479049Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-04-21T21:50:51.3479416Z   -e SKIP_SCCACHE_INITIALIZATION=1 \
2025-04-21T21:50:51.3479760Z   -e HUGGING_FACE_HUB_TOKEN \
2025-04-21T21:50:51.3480101Z   -e SCRIBE_GRAPHQL_ACCESS_TOKEN \
2025-04-21T21:50:51.3480434Z   -e DASHBOARD_TAG \
2025-04-21T21:50:51.3480710Z   -e IS_A100_RUNNER \
2025-04-21T21:50:51.3481002Z   -e ARTIFACTS_FILE_SUFFIX \
2025-04-21T21:50:51.3481367Z   --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \
2025-04-21T21:50:51.3481775Z   --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \
2025-04-21T21:50:51.3482191Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2025-04-21T21:50:51.3482589Z   --security-opt seccomp=unconfined \
2025-04-21T21:50:51.3482940Z   --cap-add=SYS_PTRACE \
2025-04-21T21:50:51.3483237Z   --ipc=host \
2025-04-21T21:50:51.3483501Z   ${SHM_OPTS} \
2025-04-21T21:50:51.3483759Z   --tty \
2025-04-21T21:50:51.3483996Z   --detach \
2025-04-21T21:50:51.3484269Z   --name="${container_name}" \
2025-04-21T21:50:51.3484588Z   ${JENKINS_USER} \
2025-04-21T21:50:51.3484936Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-04-21T21:50:51.3485342Z   -w /var/lib/jenkins/workspace \
2025-04-21T21:50:51.3485665Z   "${DOCKER_IMAGE}" \
2025-04-21T21:50:51.3485942Z   ${DOCKER_SHELL_CMD}
2025-04-21T21:50:51.3486210Z )
2025-04-21T21:50:51.3486508Z # Propagate download.pytorch.org IP to container
2025-04-21T21:50:51.3487196Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts"
2025-04-21T21:50:51.3487896Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2025-04-21T21:50:51.3488304Z 
2025-04-21T21:50:51.3488577Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then
2025-04-21T21:50:51.3489148Z   docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt"
2025-04-21T21:50:51.3489653Z fi
2025-04-21T21:50:51.3489961Z 
2025-04-21T21:50:51.3490443Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2025-04-21T21:50:51.3498765Z shell: /usr/bin/bash -e {0}
2025-04-21T21:50:51.3499034Z env:
2025-04-21T21:50:51.3499252Z   GIT_DEFAULT_BRANCH: main
2025-04-21T21:50:51.3499578Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:50:51.3500023Z   BUILD_ENVIRONMENT: linux-focal-cuda12.6-py3.10-gcc9-sm86
2025-04-21T21:50:51.3500392Z   PR_NUMBER: 
2025-04-21T21:50:51.3500638Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-04-21T21:50:51.3500940Z   GITHUB_WORKFLOW: inductor
2025-04-21T21:50:51.3501200Z   GITHUB_JOB: test
2025-04-21T21:50:51.3501553Z   GITHUB_RUN_ID: 14580756816
2025-04-21T21:50:51.3501823Z   GITHUB_RUN_NUMBER: 128203
2025-04-21T21:50:51.3502090Z   GITHUB_RUN_ATTEMPT: 1
2025-04-21T21:50:51.3502338Z   JOB_ID: 40900324329
2025-04-21T21:50:51.3502798Z   JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:50:51.3503314Z   BRANCH: 
2025-04-21T21:50:51.3503573Z   SHA1: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:50:51.3503951Z   BASE_SHA: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:50:51.3504313Z   TEST_CONFIG: inductor_torchbench
2025-04-21T21:50:51.3504612Z   SHARD_NUMBER: 2
2025-04-21T21:50:51.3504843Z   NUM_TEST_SHARDS: 2
2025-04-21T21:50:51.3505081Z   REENABLED_ISSUES: 
2025-04-21T21:50:51.3505332Z   CONTINUE_THROUGH_ERROR: False
2025-04-21T21:50:51.3505619Z   VERBOSE_TEST_LOGS: False
2025-04-21T21:50:51.3505886Z   TEST_SHOWLOCALS: False
2025-04-21T21:50:51.3506153Z   NO_TEST_TIMEOUT: False
2025-04-21T21:50:51.3506405Z   NO_TD: False
2025-04-21T21:50:51.3506643Z   TD_DISTRIBUTED: False
2025-04-21T21:50:51.3506958Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2025-04-21T21:50:51.3507317Z   SCCACHE_REGION: us-east-1
2025-04-21T21:50:51.3507577Z   SHM_SIZE: 2g
2025-04-21T21:50:51.3508365Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:50:51.3509238Z   XLA_CUDA: 
2025-04-21T21:50:51.3509595Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2025-04-21T21:50:51.3510046Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-04-21T21:50:51.3510372Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-04-21T21:50:51.3510674Z   DASHBOARD_TAG: 
2025-04-21T21:50:51.3511074Z   HUGGING_FACE_HUB_TOKEN: ***
2025-04-21T21:50:51.3511479Z   SCRIBE_GRAPHQL_ACCESS_TOKEN: ***
2025-04-21T21:50:51.3511774Z   IS_A100_RUNNER: 0
2025-04-21T21:50:51.3512235Z   ARTIFACTS_FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-21T21:50:51.3512761Z ##[endgroup]
2025-04-21T21:50:51.3539781Z + [[ inductor_torchbench == \m\u\l\t\i\g\p\u ]]
2025-04-21T21:50:51.3540195Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *onnx* ]]
2025-04-21T21:50:51.3540572Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-04-21T21:50:51.3543301Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo
2025-04-21T21:50:51.3565717Z + TOTAL_AVAILABLE_MEMORY_IN_GB='61.094 '
2025-04-21T21:50:51.3566050Z + TOTAL_MEMORY_WITH_SWAP=64
2025-04-21T21:50:51.3566417Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]]
2025-04-21T21:50:51.3566805Z + SHM_OPTS=--shm-size=2g
2025-04-21T21:50:51.3576525Z + JENKINS_USER='--user jenkins'
2025-04-21T21:50:51.3576799Z + DOCKER_SHELL_CMD=
2025-04-21T21:50:51.3577036Z +++ nproc --ignore=2
2025-04-21T21:50:51.3591096Z ++ docker run --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=14 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e IS_A100_RUNNER -e ARTIFACTS_FILE_SUFFIX --memory=61g --memory-swap=64g --env-file=/tmp/github_env_14580756816 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-21T21:51:03.3206581Z + container_name=3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-21T21:51:03.3209607Z + grep download.pytorch.org /etc/hosts
2025-04-21T21:51:03.3210959Z + docker exec -i 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746 sudo bash -c '/bin/cat >> /etc/hosts'
2025-04-21T21:51:03.4801094Z + echo DOCKER_CONTAINER_ID=3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-21T21:51:03.4801775Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *\s\3\9\0\x* ]]
2025-04-21T21:51:03.4805216Z ++ echo dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl
2025-04-21T21:51:03.4808270Z + docker exec -t 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746 sh -c 'python3 -m pip install dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh'
2025-04-21T21:51:03.8903294Z Processing ./dist/torch-2.8.0a0+gitb68c0ef-cp310-cp310-linux_x86_64.whl (from torch==2.8.0a0+gitb68c0ef)
2025-04-21T21:51:04.2246410Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.16.1)
2025-04-21T21:51:04.2249528Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (4.13.2)
2025-04-21T21:51:04.2551746Z Collecting sympy>=1.13.3 (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef)
2025-04-21T21:51:04.2563671Z   Using cached sympy-1.13.3-py3-none-any.whl.metadata (12 kB)
2025-04-21T21:51:04.2580927Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (2.8.8)
2025-04-21T21:51:04.2584027Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.1.6)
2025-04-21T21:51:04.2587332Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (2024.10.0)
2025-04-21T21:51:04.2601704Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.3.0)
2025-04-21T21:51:04.2619270Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (1.22.4)
2025-04-21T21:51:04.2627331Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (1.3.0)
2025-04-21T21:51:04.3007598Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.8.0a0+gitb68c0ef->torch==2.8.0a0+gitb68c0ef) (3.0.2)
2025-04-21T21:51:04.3088395Z Using cached sympy-1.13.3-py3-none-any.whl (6.2 MB)
2025-04-21T21:51:04.9501006Z Installing collected packages: sympy, torch
2025-04-21T21:51:04.9501392Z   Attempting uninstall: sympy
2025-04-21T21:51:04.9512273Z     Found existing installation: sympy 1.13.1
2025-04-21T21:51:05.1550754Z     Uninstalling sympy-1.13.1:
2025-04-21T21:51:06.4614679Z       Successfully uninstalled sympy-1.13.1
2025-04-21T21:51:21.1848773Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-04-21T21:51:21.1849742Z timm 1.0.14 requires torchvision, which is not installed.
2025-04-21T21:51:21.1850298Z Successfully installed sympy-1.13.3 torch-2.8.0a0+gitb68c0ef
2025-04-21T21:51:21.2744044Z + export TERM=vt100
2025-04-21T21:51:21.2744647Z + TERM=vt100
2025-04-21T21:51:21.2747096Z ++ dirname .ci/pytorch/test.sh
2025-04-21T21:51:21.2756809Z + source .ci/pytorch/common.sh
2025-04-21T21:51:21.2759775Z +++ dirname .ci/pytorch/common.sh
2025-04-21T21:51:21.2769077Z ++ source .ci/pytorch/common_utils.sh
2025-04-21T21:51:21.2770782Z +++ declare -f -t trap_add
2025-04-21T21:51:21.2776917Z ++ set -ex -o pipefail
2025-04-21T21:51:21.2777253Z ++ [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]]
2025-04-21T21:51:21.2777691Z ++ BUILD_TEST_LIBTORCH=0
2025-04-21T21:51:21.2779946Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *rocm* ]]
2025-04-21T21:51:21.2780362Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *s390x* ]]
2025-04-21T21:51:21.2780720Z + [[ -d /var/lib/jenkins/workspace ]]
2025-04-21T21:51:21.2782974Z ++ stat -c %u /var/lib/jenkins/workspace
2025-04-21T21:51:21.2805701Z + WORKSPACE_ORIGINAL_OWNER_ID=1000
2025-04-21T21:51:21.2806024Z + trap_add cleanup_workspace EXIT
2025-04-21T21:51:21.2806322Z + trap_add_cmd=cleanup_workspace
2025-04-21T21:51:21.2806600Z + shift
2025-04-21T21:51:21.2806834Z + for trap_add_name in "$@"
2025-04-21T21:51:21.2816611Z +++ trap -p EXIT
2025-04-21T21:51:21.2818989Z ++ eval 'extract_trap_cmd '
2025-04-21T21:51:21.2819320Z +++ extract_trap_cmd
2025-04-21T21:51:21.2819572Z +++ printf '%s\n' ''
2025-04-21T21:51:21.2819931Z ++ printf '%s\n' cleanup_workspace
2025-04-21T21:51:21.2822877Z + trap -- '
2025-04-21T21:51:21.2823270Z cleanup_workspace' EXIT
2025-04-21T21:51:21.2823588Z + sudo chown -R jenkins /var/lib/jenkins/workspace
2025-04-21T21:51:22.0858912Z + git config --global --add safe.directory /var/lib/jenkins/workspace
2025-04-21T21:51:22.0893628Z + echo 'Environment variables:'
2025-04-21T21:51:22.0894021Z Environment variables:
2025-04-21T21:51:22.0894273Z + env
2025-04-21T21:51:22.0904799Z NV_LIBCUBLAS_VERSION=12.6.4.1-1
2025-04-21T21:51:22.0905262Z NVIDIA_VISIBLE_DEVICES=all
2025-04-21T21:51:22.0905625Z NV_NVML_DEV_VERSION=12.6.77-1
2025-04-21T21:51:22.0906180Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-04-21T21:51:22.0906683Z CONTINUE_THROUGH_ERROR=False
2025-04-21T21:51:22.0907094Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6
2025-04-21T21:51:22.0907615Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1
2025-04-21T21:51:22.0908131Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86
2025-04-21T21:51:22.0908630Z HOSTNAME=3e061bd09dd7
2025-04-21T21:51:22.0909348Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.0909958Z GITHUB_ACTION=__self
2025-04-21T21:51:22.0910219Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-04-21T21:51:22.0914894Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551
2025-04-21T21:51:22.0920113Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1
2025-04-21T21:51:22.0920620Z NV_NVTX_VERSION=12.6.77-1
2025-04-21T21:51:22.0920882Z GITHUB_RUN_NUMBER=128203
2025-04-21T21:51:22.0921141Z TEST_CONFIG=inductor_torchbench
2025-04-21T21:51:22.0921689Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-04-21T21:51:22.0922063Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2025-04-21T21:51:22.0922349Z IS_A100_RUNNER=0
2025-04-21T21:51:22.0922592Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1
2025-04-21T21:51:22.0922890Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1
2025-04-21T21:51:22.0923382Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2025-04-21T21:51:22.0923661Z NV_LIBNPP_VERSION=12.3.1.54-1
2025-04-21T21:51:22.0923956Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-04-21T21:51:22.0924311Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2025-04-21T21:51:22.0924640Z GITHUB_REF_TYPE=tag
2025-04-21T21:51:22.0924876Z TORCH_CUDA_ARCH_LIST=Maxwell
2025-04-21T21:51:22.0925139Z NCCL_VERSION=2.23.4-1
2025-04-21T21:51:22.0925416Z BASE_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.0925736Z XLA_CUDA=
2025-04-21T21:51:22.0925963Z NCCL_LIB_DIR=/usr/local/cuda/lib64/
2025-04-21T21:51:22.0926356Z HUGGING_FACE_HUB_TOKEN=***
2025-04-21T21:51:22.0926691Z ***
2025-04-21T21:51:22.0926905Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2025-04-21T21:51:22.0927199Z GITHUB_REPOSITORY_ID=65600975
2025-04-21T21:51:22.0927460Z GITHUB_ACTIONS=true
2025-04-21T21:51:22.0927710Z NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:51:22.0928025Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1
2025-04-21T21:51:22.0928374Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1
2025-04-21T21:51:22.0928723Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.0929099Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2025-04-21T21:51:22.0929447Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.0930031Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/148893
2025-04-21T21:51:22.0930558Z UCC_HOME=/usr
2025-04-21T21:51:22.0930792Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1
2025-04-21T21:51:22.0931081Z VERBOSE_TEST_LOGS=False
2025-04-21T21:51:22.0931334Z NVIDIA_PRODUCT_NAME=CUDA
2025-04-21T21:51:22.0931635Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6
2025-04-21T21:51:22.0931997Z GITHUB_REF=refs/tags/ciflow/inductor/148893
2025-04-21T21:51:22.0932308Z NV_CUDA_CUDART_VERSION=12.6.77-1
2025-04-21T21:51:22.0932578Z SHARD_NUMBER=2
2025-04-21T21:51:22.0932813Z GITHUB_REF_PROTECTED=false
2025-04-21T21:51:22.0933063Z HOME=/var/lib/jenkins
2025-04-21T21:51:22.0933339Z GITHUB_API_URL=https://api.github.com
2025-04-21T21:51:22.0933660Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-04-21T21:51:22.0933998Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2025-04-21T21:51:22.0934321Z CUDA_VERSION=12.6.3
2025-04-21T21:51:22.0934587Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1
2025-04-21T21:51:22.0934896Z USE_SYSTEM_NCCL=1
2025-04-21T21:51:22.0935112Z NUM_TEST_SHARDS=2
2025-04-21T21:51:22.0935332Z UCX_HOME=/usr
2025-04-21T21:51:22.0935652Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1
2025-04-21T21:51:22.0936373Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.0937188Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:51:22.0937980Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.0939143Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2025-04-21T21:51:22.0939614Z GITHUB_EVENT_NAME=push
2025-04-21T21:51:22.0939853Z DASHBOARD_TAG=
2025-04-21T21:51:22.0940071Z GITHUB_RUN_ID=14580756816
2025-04-21T21:51:22.0940361Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1
2025-04-21T21:51:22.0940714Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6
2025-04-21T21:51:22.0941359Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.0942009Z GITHUB_ACTOR=pytorch-bot[bot]
2025-04-21T21:51:22.0942427Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1
2025-04-21T21:51:22.0942699Z PR_NUMBER=
2025-04-21T21:51:22.0942900Z GITHUB_RUN_ATTEMPT=1
2025-04-21T21:51:22.0943147Z ANACONDA_PYTHON_VERSION=3.10
2025-04-21T21:51:22.0943469Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-04-21T21:51:22.0943798Z TERM=vt100
2025-04-21T21:51:22.0944021Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1
2025-04-21T21:51:22.0944305Z INSTALLED_VISION=yes
2025-04-21T21:51:22.0944525Z BRANCH=
2025-04-21T21:51:22.0944728Z SCCACHE_REGION=us-east-1
2025-04-21T21:51:22.0944991Z OPENSSL_ROOT_DIR=/opt/openssl
2025-04-21T21:51:22.0945267Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2025-04-21T21:51:22.0945568Z CUDA_PATH=/usr/local/cuda
2025-04-21T21:51:22.0946052Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2025-04-21T21:51:22.0946592Z GITHUB_SERVER_URL=https://github.com
2025-04-21T21:51:22.0946935Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2025-04-21T21:51:22.0947274Z REENABLED_ISSUES=
2025-04-21T21:51:22.0947482Z SHLVL=1
2025-04-21T21:51:22.0947675Z MAX_JOBS=14
2025-04-21T21:51:22.0947890Z NV_CUDA_LIB_VERSION=12.6.3-1
2025-04-21T21:51:22.0948141Z NVARCH=x86_64
2025-04-21T21:51:22.0948353Z GITHUB_ACTOR_ID=54816060
2025-04-21T21:51:22.0948680Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.0949068Z GITHUB_REF_NAME=ciflow/inductor/148893
2025-04-21T21:51:22.0949483Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2025-04-21T21:51:22.0949891Z GITHUB_JOB=test
2025-04-21T21:51:22.0950147Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6
2025-04-21T21:51:22.0950535Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2025-04-21T21:51:22.0950902Z NO_TEST_TIMEOUT=False
2025-04-21T21:51:22.0951142Z TD_DISTRIBUTED=False
2025-04-21T21:51:22.0951394Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1
2025-04-21T21:51:22.0951710Z GITHUB_REPOSITORY=pytorch/pytorch
2025-04-21T21:51:22.0952000Z NV_NVPROF_VERSION=12.6.80-1
2025-04-21T21:51:22.0952258Z GITHUB_RETENTION_DAYS=90
2025-04-21T21:51:22.0952514Z OPENSSL_DIR=/opt/openssl
2025-04-21T21:51:22.0952769Z GITHUB_ACTION_REPOSITORY=
2025-04-21T21:51:22.0953587Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-21T21:51:22.0954342Z GITHUB_BASE_REF=
2025-04-21T21:51:22.0954787Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-21T21:51:22.0955302Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2025-04-21T21:51:22.0955559Z CI=true
2025-04-21T21:51:22.0955769Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1
2025-04-21T21:51:22.0956066Z GITHUB_REPOSITORY_OWNER=pytorch
2025-04-21T21:51:22.0956322Z JOB_ID=40900324329
2025-04-21T21:51:22.0956551Z INSTALLED_PROTOBUF=yes
2025-04-21T21:51:22.0956788Z GITHUB_HEAD_REF=
2025-04-21T21:51:22.0957005Z GITHUB_ACTION_REF=
2025-04-21T21:51:22.0957287Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2025-04-21T21:51:22.0957625Z TEST_SHOWLOCALS=False
2025-04-21T21:51:22.0957871Z GITHUB_WORKFLOW=inductor
2025-04-21T21:51:22.0958127Z DEBIAN_FRONTEND=noninteractive
2025-04-21T21:51:22.0958716Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.0959422Z NO_TD=False
2025-04-21T21:51:22.0959644Z SKIP_SCCACHE_INITIALIZATION=1
2025-04-21T21:51:22.0959939Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/
2025-04-21T21:51:22.0960230Z _=/usr/bin/env
2025-04-21T21:51:22.0960527Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-04-21T21:51:22.1087442Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-04-21T21:51:22.1088044Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-04-21T21:51:22.1088631Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-04-21T21:51:22.1089472Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-04-21T21:51:22.1089907Z + BUILD_DIR=build
2025-04-21T21:51:22.1090148Z + BUILD_RENAMED_DIR=build_renamed
2025-04-21T21:51:22.1090421Z + BUILD_BIN_DIR=build/bin
2025-04-21T21:51:22.1090665Z + SHARD_NUMBER=2
2025-04-21T21:51:22.1090882Z + NUM_TEST_SHARDS=2
2025-04-21T21:51:22.1091126Z + export TORCH_SERIALIZATION_DEBUG=1
2025-04-21T21:51:22.1091446Z + TORCH_SERIALIZATION_DEBUG=1
2025-04-21T21:51:22.1091720Z + export VALGRIND=ON
2025-04-21T21:51:22.1091953Z + VALGRIND=ON
2025-04-21T21:51:22.1092245Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *clang9* ]]
2025-04-21T21:51:22.1092665Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]]
2025-04-21T21:51:22.1093067Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *s390x* ]]
2025-04-21T21:51:22.1093400Z + [[ 0 == \1 ]]
2025-04-21T21:51:22.1093619Z + [[ False == \1 ]]
2025-04-21T21:51:22.1093906Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *bazel* ]]
2025-04-21T21:51:22.1096180Z ++ realpath build/custom_test_artifacts
2025-04-21T21:51:22.1107638Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2025-04-21T21:51:22.1108109Z + [[ -n '' ]]
2025-04-21T21:51:22.1108343Z + echo 'Environment variables'
2025-04-21T21:51:22.1108612Z Environment variables
2025-04-21T21:51:22.1108842Z + env
2025-04-21T21:51:22.1118184Z NV_LIBCUBLAS_VERSION=12.6.4.1-1
2025-04-21T21:51:22.1118470Z NVIDIA_VISIBLE_DEVICES=all
2025-04-21T21:51:22.1118828Z NV_NVML_DEV_VERSION=12.6.77-1
2025-04-21T21:51:22.1119380Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-04-21T21:51:22.1119886Z CONTINUE_THROUGH_ERROR=False
2025-04-21T21:51:22.1120213Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.23.4-1+cuda12.6
2025-04-21T21:51:22.1120634Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.23.4-1
2025-04-21T21:51:22.1121013Z BUILD_ENVIRONMENT=linux-focal-cuda12.6-py3.10-gcc9-sm86
2025-04-21T21:51:22.1121670Z HOSTNAME=3e061bd09dd7
2025-04-21T21:51:22.1122308Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.1122983Z GITHUB_ACTION=__self
2025-04-21T21:51:22.1123319Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-04-21T21:51:22.1127909Z NVIDIA_REQUIRE_CUDA=cuda>=12.6 brand=unknown,driver>=470,driver<471 brand=grid,driver>=470,driver<471 brand=tesla,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=vapps,driver>=470,driver<471 brand=vpc,driver>=470,driver<471 brand=vcs,driver>=470,driver<471 brand=vws,driver>=470,driver<471 brand=cloudgaming,driver>=470,driver<471 brand=unknown,driver>=535,driver<536 brand=grid,driver>=535,driver<536 brand=tesla,driver>=535,driver<536 brand=nvidia,driver>=535,driver<536 brand=quadro,driver>=535,driver<536 brand=quadrortx,driver>=535,driver<536 brand=nvidiartx,driver>=535,driver<536 brand=vapps,driver>=535,driver<536 brand=vpc,driver>=535,driver<536 brand=vcs,driver>=535,driver<536 brand=vws,driver>=535,driver<536 brand=cloudgaming,driver>=535,driver<536 brand=unknown,driver>=550,driver<551 brand=grid,driver>=550,driver<551 brand=tesla,driver>=550,driver<551 brand=nvidia,driver>=550,driver<551 brand=quadro,driver>=550,driver<551 brand=quadrortx,driver>=550,driver<551 brand=nvidiartx,driver>=550,driver<551 brand=vapps,driver>=550,driver<551 brand=vpc,driver>=550,driver<551 brand=vcs,driver>=550,driver<551 brand=vws,driver>=550,driver<551 brand=cloudgaming,driver>=550,driver<551
2025-04-21T21:51:22.1133283Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-12-6=12.6.4.1-1
2025-04-21T21:51:22.1133631Z NV_NVTX_VERSION=12.6.77-1
2025-04-21T21:51:22.1133879Z GITHUB_RUN_NUMBER=128203
2025-04-21T21:51:22.1134140Z TEST_CONFIG=inductor_torchbench
2025-04-21T21:51:22.1134424Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-04-21T21:51:22.1134727Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2025-04-21T21:51:22.1135015Z IS_A100_RUNNER=0
2025-04-21T21:51:22.1135252Z NV_CUDA_CUDART_DEV_VERSION=12.6.77-1
2025-04-21T21:51:22.1135742Z NV_LIBCUSPARSE_VERSION=12.5.4.2-1
2025-04-21T21:51:22.1136220Z SCRIBE_GRAPHQL_ACCESS_TOKEN=***
2025-04-21T21:51:22.1136505Z NV_LIBNPP_VERSION=12.3.1.54-1
2025-04-21T21:51:22.1136789Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-04-21T21:51:22.1137143Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2025-04-21T21:51:22.1137485Z GITHUB_REF_TYPE=tag
2025-04-21T21:51:22.1137726Z TORCH_CUDA_ARCH_LIST=Maxwell
2025-04-21T21:51:22.1137986Z NCCL_VERSION=2.23.4-1
2025-04-21T21:51:22.1138271Z BASE_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.1138584Z XLA_CUDA=
2025-04-21T21:51:22.1138800Z NCCL_LIB_DIR=/usr/local/cuda/lib64/
2025-04-21T21:51:22.1139210Z HUGGING_FACE_HUB_TOKEN=***
2025-04-21T21:51:22.1139506Z ***
2025-04-21T21:51:22.1139713Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2025-04-21T21:51:22.1139994Z GITHUB_REPOSITORY_ID=65600975
2025-04-21T21:51:22.1140254Z GITHUB_ACTIONS=true
2025-04-21T21:51:22.1140497Z NVIDIA_DRIVER_CAPABILITIES=all
2025-04-21T21:51:22.1140802Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-12-6=12.6.80-1
2025-04-21T21:51:22.1141155Z NV_LIBNPP_PACKAGE=libnpp-12-6=12.3.1.54-1
2025-04-21T21:51:22.1141492Z SHA1=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.1141819Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2025-04-21T21:51:22.1142162Z GITHUB_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.1142738Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor.yml@refs/tags/ciflow/inductor/148893
2025-04-21T21:51:22.1143250Z UCC_HOME=/usr
2025-04-21T21:51:22.1143486Z TORCH_SERIALIZATION_DEBUG=1
2025-04-21T21:51:22.1143766Z NV_LIBCUBLAS_DEV_VERSION=12.6.4.1-1
2025-04-21T21:51:22.1144055Z VERBOSE_TEST_LOGS=False
2025-04-21T21:51:22.1144294Z NVIDIA_PRODUCT_NAME=CUDA
2025-04-21T21:51:22.1144587Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-12-6
2025-04-21T21:51:22.1144940Z GITHUB_REF=refs/tags/ciflow/inductor/148893
2025-04-21T21:51:22.1145246Z NV_CUDA_CUDART_VERSION=12.6.77-1
2025-04-21T21:51:22.1145515Z SHARD_NUMBER=2
2025-04-21T21:51:22.1145741Z GITHUB_REF_PROTECTED=false
2025-04-21T21:51:22.1145992Z HOME=/var/lib/jenkins
2025-04-21T21:51:22.1146258Z GITHUB_API_URL=https://api.github.com
2025-04-21T21:51:22.1146573Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-04-21T21:51:22.1146898Z UCX_COMMIT=7bb2722ff2187a0cad557ae4a6afa090569f83fb
2025-04-21T21:51:22.1147226Z CUDA_VERSION=12.6.3
2025-04-21T21:51:22.1147491Z NV_LIBCUBLAS_PACKAGE=libcublas-12-6=12.6.4.1-1
2025-04-21T21:51:22.1147792Z USE_SYSTEM_NCCL=1
2025-04-21T21:51:22.1148013Z NUM_TEST_SHARDS=2
2025-04-21T21:51:22.1148229Z UCX_HOME=/usr
2025-04-21T21:51:22.1148547Z NV_CUDA_NSIGHT_COMPUTE_DEV_PACKAGE=cuda-nsight-compute-12-6=12.6.3-1
2025-04-21T21:51:22.1149318Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.1150118Z JOB_NAME=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-21T21:51:22.1150900Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.1151645Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2025-04-21T21:51:22.1152117Z GITHUB_EVENT_NAME=push
2025-04-21T21:51:22.1152355Z DASHBOARD_TAG=
2025-04-21T21:51:22.1152569Z GITHUB_RUN_ID=14580756816
2025-04-21T21:51:22.1152960Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-12-6=12.3.1.54-1
2025-04-21T21:51:22.1153450Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-12-6
2025-04-21T21:51:22.1154098Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.1154731Z GITHUB_ACTOR=pytorch-bot[bot]
2025-04-21T21:51:22.1155013Z NV_LIBNPP_DEV_VERSION=12.3.1.54-1
2025-04-21T21:51:22.1155280Z PR_NUMBER=
2025-04-21T21:51:22.1155484Z GITHUB_RUN_ATTEMPT=1
2025-04-21T21:51:22.1155713Z VALGRIND=ON
2025-04-21T21:51:22.1155933Z ANACONDA_PYTHON_VERSION=3.10
2025-04-21T21:51:22.1156251Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-04-21T21:51:22.1156579Z TERM=vt100
2025-04-21T21:51:22.1156907Z NV_LIBCUSPARSE_DEV_VERSION=12.5.4.2-1
2025-04-21T21:51:22.1157192Z INSTALLED_VISION=yes
2025-04-21T21:51:22.1157440Z BRANCH=
2025-04-21T21:51:22.1157637Z SCCACHE_REGION=us-east-1
2025-04-21T21:51:22.1157898Z OPENSSL_ROOT_DIR=/opt/openssl
2025-04-21T21:51:22.1158188Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2025-04-21T21:51:22.1158491Z CUDA_PATH=/usr/local/cuda
2025-04-21T21:51:22.1159030Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2025-04-21T21:51:22.1159577Z GITHUB_SERVER_URL=https://github.com
2025-04-21T21:51:22.1159923Z UCC_COMMIT=20eae37090a4ce1b32bcce6144ccad0b49943e0b
2025-04-21T21:51:22.1160247Z REENABLED_ISSUES=
2025-04-21T21:51:22.1160464Z SHLVL=1
2025-04-21T21:51:22.1160655Z MAX_JOBS=14
2025-04-21T21:51:22.1160865Z NV_CUDA_LIB_VERSION=12.6.3-1
2025-04-21T21:51:22.1161124Z NVARCH=x86_64
2025-04-21T21:51:22.1161345Z GITHUB_ACTOR_ID=54816060
2025-04-21T21:51:22.1161671Z GITHUB_WORKFLOW_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-21T21:51:22.1162058Z GITHUB_REF_NAME=ciflow/inductor/148893
2025-04-21T21:51:22.1173915Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2025-04-21T21:51:22.1174507Z GITHUB_JOB=test
2025-04-21T21:51:22.1174765Z NV_LIBNCCL_PACKAGE=libnccl2=2.23.4-1+cuda12.6
2025-04-21T21:51:22.1175196Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2025-04-21T21:51:22.1175571Z NO_TEST_TIMEOUT=False
2025-04-21T21:51:22.1175818Z TD_DISTRIBUTED=False
2025-04-21T21:51:22.1176076Z NV_CUDA_NSIGHT_COMPUTE_VERSION=12.6.3-1
2025-04-21T21:51:22.1176381Z GITHUB_REPOSITORY=pytorch/pytorch
2025-04-21T21:51:22.1176664Z NV_NVPROF_VERSION=12.6.80-1
2025-04-21T21:51:22.1176927Z GITHUB_RETENTION_DAYS=90
2025-04-21T21:51:22.1177175Z OPENSSL_DIR=/opt/openssl
2025-04-21T21:51:22.1177429Z GITHUB_ACTION_REPOSITORY=
2025-04-21T21:51:22.1178171Z PATH=/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-21T21:51:22.1178918Z GITHUB_BASE_REF=
2025-04-21T21:51:22.1179362Z ARTIFACTS_FILE_SUFFIX=test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-21T21:51:22.1179887Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2025-04-21T21:51:22.1180156Z CI=true
2025-04-21T21:51:22.1180370Z NV_LIBNCCL_PACKAGE_VERSION=2.23.4-1
2025-04-21T21:51:22.1180674Z GITHUB_REPOSITORY_OWNER=pytorch
2025-04-21T21:51:22.1180941Z JOB_ID=40900324329
2025-04-21T21:51:22.1181163Z INSTALLED_PROTOBUF=yes
2025-04-21T21:51:22.1181405Z GITHUB_HEAD_REF=
2025-04-21T21:51:22.1181626Z GITHUB_ACTION_REF=
2025-04-21T21:51:22.1181903Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2025-04-21T21:51:22.1182243Z TEST_SHOWLOCALS=False
2025-04-21T21:51:22.1182489Z GITHUB_WORKFLOW=inductor
2025-04-21T21:51:22.1182745Z DEBIAN_FRONTEND=noninteractive
2025-04-21T21:51:22.1183342Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_14b9c959-68ac-41b0-bcbd-2ec1a752dd13
2025-04-21T21:51:22.1183958Z NO_TD=False
2025-04-21T21:51:22.1184187Z SKIP_SCCACHE_INITIALIZATION=1
2025-04-21T21:51:22.1184476Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/
2025-04-21T21:51:22.1184772Z _=/usr/bin/env
2025-04-21T21:51:22.1184993Z + echo 'Testing pytorch'
2025-04-21T21:51:22.1185233Z Testing pytorch
2025-04-21T21:51:22.1185602Z + export LANG=C.UTF-8
2025-04-21T21:51:22.1185831Z + LANG=C.UTF-8
2025-04-21T21:51:22.1186036Z + PR_NUMBER=
2025-04-21T21:51:22.1186271Z + [[ inductor_torchbench == \d\e\f\a\u\l\t ]]
2025-04-21T21:51:22.1186621Z + [[ inductor_torchbench == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-04-21T21:51:22.1186956Z + [[ inductor_torchbench == \s\l\o\w ]]
2025-04-21T21:51:22.1187351Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *slow-gradcheck* ]]
2025-04-21T21:51:22.1187804Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *cuda* ]]
2025-04-21T21:51:22.1188173Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-04-21T21:51:22.1188505Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-04-21T21:51:22.1188910Z + [[ inductor_torchbench == *crossref* ]]
2025-04-21T21:51:22.1189266Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]]
2025-04-21T21:51:22.1189657Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *xpu* ]]
2025-04-21T21:51:22.1190243Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]]
2025-04-21T21:51:22.1190628Z + pip_install --user ninja==1.10.2
2025-04-21T21:51:22.1190995Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T21:51:22.1191463Z + python3 -m pip install --progress-bar off --user ninja==1.10.2
2025-04-21T21:51:22.5054970Z Collecting ninja==1.10.2
2025-04-21T21:51:22.5233653Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-04-21T21:51:22.5357544Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-04-21T21:51:23.1263816Z Installing collected packages: ninja
2025-04-21T21:51:23.1342470Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2025-04-21T21:51:23.1344354Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2025-04-21T21:51:23.1409710Z Successfully installed ninja-1.10.2
2025-04-21T21:51:23.2190381Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-21T21:51:23.2191923Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-21T21:51:23.2192868Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]]
2025-04-21T21:51:23.2193238Z + install_tlparse
2025-04-21T21:51:23.2193550Z + pip_install --user tlparse==0.3.30
2025-04-21T21:51:23.2193942Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T21:51:23.2194444Z + python3 -m pip install --progress-bar off --user tlparse==0.3.30
2025-04-21T21:51:23.6180045Z Collecting tlparse==0.3.30
2025-04-21T21:51:23.6349731Z   Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.9 kB)
2025-04-21T21:51:23.6491119Z Downloading tlparse-0.3.30-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.3 MB)
2025-04-21T21:51:24.2532330Z Installing collected packages: tlparse
2025-04-21T21:51:24.2887239Z Successfully installed tlparse-0.3.30
2025-04-21T21:51:24.3685499Z ++ python -m site --user-base
2025-04-21T21:51:24.3875932Z + PATH=/var/lib/jenkins/.local/bin:/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-04-21T21:51:24.3878050Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *asan* ]]
2025-04-21T21:51:24.3878891Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-debug* ]]
2025-04-21T21:51:24.3879573Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 != *-bazel-* ]]
2025-04-21T21:51:24.3880175Z + echo 'We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass'
2025-04-21T21:51:24.3880884Z We are not in debug mode: linux-focal-cuda12.6-py3.10-gcc9-sm86. Expect the assertion to pass
2025-04-21T21:51:24.3881729Z + cd test
2025-04-21T21:51:24.3882060Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-04-21T21:51:26.1002809Z + [[ inductor_torchbench == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-04-21T21:51:26.1003257Z + [[ inductor_torchbench == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-04-21T21:51:26.1007743Z + DYNAMO_BENCHMARK_FLAGS=()
2025-04-21T21:51:26.1009293Z + [[ inductor_torchbench == *pr_time_benchmarks* ]]
2025-04-21T21:51:26.1009681Z + [[ inductor_torchbench == *dynamo_eager* ]]
2025-04-21T21:51:26.1010016Z + [[ inductor_torchbench == *aot_eager* ]]
2025-04-21T21:51:26.1010341Z + [[ inductor_torchbench == *aot_inductor* ]]
2025-04-21T21:51:26.1011148Z + [[ inductor_torchbench == *max_autotune_inductor* ]]
2025-04-21T21:51:26.1011518Z + [[ inductor_torchbench == *inductor* ]]
2025-04-21T21:51:26.1011834Z + [[ inductor_torchbench != *perf* ]]
2025-04-21T21:51:26.1012164Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor)
2025-04-21T21:51:26.1012482Z + [[ inductor_torchbench == *dynamic* ]]
2025-04-21T21:51:26.1012814Z + [[ inductor_torchbench == *cpu* ]]
2025-04-21T21:51:26.1013117Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-04-21T21:51:26.1043645Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]]
2025-04-21T21:51:26.1044104Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *-bazel-* ]]
2025-04-21T21:51:26.1047668Z + cd test
2025-04-21T21:51:26.1047968Z + python -c 'import torch; print(torch.__config__.show())'
2025-04-21T21:51:27.6185634Z PyTorch built with:
2025-04-21T21:51:27.6185949Z   - GCC 9.4
2025-04-21T21:51:27.6186183Z   - C++ Version: 201703
2025-04-21T21:51:27.6186775Z   - Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2025-04-21T21:51:27.6187484Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-04-21T21:51:27.6187930Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-04-21T21:51:27.6188274Z   - LAPACK is enabled (usually provided by MKL)
2025-04-21T21:51:27.6188607Z   - NNPACK is enabled
2025-04-21T21:51:27.6188881Z   - CPU capability usage: AVX2
2025-04-21T21:51:27.6189161Z   - CUDA Runtime 12.6
2025-04-21T21:51:27.6189514Z   - NVCC architecture flags: -gencode;arch=compute_86,code=sm_86
2025-04-21T21:51:27.6189893Z   - CuDNN 90.5.1
2025-04-21T21:51:27.6190122Z   - Magma 2.6.1
2025-04-21T21:51:27.6194529Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=b68c0effe97351b0c24b2f7aef671559be6aa977, CUDA_VERSION=12.6, CUDNN_VERSION=9.5.1, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.8.0, USE_CUDA=ON, USE_CUDNN=ON, USE_CUSPARSELT=ON, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, 
2025-04-21T21:51:27.6199012Z 
2025-04-21T21:51:28.0245531Z + cd test
2025-04-21T21:51:28.0245911Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-04-21T21:51:29.4199809Z ATen/Parallel:
2025-04-21T21:51:29.4200171Z 	at::get_num_threads() : 8
2025-04-21T21:51:29.4200464Z 	at::get_num_interop_threads() : 16
2025-04-21T21:51:29.4200775Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-04-21T21:51:29.4201062Z 	omp_get_max_threads() : 8
2025-04-21T21:51:29.4201611Z Intel(R) oneAPI Math Kernel Library Version 2021.4-Product Build 20210904 for Intel(R) 64 architecture applications
2025-04-21T21:51:29.4202605Z 	mkl_get_max_threads() : 8
2025-04-21T21:51:29.4202981Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-04-21T21:51:29.4203424Z std::thread::hardware_concurrency() : 16
2025-04-21T21:51:29.4203741Z Environment variables:
2025-04-21T21:51:29.4203995Z 	OMP_NUM_THREADS : [not set]
2025-04-21T21:51:29.4204271Z 	MKL_NUM_THREADS : [not set]
2025-04-21T21:51:29.4204545Z ATen parallel backend: OpenMP
2025-04-21T21:51:29.4204726Z 
2025-04-21T21:51:29.7215304Z + [[ inductor_torchbench == *numpy_2* ]]
2025-04-21T21:51:29.7215814Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *aarch64* ]]
2025-04-21T21:51:29.7216641Z + [[ inductor_torchbench == *backward* ]]
2025-04-21T21:51:29.7216969Z + [[ inductor_torchbench == *xla* ]]
2025-04-21T21:51:29.7217277Z + [[ inductor_torchbench == *executorch* ]]
2025-04-21T21:51:29.7217625Z + [[ inductor_torchbench == \j\i\t\_\l\e\g\a\c\y ]]
2025-04-21T21:51:29.7218027Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *libtorch* ]]
2025-04-21T21:51:29.7218426Z + [[ inductor_torchbench == distributed ]]
2025-04-21T21:51:29.7218775Z + [[ inductor_torchbench == *operator_benchmark* ]]
2025-04-21T21:51:29.7219160Z + [[ inductor_torchbench == *inductor_distributed* ]]
2025-04-21T21:51:29.7219538Z + [[ inductor_torchbench == *inductor-halide* ]]
2025-04-21T21:51:29.7219913Z + [[ inductor_torchbench == *inductor-triton-cpu* ]]
2025-04-21T21:51:29.7220317Z + [[ inductor_torchbench == *inductor-micro-benchmark* ]]
2025-04-21T21:51:29.7220700Z + [[ inductor_torchbench == *huggingface* ]]
2025-04-21T21:51:29.7221018Z + [[ inductor_torchbench == *timm* ]]
2025-04-21T21:51:29.7221337Z + [[ inductor_torchbench == cachebench ]]
2025-04-21T21:51:29.7222003Z + [[ inductor_torchbench == verify_cachebench ]]
2025-04-21T21:51:29.7222375Z + [[ inductor_torchbench == *torchbench* ]]
2025-04-21T21:51:29.7222696Z + [[ inductor_torchbench == *cpu* ]]
2025-04-21T21:51:29.7222991Z + install_torchaudio cuda
2025-04-21T21:51:29.7223247Z + local commit
2025-04-21T21:51:29.7223470Z ++ get_pinned_commit audio
2025-04-21T21:51:29.7223749Z ++ cat .github/ci_commit_pins/audio.txt
2025-04-21T21:51:29.7241741Z + commit=bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:29.7242119Z + [[ cuda == \c\u\d\a ]]
2025-04-21T21:51:29.7242391Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2025-04-21T21:51:29.7243002Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:29.7243695Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T21:51:29.7244473Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:30.0691211Z Collecting git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:30.0694758Z   Cloning https://github.com/pytorch/audio.git (to revision bccaa454a54c3c648697cc2f46a4fb0500b1f01b) to /tmp/pip-req-build-8wwhla43
2025-04-21T21:51:30.0725814Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/audio.git /tmp/pip-req-build-8wwhla43
2025-04-21T21:51:30.8910000Z   Running command git rev-parse -q --verify 'sha^bccaa454a54c3c648697cc2f46a4fb0500b1f01b'
2025-04-21T21:51:30.8936035Z   Running command git fetch -q https://github.com/pytorch/audio.git bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:31.3103306Z   Running command git checkout -q bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:31.4606844Z   Resolved https://github.com/pytorch/audio.git to commit bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T21:51:31.4608115Z   Running command git submodule update --init --recursive -q
2025-04-21T21:51:34.1111816Z   Preparing metadata (setup.py) ... [?25l- done
2025-04-21T21:51:34.1142028Z [?25hRequirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchaudio==2.6.0a0+bccaa45) (2.8.0a0+gitb68c0ef)
2025-04-21T21:51:34.1165969Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (3.16.1)
2025-04-21T21:51:34.1169735Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (4.13.2)
2025-04-21T21:51:34.1173698Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (1.13.3)
2025-04-21T21:51:34.1176559Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (2.8.8)
2025-04-21T21:51:34.1179770Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (3.1.6)
2025-04-21T21:51:34.1182351Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchaudio==2.6.0a0+bccaa45) (2024.10.0)
2025-04-21T21:51:34.1196832Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchaudio==2.6.0a0+bccaa45) (1.3.0)
2025-04-21T21:51:34.1665448Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchaudio==2.6.0a0+bccaa45) (3.0.2)
2025-04-21T21:51:34.1729625Z Building wheels for collected packages: torchaudio
2025-04-21T21:54:02.8539254Z   Building wheel for torchaudio (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - done
2025-04-21T21:54:02.8575729Z [?25h  Created wheel for torchaudio: filename=torchaudio-2.6.0a0+bccaa45-cp310-cp310-linux_x86_64.whl size=2396804 sha256=aefebbfb90d41a7cac1dd6d5f72b53d5c9f834ef6bd6c5a94e488879e401a33a
2025-04-21T21:54:02.8577015Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/a2/00/0f/924e514cbe6637d9ca623106f058b6c50bd5c3ca92b0f3facc
2025-04-21T21:54:02.8606491Z Successfully built torchaudio
2025-04-21T21:54:03.3831108Z Installing collected packages: torchaudio
2025-04-21T21:54:03.5998295Z Successfully installed torchaudio-2.6.0a0+bccaa45
2025-04-21T21:54:03.7852053Z + install_torchvision
2025-04-21T21:54:03.7852477Z + local orig_preload
2025-04-21T21:54:03.7852797Z + local commit
2025-04-21T21:54:03.7857035Z ++ get_pinned_commit vision
2025-04-21T21:54:03.7857445Z ++ cat .github/ci_commit_pins/vision.txt
2025-04-21T21:54:03.7880997Z + commit=d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:03.7881521Z + orig_preload=
2025-04-21T21:54:03.7881840Z + '[' -n '' ']'
2025-04-21T21:54:03.7882493Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:03.7883182Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T21:54:03.7884055Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:04.1397065Z Collecting git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:04.1400725Z   Cloning https://github.com/pytorch/vision.git (to revision d23a6e1664d20707c11781299611436e1f0c104f) to /tmp/pip-req-build-qczwj4v3
2025-04-21T21:54:04.1441259Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/vision.git /tmp/pip-req-build-qczwj4v3
2025-04-21T21:54:05.6696855Z   Running command git rev-parse -q --verify 'sha^d23a6e1664d20707c11781299611436e1f0c104f'
2025-04-21T21:54:05.6726769Z   Running command git fetch -q https://github.com/pytorch/vision.git d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:07.1687136Z   Running command git checkout -q d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:07.5507232Z   Resolved https://github.com/pytorch/vision.git to commit d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T21:54:10.4228249Z   Preparing metadata (setup.py) ... [?25l- \ done
2025-04-21T21:54:10.4265434Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (1.22.4)
2025-04-21T21:54:10.4268904Z Requirement already satisfied: torch in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (2.8.0a0+gitb68c0ef)
2025-04-21T21:54:10.4272761Z Requirement already satisfied: pillow!=8.3.*,>=5.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchvision==0.19.0a0+d23a6e1) (11.0.0)
2025-04-21T21:54:10.4341467Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.16.1)
2025-04-21T21:54:10.4345376Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (4.13.2)
2025-04-21T21:54:10.4349521Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (1.13.3)
2025-04-21T21:54:10.4352519Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2.8.8)
2025-04-21T21:54:10.4356215Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (3.1.6)
2025-04-21T21:54:10.4359442Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch->torchvision==0.19.0a0+d23a6e1) (2024.10.0)
2025-04-21T21:54:10.4374023Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch->torchvision==0.19.0a0+d23a6e1) (1.3.0)
2025-04-21T21:54:10.4852822Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch->torchvision==0.19.0a0+d23a6e1) (3.0.2)
2025-04-21T21:54:10.4919386Z Building wheels for collected packages: torchvision
2025-04-21T21:55:27.1177258Z   Building wheel for torchvision (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / done
2025-04-21T21:55:27.1211028Z [?25h  Created wheel for torchvision: filename=torchvision-0.19.0a0+d23a6e1-cp310-cp310-linux_x86_64.whl size=2077107 sha256=daf289772afac86932aa91f0505af7832c7a0420d72f8862b0cc89cbcfb5af8c
2025-04-21T21:55:27.1212222Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/0e/56/35/02931e71eb23fd2b85591c7ec05b733ca7c8b328a2fd151f96
2025-04-21T21:55:27.1248302Z Successfully built torchvision
2025-04-21T21:55:27.6488151Z Installing collected packages: torchvision
2025-04-21T21:55:28.0709524Z Successfully installed torchvision-0.19.0a0+d23a6e1
2025-04-21T21:55:28.2166165Z + '[' -n '' ']'
2025-04-21T21:55:28.2166464Z + TORCH_CUDA_ARCH_LIST='8.0;8.6'
2025-04-21T21:55:28.2166840Z + pip_install git+https://github.com/pytorch/ao.git
2025-04-21T21:55:28.2167286Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T21:55:28.2167843Z + python3 -m pip install --progress-bar off git+https://github.com/pytorch/ao.git
2025-04-21T21:55:28.5681561Z Collecting git+https://github.com/pytorch/ao.git
2025-04-21T21:55:28.5684765Z   Cloning https://github.com/pytorch/ao.git to /tmp/pip-req-build-vb684j5l
2025-04-21T21:55:28.5714560Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/ao.git /tmp/pip-req-build-vb684j5l
2025-04-21T21:55:29.6663536Z   Resolved https://github.com/pytorch/ao.git to commit 4805efd03b1aea9b5b9fe84df5c5a7171b5a7f2c
2025-04-21T21:55:29.6665172Z   Running command git submodule update --init --recursive -q
2025-04-21T21:55:37.0933696Z   Preparing metadata (setup.py) ... [?25l- done
2025-04-21T21:55:37.0975619Z [?25hBuilding wheels for collected packages: torchao
2025-04-21T22:00:54.6326766Z   Building wheel for torchao (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2025-04-21T22:00:54.6371909Z [?25h  Created wheel for torchao: filename=torchao-0.11.0+git4805efd0-cp39-abi3-linux_x86_64.whl size=5609505 sha256=97e6efa0f065f1242835fc3baf2a5e2142a8531d5b59b3385542a9c41c6f9705
2025-04-21T22:00:54.6373209Z   Stored in directory: /tmp/pip-ephem-wheel-cache-xns32xmy/wheels/c0/df/d8/b1e2f5f1ea3bf141268a83a667137647b234f04c636e1e7a5e
2025-04-21T22:00:54.6416210Z Successfully built torchao
2025-04-21T22:00:55.2187729Z Installing collected packages: torchao
2025-04-21T22:00:56.0076614Z Successfully installed torchao-0.11.0+git4805efd0
2025-04-21T22:00:56.4545859Z + id=1
2025-04-21T22:00:56.4548152Z + pip_install opencv-python==4.8.0.74
2025-04-21T22:00:56.4581935Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T22:00:56.4582460Z + python3 -m pip install --progress-bar off opencv-python==4.8.0.74
2025-04-21T22:00:57.0233017Z Collecting opencv-python==4.8.0.74
2025-04-21T22:00:57.0416733Z   Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)
2025-04-21T22:00:57.0518198Z Requirement already satisfied: numpy>=1.21.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from opencv-python==4.8.0.74) (1.22.4)
2025-04-21T22:00:57.0585010Z Downloading opencv_python-4.8.0.74-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (61.7 MB)
2025-04-21T22:00:58.0848117Z Installing collected packages: opencv-python
2025-04-21T22:00:58.9552766Z Successfully installed opencv-python-4.8.0.74
2025-04-21T22:00:59.0491662Z + [[ inductor_torchbench == *inductor_torchbench_smoketest_perf* ]]
2025-04-21T22:00:59.0495142Z + [[ inductor_torchbench == *inductor_torchbench_cpu_smoketest_perf* ]]
2025-04-21T22:00:59.0495781Z + [[ inductor_torchbench == *torchbench_gcp_smoketest* ]]
2025-04-21T22:00:59.0496263Z + checkout_install_torchbench
2025-04-21T22:00:59.0496596Z + local commit
2025-04-21T22:00:59.0505119Z ++ get_pinned_commit torchbench
2025-04-21T22:00:59.0505605Z ++ cat .github/ci_commit_pins/torchbench.txt
2025-04-21T22:00:59.0593216Z + commit=373ffb19dc470f4423a3176a4133f8f4b3cdb5bd
2025-04-21T22:00:59.0593848Z + git clone https://github.com/pytorch/benchmark torchbench
2025-04-21T22:00:59.0638310Z Cloning into 'torchbench'...
2025-04-21T22:00:59.2428030Z remote: Enumerating objects: 35746, done.
2025-04-21T22:00:59.2428501Z remote: Counting objects:   7% (1/14)
2025-04-21T22:00:59.2428876Z remote: Counting objects:  14% (2/14)
2025-04-21T22:00:59.2429247Z remote: Counting objects:  21% (3/14)
2025-04-21T22:00:59.2429651Z remote: Counting objects:  28% (4/14)
2025-04-21T22:00:59.2430012Z remote: Counting objects:  35% (5/14)
2025-04-21T22:00:59.2430368Z remote: Counting objects:  42% (6/14)
2025-04-21T22:00:59.2430726Z remote: Counting objects:  50% (7/14)
2025-04-21T22:00:59.2431089Z remote: Counting objects:  57% (8/14)
2025-04-21T22:00:59.2431444Z remote: Counting objects:  64% (9/14)
2025-04-21T22:00:59.2431816Z remote: Counting objects:  71% (10/14)
2025-04-21T22:00:59.2432187Z remote: Counting objects:  78% (11/14)
2025-04-21T22:00:59.2432562Z remote: Counting objects:  85% (12/14)
2025-04-21T22:00:59.2432915Z remote: Counting objects:  92% (13/14)
2025-04-21T22:00:59.2433290Z remote: Counting objects: 100% (14/14)
2025-04-21T22:00:59.2433737Z remote: Counting objects: 100% (14/14), done.
2025-04-21T22:00:59.2434127Z remote: Compressing objects:   9% (1/11)
2025-04-21T22:00:59.2441121Z remote: Compressing objects:  18% (2/11)
2025-04-21T22:00:59.2441631Z remote: Compressing objects:  27% (3/11)
2025-04-21T22:00:59.2442453Z remote: Compressing objects:  36% (4/11)
2025-04-21T22:00:59.2442819Z remote: Compressing objects:  45% (5/11)
2025-04-21T22:00:59.2443190Z remote: Compressing objects:  54% (6/11)
2025-04-21T22:00:59.2443561Z remote: Compressing objects:  63% (7/11)
2025-04-21T22:00:59.2443927Z remote: Compressing objects:  72% (8/11)
2025-04-21T22:00:59.2444303Z remote: Compressing objects:  81% (9/11)
2025-04-21T22:00:59.2444683Z remote: Compressing objects:  90% (10/11)
2025-04-21T22:00:59.2445059Z remote: Compressing objects: 100% (11/11)
2025-04-21T22:00:59.2445469Z remote: Compressing objects: 100% (11/11), done.
2025-04-21T22:00:59.2530551Z Receiving objects:   0% (1/35746)
2025-04-21T22:00:59.2587961Z Receiving objects:   1% (358/35746)
2025-04-21T22:00:59.2662244Z Receiving objects:   2% (715/35746)
2025-04-21T22:00:59.2708556Z Receiving objects:   3% (1073/35746)
2025-04-21T22:00:59.2750350Z Receiving objects:   4% (1430/35746)
2025-04-21T22:00:59.2788537Z Receiving objects:   5% (1788/35746)
2025-04-21T22:00:59.2825254Z Receiving objects:   6% (2145/35746)
2025-04-21T22:00:59.2863110Z Receiving objects:   7% (2503/35746)
2025-04-21T22:00:59.2909846Z Receiving objects:   8% (2860/35746)
2025-04-21T22:00:59.2946234Z Receiving objects:   9% (3218/35746)
2025-04-21T22:00:59.2983556Z Receiving objects:  10% (3575/35746)
2025-04-21T22:00:59.3062482Z Receiving objects:  11% (3933/35746)
2025-04-21T22:00:59.3116749Z Receiving objects:  12% (4290/35746)
2025-04-21T22:00:59.3196244Z Receiving objects:  13% (4647/35746)
2025-04-21T22:00:59.3383921Z Receiving objects:  14% (5005/35746)
2025-04-21T22:00:59.3529914Z Receiving objects:  15% (5362/35746)
2025-04-21T22:00:59.3611352Z Receiving objects:  16% (5720/35746)
2025-04-21T22:00:59.3658108Z Receiving objects:  17% (6077/35746)
2025-04-21T22:00:59.3732977Z Receiving objects:  18% (6435/35746)
2025-04-21T22:00:59.3790260Z Receiving objects:  19% (6792/35746)
2025-04-21T22:00:59.3826200Z Receiving objects:  20% (7150/35746)
2025-04-21T22:00:59.3897963Z Receiving objects:  21% (7507/35746)
2025-04-21T22:00:59.3962837Z Receiving objects:  22% (7865/35746)
2025-04-21T22:00:59.4047323Z Receiving objects:  23% (8222/35746)
2025-04-21T22:00:59.4060242Z Receiving objects:  24% (8580/35746)
2025-04-21T22:00:59.4079588Z Receiving objects:  25% (8937/35746)
2025-04-21T22:00:59.4095356Z Receiving objects:  26% (9294/35746)
2025-04-21T22:00:59.4113096Z Receiving objects:  27% (9652/35746)
2025-04-21T22:00:59.4138727Z Receiving objects:  28% (10009/35746)
2025-04-21T22:00:59.4160716Z Receiving objects:  29% (10367/35746)
2025-04-21T22:00:59.4175564Z Receiving objects:  30% (10724/35746)
2025-04-21T22:00:59.4218389Z Receiving objects:  31% (11082/35746)
2025-04-21T22:00:59.4233444Z Receiving objects:  32% (11439/35746)
2025-04-21T22:00:59.4263398Z Receiving objects:  33% (11797/35746)
2025-04-21T22:00:59.4282693Z Receiving objects:  34% (12154/35746)
2025-04-21T22:00:59.4314349Z Receiving objects:  35% (12512/35746)
2025-04-21T22:00:59.4332013Z Receiving objects:  36% (12869/35746)
2025-04-21T22:00:59.4371174Z Receiving objects:  37% (13227/35746)
2025-04-21T22:01:00.1685743Z Receiving objects:  38% (13584/35746)
2025-04-21T22:01:00.2464461Z Receiving objects:  39% (13941/35746), 35.05 MiB | 70.09 MiB/s
2025-04-21T22:01:00.9610925Z Receiving objects:  39% (13947/35746), 73.73 MiB | 73.72 MiB/s
2025-04-21T22:01:01.0427856Z Receiving objects:  40% (14299/35746), 112.36 MiB | 74.90 MiB/s
2025-04-21T22:01:01.1246738Z Receiving objects:  41% (14656/35746), 112.36 MiB | 74.90 MiB/s
2025-04-21T22:01:01.2060408Z Receiving objects:  42% (15014/35746), 112.36 MiB | 74.90 MiB/s
2025-04-21T22:01:01.2463987Z Receiving objects:  43% (15371/35746), 112.36 MiB | 74.90 MiB/s
2025-04-21T22:01:01.2875190Z Receiving objects:  43% (15549/35746), 112.36 MiB | 74.90 MiB/s
2025-04-21T22:01:01.3700636Z Receiving objects:  44% (15729/35746), 150.23 MiB | 75.11 MiB/s
2025-04-21T22:01:01.4514090Z Receiving objects:  45% (16086/35746), 150.23 MiB | 75.11 MiB/s
2025-04-21T22:01:01.6935148Z Receiving objects:  46% (16444/35746), 150.23 MiB | 75.11 MiB/s
2025-04-21T22:01:01.7688321Z Receiving objects:  47% (16801/35746), 150.23 MiB | 75.11 MiB/s
2025-04-21T22:01:01.8381827Z Receiving objects:  48% (17159/35746), 186.07 MiB | 74.43 MiB/s
2025-04-21T22:01:02.2465860Z Receiving objects:  49% (17516/35746), 186.07 MiB | 74.43 MiB/s
2025-04-21T22:01:02.2591579Z Receiving objects:  49% (17865/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2855414Z Receiving objects:  50% (17873/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2865737Z Receiving objects:  51% (18231/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2886391Z Receiving objects:  52% (18588/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2900586Z Receiving objects:  53% (18946/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2918086Z Receiving objects:  54% (19303/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2926466Z Receiving objects:  55% (19661/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2932932Z Receiving objects:  56% (20018/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2946936Z Receiving objects:  57% (20376/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2957042Z Receiving objects:  58% (20733/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2965487Z Receiving objects:  59% (21091/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2971691Z Receiving objects:  60% (21448/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2976937Z Receiving objects:  61% (21806/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.2992420Z Receiving objects:  62% (22163/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3191251Z Receiving objects:  63% (22520/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3197319Z Receiving objects:  64% (22878/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3205825Z Receiving objects:  65% (23235/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3212592Z Receiving objects:  66% (23593/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3226140Z Receiving objects:  67% (23950/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3232802Z Receiving objects:  68% (24308/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3250592Z Receiving objects:  69% (24665/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3256842Z Receiving objects:  70% (25023/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3268498Z Receiving objects:  71% (25380/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3404613Z Receiving objects:  72% (25738/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3411906Z Receiving objects:  73% (26095/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3421155Z Receiving objects:  74% (26453/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3430622Z Receiving objects:  75% (26810/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3677263Z Receiving objects:  76% (27167/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.3694748Z Receiving objects:  77% (27525/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4090315Z Receiving objects:  78% (27882/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4102162Z Receiving objects:  79% (28240/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4119490Z Receiving objects:  80% (28597/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4131893Z Receiving objects:  81% (28955/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4142334Z Receiving objects:  82% (29312/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4162928Z Receiving objects:  83% (29670/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4171349Z Receiving objects:  84% (30027/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4180638Z Receiving objects:  85% (30385/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4393882Z Receiving objects:  86% (30742/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4431258Z Receiving objects:  87% (31100/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4444990Z Receiving objects:  88% (31457/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4457076Z Receiving objects:  89% (31814/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4514635Z Receiving objects:  90% (32172/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4686700Z Receiving objects:  91% (32529/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4694885Z Receiving objects:  92% (32887/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4704319Z Receiving objects:  93% (33244/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4714322Z Receiving objects:  94% (33602/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4721416Z Receiving objects:  95% (33959/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4838572Z Receiving objects:  96% (34317/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4856832Z Receiving objects:  97% (34674/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4870501Z Receiving objects:  98% (35032/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4886782Z Receiving objects:  99% (35389/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4887570Z remote: Total 35746 (delta 3), reused 11 (delta 3), pack-reused 35732 (from 1)
2025-04-21T22:01:02.4906882Z Receiving objects: 100% (35746/35746), 221.28 MiB | 73.76 MiB/s
2025-04-21T22:01:02.4907365Z Receiving objects: 100% (35746/35746), 234.18 MiB | 72.19 MiB/s, done.
2025-04-21T22:01:02.4946600Z Resolving deltas:   0% (0/19933)
2025-04-21T22:01:02.4959189Z Resolving deltas:   1% (200/19933)
2025-04-21T22:01:02.4971303Z Resolving deltas:   2% (408/19933)
2025-04-21T22:01:02.4985709Z Resolving deltas:   3% (626/19933)
2025-04-21T22:01:02.4999298Z Resolving deltas:   4% (811/19933)
2025-04-21T22:01:02.5015962Z Resolving deltas:   5% (1010/19933)
2025-04-21T22:01:02.5031792Z Resolving deltas:   6% (1208/19933)
2025-04-21T22:01:02.5044987Z Resolving deltas:   7% (1411/19933)
2025-04-21T22:01:02.5065810Z Resolving deltas:   8% (1595/19933)
2025-04-21T22:01:02.5079110Z Resolving deltas:   9% (1794/19933)
2025-04-21T22:01:02.5091383Z Resolving deltas:  10% (2001/19933)
2025-04-21T22:01:02.5109483Z Resolving deltas:  11% (2195/19933)
2025-04-21T22:01:02.5120794Z Resolving deltas:  12% (2405/19933)
2025-04-21T22:01:02.5143084Z Resolving deltas:  13% (2593/19933)
2025-04-21T22:01:02.5154498Z Resolving deltas:  14% (2803/19933)
2025-04-21T22:01:02.5203485Z Resolving deltas:  15% (3017/19933)
2025-04-21T22:01:02.5232973Z Resolving deltas:  16% (3192/19933)
2025-04-21T22:01:02.5268109Z Resolving deltas:  17% (3393/19933)
2025-04-21T22:01:02.5340051Z Resolving deltas:  18% (3593/19933)
2025-04-21T22:01:02.5385410Z Resolving deltas:  19% (3803/19933)
2025-04-21T22:01:02.5431783Z Resolving deltas:  20% (3987/19933)
2025-04-21T22:01:02.5463341Z Resolving deltas:  21% (4192/19933)
2025-04-21T22:01:02.5497154Z Resolving deltas:  22% (4386/19933)
2025-04-21T22:01:02.5537711Z Resolving deltas:  23% (4626/19933)
2025-04-21T22:01:02.5583114Z Resolving deltas:  24% (4824/19933)
2025-04-21T22:01:02.5633617Z Resolving deltas:  25% (4984/19933)
2025-04-21T22:01:02.5693605Z Resolving deltas:  26% (5238/19933)
2025-04-21T22:01:02.5715822Z Resolving deltas:  27% (5410/19933)
2025-04-21T22:01:02.5744784Z Resolving deltas:  29% (5824/19933)
2025-04-21T22:01:02.5767890Z Resolving deltas:  30% (6003/19933)
2025-04-21T22:01:02.5803367Z Resolving deltas:  31% (6180/19933)
2025-04-21T22:01:02.5828375Z Resolving deltas:  32% (6379/19933)
2025-04-21T22:01:02.5873698Z Resolving deltas:  33% (6601/19933)
2025-04-21T22:01:02.5887114Z Resolving deltas:  35% (7043/19933)
2025-04-21T22:01:02.5969844Z Resolving deltas:  36% (7176/19933)
2025-04-21T22:01:02.6001475Z Resolving deltas:  37% (7381/19933)
2025-04-21T22:01:02.6021407Z Resolving deltas:  38% (7576/19933)
2025-04-21T22:01:02.6042125Z Resolving deltas:  39% (7784/19933)
2025-04-21T22:01:02.6058771Z Resolving deltas:  40% (7982/19933)
2025-04-21T22:01:02.6075689Z Resolving deltas:  41% (8175/19933)
2025-04-21T22:01:02.6087796Z Resolving deltas:  42% (8388/19933)
2025-04-21T22:01:02.6103825Z Resolving deltas:  43% (8573/19933)
2025-04-21T22:01:02.6120240Z Resolving deltas:  44% (8773/19933)
2025-04-21T22:01:02.6130923Z Resolving deltas:  45% (8973/19933)
2025-04-21T22:01:02.6144556Z Resolving deltas:  46% (9174/19933)
2025-04-21T22:01:02.6157713Z Resolving deltas:  47% (9371/19933)
2025-04-21T22:01:02.6169169Z Resolving deltas:  48% (9595/19933)
2025-04-21T22:01:02.6179484Z Resolving deltas:  49% (9768/19933)
2025-04-21T22:01:02.6192340Z Resolving deltas:  50% (10013/19933)
2025-04-21T22:01:02.6221936Z Resolving deltas:  51% (10173/19933)
2025-04-21T22:01:02.6224493Z Resolving deltas:  52% (10535/19933)
2025-04-21T22:01:02.6240035Z Resolving deltas:  53% (10581/19933)
2025-04-21T22:01:02.6249929Z Resolving deltas:  54% (10784/19933)
2025-04-21T22:01:02.6259457Z Resolving deltas:  55% (10969/19933)
2025-04-21T22:01:02.6269719Z Resolving deltas:  56% (11168/19933)
2025-04-21T22:01:02.6283355Z Resolving deltas:  57% (11377/19933)
2025-04-21T22:01:02.6302911Z Resolving deltas:  58% (11588/19933)
2025-04-21T22:01:02.6316844Z Resolving deltas:  59% (11811/19933)
2025-04-21T22:01:02.6329973Z Resolving deltas:  60% (12093/19933)
2025-04-21T22:01:02.6332620Z Resolving deltas:  61% (12354/19933)
2025-04-21T22:01:02.6340646Z Resolving deltas:  62% (12408/19933)
2025-04-21T22:01:02.6351080Z Resolving deltas:  63% (12575/19933)
2025-04-21T22:01:02.6363010Z Resolving deltas:  64% (12763/19933)
2025-04-21T22:01:02.6372769Z Resolving deltas:  65% (12975/19933)
2025-04-21T22:01:02.6387474Z Resolving deltas:  66% (13161/19933)
2025-04-21T22:01:02.6406446Z Resolving deltas:  67% (13356/19933)
2025-04-21T22:01:02.6418145Z Resolving deltas:  68% (13579/19933)
2025-04-21T22:01:02.6428863Z Resolving deltas:  69% (13771/19933)
2025-04-21T22:01:02.6440315Z Resolving deltas:  70% (13964/19933)
2025-04-21T22:01:02.6449122Z Resolving deltas:  71% (14162/19933)
2025-04-21T22:01:02.6458163Z Resolving deltas:  72% (14353/19933)
2025-04-21T22:01:02.6472969Z Resolving deltas:  73% (14559/19933)
2025-04-21T22:01:02.6484602Z Resolving deltas:  74% (14787/19933)
2025-04-21T22:01:02.6491437Z Resolving deltas:  75% (15001/19933)
2025-04-21T22:01:02.6508809Z Resolving deltas:  76% (15197/19933)
2025-04-21T22:01:02.6513889Z Resolving deltas:  77% (15516/19933)
2025-04-21T22:01:02.6533700Z Resolving deltas:  78% (15616/19933)
2025-04-21T22:01:02.6540080Z Resolving deltas:  79% (15878/19933)
2025-04-21T22:01:02.6557787Z Resolving deltas:  80% (15998/19933)
2025-04-21T22:01:02.6563745Z Resolving deltas:  81% (16204/19933)
2025-04-21T22:01:02.6571648Z Resolving deltas:  82% (16366/19933)
2025-04-21T22:01:02.6583004Z Resolving deltas:  83% (16546/19933)
2025-04-21T22:01:02.6599173Z Resolving deltas:  84% (16746/19933)
2025-04-21T22:01:02.6609778Z Resolving deltas:  85% (16968/19933)
2025-04-21T22:01:02.6618359Z Resolving deltas:  86% (17190/19933)
2025-04-21T22:01:02.6641048Z Resolving deltas:  87% (17342/19933)
2025-04-21T22:01:02.6650695Z Resolving deltas:  88% (17556/19933)
2025-04-21T22:01:02.6665527Z Resolving deltas:  89% (17752/19933)
2025-04-21T22:01:02.6677411Z Resolving deltas:  90% (17948/19933)
2025-04-21T22:01:02.6687314Z Resolving deltas:  91% (18144/19933)
2025-04-21T22:01:02.6698086Z Resolving deltas:  92% (18342/19933)
2025-04-21T22:01:02.6710185Z Resolving deltas:  93% (18543/19933)
2025-04-21T22:01:02.6720288Z Resolving deltas:  94% (18750/19933)
2025-04-21T22:01:02.6733300Z Resolving deltas:  95% (18938/19933)
2025-04-21T22:01:02.6748502Z Resolving deltas:  96% (19136/19933)
2025-04-21T22:01:02.6758012Z Resolving deltas:  97% (19392/19933)
2025-04-21T22:01:02.6767916Z Resolving deltas:  98% (19541/19933)
2025-04-21T22:01:02.6777855Z Resolving deltas:  99% (19740/19933)
2025-04-21T22:01:02.6778143Z Resolving deltas: 100% (19933/19933)
2025-04-21T22:01:02.6778454Z Resolving deltas: 100% (19933/19933), done.
2025-04-21T22:01:03.6943735Z + pushd torchbench
2025-04-21T22:01:03.6944022Z ~/workspace/torchbench ~/workspace
2025-04-21T22:01:03.6944379Z + git checkout 373ffb19dc470f4423a3176a4133f8f4b3cdb5bd
2025-04-21T22:01:03.7405850Z Note: switching to '373ffb19dc470f4423a3176a4133f8f4b3cdb5bd'.
2025-04-21T22:01:03.7406430Z 
2025-04-21T22:01:03.7406650Z You are in 'detached HEAD' state. You can look around, make experimental
2025-04-21T22:01:03.7407164Z changes and commit them, and you can discard any commits you make in this
2025-04-21T22:01:03.7407681Z state without impacting any branches by switching back to a branch.
2025-04-21T22:01:03.7407991Z 
2025-04-21T22:01:03.7408188Z If you want to create a new branch to retain commits you create, you may
2025-04-21T22:01:03.7408669Z do so (now or later) by using -c with the switch command. Example:
2025-04-21T22:01:03.7408943Z 
2025-04-21T22:01:03.7409053Z   git switch -c <new-branch-name>
2025-04-21T22:01:03.7409253Z 
2025-04-21T22:01:03.7409507Z Or undo this operation with:
2025-04-21T22:01:03.7409685Z 
2025-04-21T22:01:03.7409779Z   git switch -
2025-04-21T22:01:03.7409913Z 
2025-04-21T22:01:03.7410148Z Turn off this advice by setting config variable advice.detachedHead to false
2025-04-21T22:01:03.7410485Z 
2025-04-21T22:01:03.7410692Z HEAD is now at 373ffb19 Copy model before benchmark warmup runs (#145858)
2025-04-21T22:01:03.7416520Z + '[' '' ']'
2025-04-21T22:01:03.7416805Z + python install.py --continue_on_fail
2025-04-21T22:01:10.9743434Z checking packages numpy, torch, torchvision, torchaudio are installed, generating constaints...OK
2025-04-21T22:01:45.8108406Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/BERT_pytorch...OK
2025-04-21T22:02:00.8556320Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Background_Matting...OK
2025-04-21T22:02:16.0985848Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/LearningToPaint...OK
2025-04-21T22:02:30.9746249Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/Super_SloMo...OK
2025-04-21T22:02:44.1427876Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/alexnet...OK
2025-04-21T22:03:05.2919357Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_edgecnn...OK
2025-04-21T22:03:21.3020423Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gcn...OK
2025-04-21T22:03:37.3372068Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_gin...OK
2025-04-21T22:03:53.3266478Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/basic_gnn_sage...OK
2025-04-21T22:03:53.3269527Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/cm3leon_generate...SKIP - No install.py is found
2025-04-21T22:04:07.5784443Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dcgan...OK
2025-04-21T22:04:23.5233812Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/demucs...OK
2025-04-21T22:04:36.7088473Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/densenet121...OK
2025-04-21T22:05:48.1369036Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_c4...OK
2025-04-21T22:06:11.9052567Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_dc5...OK
2025-04-21T22:06:34.0012752Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_101_fpn...OK
2025-04-21T22:06:55.7047371Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_c4...OK
2025-04-21T22:07:19.3189838Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_dc5...OK
2025-04-21T22:07:41.1976717Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fasterrcnn_r_50_fpn...OK
2025-04-21T22:08:02.4330644Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_fcos_r_50_fpn...OK
2025-04-21T22:08:24.2029971Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn...OK
2025-04-21T22:08:46.2114175Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_c4...OK
2025-04-21T22:09:08.1892646Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_101_fpn...OK
2025-04-21T22:09:29.9536699Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_c4...OK
2025-04-21T22:09:51.8704031Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/detectron2_maskrcnn_r_50_fpn...OK
2025-04-21T22:10:06.4467320Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/dlrm...OK
2025-04-21T22:10:42.1641412Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_det_predictor...OK
2025-04-21T22:11:02.5089268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/doctr_reco_predictor...OK
2025-04-21T22:11:21.1899792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/drq...OK
2025-04-21T22:11:45.4994873Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/fastNLP_Bert...OK
2025-04-21T22:12:00.1964635Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_dp_cifar10...OK
2025-04-21T22:12:15.2030831Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/functorch_maml_omniglot...OK
2025-04-21T22:12:36.8866315Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Albert...OK
2025-04-21T22:12:58.8611515Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bart...OK
2025-04-21T22:13:19.3202787Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert...OK
2025-04-21T22:13:41.9146153Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Bert_large...OK
2025-04-21T22:14:02.7868693Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_BigBird...OK
2025-04-21T22:14:22.8176011Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_DistilBert...OK
2025-04-21T22:14:44.2622806Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2...OK
2025-04-21T22:15:15.1285024Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_GPT2_large...OK
2025-04-21T22:15:36.3593792Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Longformer...OK
2025-04-21T22:15:55.4053923Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Reformer...OK
2025-04-21T22:16:20.0564739Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Roberta_base...OK
2025-04-21T22:16:40.2987492Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5...OK
2025-04-21T22:17:02.1503279Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_base...OK
2025-04-21T22:17:02.1507410Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_generate...SKIP - No install.py is found
2025-04-21T22:17:30.0952816Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_T5_large...OK
2025-04-21T22:17:47.8942382Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_Whisper...OK
2025-04-21T22:17:47.8945568Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_clip...SKIP - No install.py is found
2025-04-21T22:18:10.7149721Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/hf_distil_whisper...OK
2025-04-21T22:18:25.5482349Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/lennard_jones...OK
2025-04-21T22:18:40.3907356Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama...OK
2025-04-21T22:19:38.8580869Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h...OK
2025-04-21T22:21:20.1705507Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava...OK
2025-04-21T22:21:33.7279132Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml...OK
2025-04-21T22:21:48.8356449Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/maml_omniglot...OK
2025-04-21T22:21:48.8358877Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/microbench_unbacked_tolist_sum...SKIP - No install.py is found
2025-04-21T22:22:02.3745173Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mnasnet1_0...OK
2025-04-21T22:22:15.9118180Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2...OK
2025-04-21T22:22:29.4747674Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat...OK
2025-04-21T22:22:43.0446179Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v3_large...OK
2025-04-21T22:22:56.6104777Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco...OK
2025-04-21T22:23:32.1431825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream...OK
2025-04-21T22:23:32.1434729Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt...SKIP - No install.py is found
2025-04-21T22:23:46.9829942Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nvidia_deeprecommender...OK
2025-04-21T22:24:03.0839588Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/opacus_cifar10...OK
2025-04-21T22:24:16.6321714Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_densenet...OK
2025-04-21T22:24:30.1726633Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/phlippe_resnet...OK
2025-04-21T22:24:43.7352830Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_equation_of_state...OK
2025-04-21T22:24:57.3077729Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing...OK
2025-04-21T22:25:10.8983561Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_turbulent_kinetic_energy...OK
2025-04-21T22:25:31.3579135Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_CycleGAN_and_pix2pix...OK
2025-04-21T22:25:46.4355746Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_stargan...OK
2025-04-21T22:26:03.7035460Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pytorch_unet...OK
2025-04-21T22:26:17.2442513Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet152...OK
2025-04-21T22:26:30.7892834Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet18...OK
2025-04-21T22:26:44.3018825Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50...OK
2025-04-21T22:26:57.8483800Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat...OK
2025-04-21T22:27:11.3973184Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnext50_32x4d...OK
2025-04-21T22:27:34.5904765Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam...OK
2025-04-21T22:27:58.3166108Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/sam_fast...OK
2025-04-21T22:28:11.9239287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/shufflenet_v2_x1_0...OK
2025-04-21T22:28:11.9241936Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt...SKIP - No install.py is found
2025-04-21T22:28:11.9245287Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/simple_gpt_tp_manual...SKIP - No install.py is found
2025-04-21T22:28:28.1927120Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/soft_actor_critic...OK
2025-04-21T22:28:43.5657426Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer...OK
2025-04-21T22:28:57.0489504Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/squeezenet1_1...OK
2025-04-21T22:29:26.8874189Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_text_encoder...OK
2025-04-21T22:29:47.7551738Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/stable_diffusion_unet...OK
2025-04-21T22:30:06.0620277Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tacotron2...OK
2025-04-21T22:30:27.7566898Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientdet...OK
2025-04-21T22:30:41.2336389Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_efficientnet...OK
2025-04-21T22:30:54.7025370Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_nfnet...OK
2025-04-21T22:31:08.1797183Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_regnet...OK
2025-04-21T22:31:21.6509339Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_resnest...OK
2025-04-21T22:31:35.1427673Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer...OK
2025-04-21T22:31:48.5892252Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vision_transformer_large...OK
2025-04-21T22:32:02.0829931Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/timm_vovnet...OK
2025-04-21T22:32:21.8044537Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/torch_multimodal_clip...OK
2025-04-21T22:32:41.1744227Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/tts_angular...OK
2025-04-21T22:32:54.6363657Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vgg16...OK
2025-04-21T22:33:09.7860083Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/vision_maskrcnn...OK
2025-04-21T22:33:25.2866268Z running setup for /var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3...OK
2025-04-21T22:33:32.0228391Z installed torchbench with package constraints: {'numpy': '1.22.4', 'torch': '2.8.0a0+gitb68c0ef', 'torchvision': '0.19.0a0+d23a6e1', 'torchaudio': '2.6.0a0+bccaa45'}
2025-04-21T22:33:32.3342574Z + pip install transformers==4.38.1
2025-04-21T22:33:32.7901768Z Collecting transformers==4.38.1
2025-04-21T22:33:32.8094873Z   Downloading transformers-4.38.1-py3-none-any.whl.metadata (131 kB)
2025-04-21T22:33:33.0077960Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (3.16.1)
2025-04-21T22:33:33.0082748Z Requirement already satisfied: huggingface-hub<1.0,>=0.19.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.30.2)
2025-04-21T22:33:33.0087437Z Requirement already satisfied: numpy>=1.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (1.22.4)
2025-04-21T22:33:33.0090118Z Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (24.2)
2025-04-21T22:33:33.0094258Z Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (6.0.2)
2025-04-21T22:33:33.0098510Z Requirement already satisfied: regex!=2019.12.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2024.11.6)
2025-04-21T22:33:33.0102055Z Requirement already satisfied: requests in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (2.32.3)
2025-04-21T22:33:33.1489760Z Collecting tokenizers<0.19,>=0.14 (from transformers==4.38.1)
2025-04-21T22:33:33.1507871Z   Using cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)
2025-04-21T22:33:33.1536893Z Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (0.5.3)
2025-04-21T22:33:33.1541569Z Requirement already satisfied: tqdm>=4.27 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from transformers==4.38.1) (4.67.1)
2025-04-21T22:33:33.1765378Z Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (2024.10.0)
2025-04-21T22:33:33.1771961Z Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.19.3->transformers==4.38.1) (4.13.2)
2025-04-21T22:33:33.1949187Z Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.4.1)
2025-04-21T22:33:33.1951703Z Requirement already satisfied: idna<4,>=2.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (3.10)
2025-04-21T22:33:33.1955520Z Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2.4.0)
2025-04-21T22:33:33.1959235Z Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from requests->transformers==4.38.1) (2025.1.31)
2025-04-21T22:33:33.2408026Z Downloading transformers-4.38.1-py3-none-any.whl (8.5 MB)
2025-04-21T22:33:33.3024760Z [?25l   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0.0/8.5 MB ? eta -:--:--
2025-04-21T22:33:33.3025507Z    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.5/8.5 MB 144.6 MB/s eta 0:00:00
2025-04-21T22:33:33.3039641Z [?25hUsing cached tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)
2025-04-21T22:33:34.3015375Z Installing collected packages: tokenizers, transformers
2025-04-21T22:33:34.3016513Z   Attempting uninstall: tokenizers
2025-04-21T22:33:34.3029951Z     Found existing installation: tokenizers 0.19.1
2025-04-21T22:33:34.3073311Z     Uninstalling tokenizers-0.19.1:
2025-04-21T22:33:34.3089330Z       Successfully uninstalled tokenizers-0.19.1
2025-04-21T22:33:34.3845943Z   Attempting uninstall: transformers
2025-04-21T22:33:34.3873664Z     Found existing installation: transformers 4.44.2
2025-04-21T22:33:34.5966499Z     Uninstalling transformers-4.44.2:
2025-04-21T22:33:34.6290294Z       Successfully uninstalled transformers-4.44.2
2025-04-21T22:33:38.7631121Z Successfully installed tokenizers-0.15.2 transformers-4.38.1
2025-04-21T22:33:38.8965163Z + echo 'Print all dependencies after TorchBench is installed'
2025-04-21T22:33:38.8965633Z Print all dependencies after TorchBench is installed
2025-04-21T22:33:38.8966032Z + python -mpip freeze
2025-04-21T22:33:39.3443042Z absl-py==2.2.2
2025-04-21T22:33:39.3443280Z accelerate==1.6.0
2025-04-21T22:33:39.3443523Z aiohappyeyeballs==2.6.1
2025-04-21T22:33:39.3443767Z aiohttp==3.11.16
2025-04-21T22:33:39.3444018Z aiosignal==1.3.2
2025-04-21T22:33:39.3444253Z annotated-types==0.7.0
2025-04-21T22:33:39.3444523Z antlr4-python3-runtime==4.9.3
2025-04-21T22:33:39.3444790Z anyascii==0.3.2
2025-04-21T22:33:39.3445016Z astroid==3.3.9
2025-04-21T22:33:39.3445236Z astunparse==1.6.3
2025-04-21T22:33:39.3445474Z async-timeout==5.0.1
2025-04-21T22:33:39.3445708Z attrs==23.1.0
2025-04-21T22:33:39.3445920Z audioread==3.0.1
2025-04-21T22:33:39.3446130Z babel==2.17.0
2025-04-21T22:33:39.3446367Z beautifulsoup4==4.13.4
2025-04-21T22:33:39.3447129Z -e git+https://github.com/pytorch/benchmark@373ffb19dc470f4423a3176a4133f8f4b3cdb5bd#egg=bert_pytorch&subdirectory=torchbenchmark/models/BERT_pytorch
2025-04-21T22:33:39.3447902Z black==25.1.0
2025-04-21T22:33:39.3448118Z blinker==1.9.0
2025-04-21T22:33:39.3448921Z blis==1.2.1
2025-04-21T22:33:39.3449208Z blobfile==3.0.0
2025-04-21T22:33:39.3449511Z bokeh==3.7.2
2025-04-21T22:33:39.3449792Z boto3==1.35.42
2025-04-21T22:33:39.3450092Z botocore==1.35.99
2025-04-21T22:33:39.3462033Z bs4==0.0.2
2025-04-21T22:33:39.3462405Z cachetools==5.5.2
2025-04-21T22:33:39.3462755Z cardboardlint==1.3.1
2025-04-21T22:33:39.3463065Z catalogue==2.0.10
2025-04-21T22:33:39.3463383Z certifi==2025.1.31
2025-04-21T22:33:39.3463723Z cffi==1.17.1
2025-04-21T22:33:39.3464028Z charset-normalizer==3.4.1
2025-04-21T22:33:39.3464374Z click==8.1.8
2025-04-21T22:33:39.3464645Z cloudpathlib==0.21.0
2025-04-21T22:33:39.3464880Z cloudpickle==3.1.1
2025-04-21T22:33:39.3465417Z colorama==0.4.6
2025-04-21T22:33:39.3465698Z confection==0.1.5
2025-04-21T22:33:39.3465925Z contourpy==1.2.1
2025-04-21T22:33:39.3466152Z coremltools==5.0b5
2025-04-21T22:33:39.3466431Z cryptography==44.0.2
2025-04-21T22:33:39.3466742Z csvw==3.5.1
2025-04-21T22:33:39.3467036Z cycler==0.12.1
2025-04-21T22:33:39.3467349Z cymem==2.0.11
2025-04-21T22:33:39.3467657Z Cython==3.0.12
2025-04-21T22:33:39.3467935Z DALL-E==0.1
2025-04-21T22:33:39.3468151Z dataclasses-json==0.6.7
2025-04-21T22:33:39.3468403Z datasets==3.5.0
2025-04-21T22:33:39.3468628Z decorator==5.2.1
2025-04-21T22:33:39.3468848Z defusedxml==0.7.1
2025-04-21T22:33:39.3469079Z Deprecated==1.2.18
2025-04-21T22:33:39.3469621Z detectron2 @ git+https://github.com/facebookresearch/detectron2.git@0df2d73d0013db7de629602c23cc120219b4f2b8
2025-04-21T22:33:39.3470216Z diffusers==0.30.3
2025-04-21T22:33:39.3470443Z dill==0.3.8
2025-04-21T22:33:39.3470651Z diskcache==5.6.3
2025-04-21T22:33:39.3470867Z distro==1.9.0
2025-04-21T22:33:39.3471084Z dlinfo==2.0.0
2025-04-21T22:33:39.3471318Z docker-pycreds==0.4.0
2025-04-21T22:33:39.3471623Z dominate==2.9.1
2025-04-21T22:33:39.3472141Z effdet @ git+https://github.com/rwightman/efficientdet-pytorch.git@d43c9e34cd62d22b4205831bb735f6dd83b8e881
2025-04-21T22:33:39.3472715Z einops==0.8.1
2025-04-21T22:33:39.3472943Z exceptiongroup==1.2.2
2025-04-21T22:33:39.3473186Z execnet==2.1.1
2025-04-21T22:33:39.3473488Z expecttest==0.3.0
2025-04-21T22:33:39.3473717Z FastNLP==0.6.0
2025-04-21T22:33:39.3473936Z fbscribelogger==0.1.7
2025-04-21T22:33:39.3474190Z ffmpeg-python==0.2.0
2025-04-21T22:33:39.3474429Z filelock==3.16.1
2025-04-21T22:33:39.3474643Z Flask==3.1.0
2025-04-21T22:33:39.3474858Z flatbuffers==2.0
2025-04-21T22:33:39.3475085Z fonttools==4.57.0
2025-04-21T22:33:39.3475305Z frozenlist==1.6.0
2025-04-21T22:33:39.3475528Z fsspec==2024.10.0
2025-04-21T22:33:39.3475749Z ftfy==6.3.1
2025-04-21T22:33:39.3476186Z functorch @ git+https://github.com/pytorch/functorch.git@b71aa0b4387b86c278132209b99538be48ef4c74
2025-04-21T22:33:39.3476710Z future==1.0.0
2025-04-21T22:33:39.3476940Z fvcore==0.1.5.post20221221
2025-04-21T22:33:39.3477187Z gdown==5.2.0
2025-04-21T22:33:39.3477402Z ghstack==0.8.0
2025-04-21T22:33:39.3477619Z gitdb==4.0.12
2025-04-21T22:33:39.3477828Z GitPython==3.1.44
2025-04-21T22:33:39.3478056Z google-auth==2.39.0
2025-04-21T22:33:39.3478308Z google-auth-oauthlib==1.0.0
2025-04-21T22:33:39.3478565Z grpcio==1.71.0
2025-04-21T22:33:39.3478782Z gym==0.26.2
2025-04-21T22:33:39.3478995Z gym-notices==0.0.8
2025-04-21T22:33:39.3479214Z h5py==3.13.0
2025-04-21T22:33:39.3479427Z higher==0.2.1
2025-04-21T22:33:39.3479655Z huggingface-hub==0.30.2
2025-04-21T22:33:39.3479905Z hydra-core==1.3.2
2025-04-21T22:33:39.3480130Z hypothesis==5.35.1
2025-04-21T22:33:39.3480355Z idna==3.10
2025-04-21T22:33:39.3480566Z imageio==2.37.0
2025-04-21T22:33:39.3480792Z importlib_metadata==8.6.1
2025-04-21T22:33:39.3481050Z inflect==7.5.0
2025-04-21T22:33:39.3481269Z iniconfig==2.1.0
2025-04-21T22:33:39.3481485Z iopath==0.1.9
2025-04-21T22:33:39.3481703Z isodate==0.7.2
2025-04-21T22:33:39.3481918Z isort==6.0.1
2025-04-21T22:33:39.3482129Z itsdangerous==2.2.0
2025-04-21T22:33:39.3482358Z Jinja2==3.1.6
2025-04-21T22:33:39.3482577Z jmespath==1.0.1
2025-04-21T22:33:39.3482789Z joblib==1.4.2
2025-04-21T22:33:39.3483005Z jsonpatch==1.33
2025-04-21T22:33:39.3483363Z jsonpointer==3.0.0
2025-04-21T22:33:39.3483587Z jsonschema==4.23.0
2025-04-21T22:33:39.3483848Z jsonschema-specifications==2024.10.1
2025-04-21T22:33:39.3484150Z junitparser==2.1.1
2025-04-21T22:33:39.3484373Z kaldi-io==0.9.8
2025-04-21T22:33:39.3484596Z kiwisolver==1.4.8
2025-04-21T22:33:39.3484818Z kornia==0.8.0
2025-04-21T22:33:39.3485027Z kornia_rs==0.1.8
2025-04-21T22:33:39.3485249Z lameenc==1.8.1
2025-04-21T22:33:39.3485467Z langcodes==3.5.0
2025-04-21T22:33:39.3485685Z langdetect==1.0.9
2025-04-21T22:33:39.3485917Z language-tags==1.2.0
2025-04-21T22:33:39.3486157Z language_data==1.3.0
2025-04-21T22:33:39.3486381Z lark==0.12.0
2025-04-21T22:33:39.3486595Z lazy_loader==0.4
2025-04-21T22:33:39.3486902Z libcst==1.7.0
2025-04-21T22:33:39.3487110Z librosa==0.9.2
2025-04-21T22:33:39.3487329Z lintrunner==0.12.7
2025-04-21T22:33:39.3487554Z llvmlite==0.38.1
2025-04-21T22:33:39.3487765Z lxml==5.3.0
2025-04-21T22:33:39.3487976Z marisa-trie==1.2.1
2025-04-21T22:33:39.3488200Z Markdown==3.8
2025-04-21T22:33:39.3488424Z markdown-it-py==3.0.0
2025-04-21T22:33:39.3488671Z MarkupSafe==3.0.2
2025-04-21T22:33:39.3488907Z marshmallow==3.26.1
2025-04-21T22:33:39.3489138Z matplotlib==3.8.4
2025-04-21T22:33:39.3489354Z mccabe==0.7.0
2025-04-21T22:33:39.3489567Z mdurl==0.1.2
2025-04-21T22:33:39.3489781Z ml_dtypes==0.5.1
2025-04-21T22:33:39.3490000Z MonkeyType==23.3.0
2025-04-21T22:33:39.3490235Z more-itertools==10.6.0
2025-04-21T22:33:39.3490480Z mpmath==1.3.0
2025-04-21T22:33:39.3490689Z msgpack==1.1.0
2025-04-21T22:33:39.3490908Z multidict==6.4.3
2025-04-21T22:33:39.3491140Z multiprocess==0.70.16
2025-04-21T22:33:39.3491377Z murmurhash==1.0.12
2025-04-21T22:33:39.3491603Z musdb==0.4.2
2025-04-21T22:33:39.3491821Z museval==0.4.1
2025-04-21T22:33:39.3492029Z mypy==1.14.0
2025-04-21T22:33:39.3492252Z mypy-extensions==1.0.0
2025-04-21T22:33:39.3492497Z narwhals==1.35.0
2025-04-21T22:33:39.3492715Z networkx==2.8.8
2025-04-21T22:33:39.3492933Z ninja==1.10.2
2025-04-21T22:33:39.3493144Z nose==1.3.7
2025-04-21T22:33:39.3493348Z numba==0.55.2
2025-04-21T22:33:39.3493561Z numpy==1.22.4
2025-04-21T22:33:39.3493792Z nvidia-cublas-cu12==12.4.5.8
2025-04-21T22:33:39.3494074Z nvidia-cuda-cupti-cu12==12.4.127
2025-04-21T22:33:39.3494374Z nvidia-cuda-nvrtc-cu12==12.4.127
2025-04-21T22:33:39.3494677Z nvidia-cuda-runtime-cu12==12.4.127
2025-04-21T22:33:39.3494972Z nvidia-cudnn-cu12==9.1.0.70
2025-04-21T22:33:39.3495245Z nvidia-cufft-cu12==11.2.1.3
2025-04-21T22:33:39.3495519Z nvidia-curand-cu12==10.3.5.147
2025-04-21T22:33:39.3495801Z nvidia-cusolver-cu12==11.6.1.9
2025-04-21T22:33:39.3496088Z nvidia-cusparse-cu12==12.3.1.170
2025-04-21T22:33:39.3496382Z nvidia-cusparselt-cu12==0.6.2
2025-04-21T22:33:39.3496664Z nvidia-ml-py==12.570.86
2025-04-21T22:33:39.3496920Z nvidia-nccl-cu12==2.21.5
2025-04-21T22:33:39.3497194Z nvidia-nvjitlink-cu12==12.4.127
2025-04-21T22:33:39.3497480Z nvidia-nvtx-cu12==12.4.127
2025-04-21T22:33:39.3497734Z oauthlib==3.2.2
2025-04-21T22:33:39.3497958Z omegaconf==2.3.0
2025-04-21T22:33:39.3498179Z onnx==1.17.0
2025-04-21T22:33:39.3498395Z onnxscript==0.2.2
2025-04-21T22:33:39.3498621Z opacus==1.5.3
2025-04-21T22:33:39.3498846Z opencv-python==4.8.0.74
2025-04-21T22:33:39.3499088Z opt-einsum==3.3.0
2025-04-21T22:33:39.3499317Z optree==0.13.0
2025-04-21T22:33:39.3499536Z packaging==24.2
2025-04-21T22:33:39.3499754Z pandas==2.0.3
2025-04-21T22:33:39.3499984Z parameterized==0.8.1
2025-04-21T22:33:39.3500222Z patch==1.16
2025-04-21T22:33:39.3500428Z pathspec==0.12.1
2025-04-21T22:33:39.3500657Z phonemizer==3.3.0
2025-04-21T22:33:39.3500884Z pillow==11.0.0
2025-04-21T22:33:39.3501103Z platformdirs==4.3.7
2025-04-21T22:33:39.3501336Z pluggy==1.5.0
2025-04-21T22:33:39.3501548Z ply==3.11
2025-04-21T22:33:39.3501748Z pooch==1.8.2
2025-04-21T22:33:39.3501969Z portalocker==3.1.1
2025-04-21T22:33:39.3502200Z preshed==3.0.9
2025-04-21T22:33:39.3502416Z prettytable==3.16.0
2025-04-21T22:33:39.3502653Z propcache==0.3.1
2025-04-21T22:33:39.3502885Z protobuf==3.20.2
2025-04-21T22:33:39.3503101Z psutil==7.0.0
2025-04-21T22:33:39.3503315Z PuLP==2.9.0
2025-04-21T22:33:39.3503616Z pwlf==2.2.1
2025-04-21T22:33:39.3503829Z py-cpuinfo==9.0.0
2025-04-21T22:33:39.3504058Z pyaml==25.1.0
2025-04-21T22:33:39.3504272Z pyarrow==19.0.1
2025-04-21T22:33:39.3504486Z pyasn1==0.6.1
2025-04-21T22:33:39.3504709Z pyasn1_modules==0.4.2
2025-04-21T22:33:39.3504958Z pyclipper==1.3.0.post6
2025-04-21T22:33:39.3505207Z pycocotools==2.0.8
2025-04-21T22:33:39.3505432Z pycparser==2.22
2025-04-21T22:33:39.3505661Z pycryptodomex==3.22.0
2025-04-21T22:33:39.3505904Z pydantic==2.11.3
2025-04-21T22:33:39.3506126Z pydantic_core==2.33.1
2025-04-21T22:33:39.3506359Z pyDOE==0.3.8
2025-04-21T22:33:39.3506569Z pydot==3.0.4
2025-04-21T22:33:39.3506780Z pygame==2.6.1
2025-04-21T22:33:39.3507081Z PyGithub==2.3.0
2025-04-21T22:33:39.3507298Z Pygments==2.15.0
2025-04-21T22:33:39.3507518Z PyJWT==2.10.1
2025-04-21T22:33:39.3507730Z pylint==3.3.6
2025-04-21T22:33:39.3507939Z PyNaCl==1.5.0
2025-04-21T22:33:39.3508149Z pynvml==12.0.0
2025-04-21T22:33:39.3508373Z pyparsing==3.2.3
2025-04-21T22:33:39.3508602Z pypdfium2==4.30.1
2025-04-21T22:33:39.3508833Z pysbd==0.3.4
2025-04-21T22:33:39.3509046Z PySocks==1.7.1
2025-04-21T22:33:39.3509258Z pytest==8.3.5
2025-04-21T22:33:39.3509484Z pytest-benchmark==5.1.0
2025-04-21T22:33:39.3509742Z pytest-cpp==2.3.0
2025-04-21T22:33:39.3509982Z pytest-flakefinder==1.1.0
2025-04-21T22:33:39.3510263Z pytest-rerunfailures==14.0
2025-04-21T22:33:39.3510539Z pytest-subtests==0.13.1
2025-04-21T22:33:39.3510792Z pytest-xdist==3.3.1
2025-04-21T22:33:39.3511036Z python-dateutil==2.9.0.post0
2025-04-21T22:33:39.3511307Z python-doctr==0.11.0
2025-04-21T22:33:39.3511994Z pytorch-labs-segment-anything-fast @ git+https://github.com/pytorch-labs/segment-anything-fast.git@e6aadeb86f3ae1f58c3f98e2a91e251716e0f2aa
2025-04-21T22:33:39.3512706Z pytz==2025.2
2025-04-21T22:33:39.3512932Z PyWavelets==1.4.1
2025-04-21T22:33:39.3513160Z PyYAML==6.0.2
2025-04-21T22:33:39.3513431Z pyzstd==0.16.2
2025-04-21T22:33:39.3513663Z RapidFuzz==3.13.0
2025-04-21T22:33:39.3513884Z rdflib==7.1.4
2025-04-21T22:33:39.3514095Z redis==5.2.1
2025-04-21T22:33:39.3514312Z referencing==0.36.2
2025-04-21T22:33:39.3514542Z regex==2024.11.6
2025-04-21T22:33:39.3514763Z requests==2.32.3
2025-04-21T22:33:39.3514998Z requests-oauthlib==2.0.0
2025-04-21T22:33:39.3515265Z resampy==0.4.3
2025-04-21T22:33:39.3515476Z rfc3986==1.5.0
2025-04-21T22:33:39.3515694Z rich==14.0.0
2025-04-21T22:33:39.3515907Z rpds-py==0.24.0
2025-04-21T22:33:39.3516119Z rsa==4.9.1
2025-04-21T22:33:39.3516328Z s3transfer==0.10.4
2025-04-21T22:33:39.3516562Z safetensors==0.5.3
2025-04-21T22:33:39.3516792Z scikit-image==0.22.0
2025-04-21T22:33:39.3517036Z scikit-learn==1.6.1
2025-04-21T22:33:39.3517268Z scipy==1.10.1
2025-04-21T22:33:39.3517488Z segments==2.3.0
2025-04-21T22:33:39.3517722Z sentencepiece==0.2.0
2025-04-21T22:33:39.3517965Z sentry-sdk==2.26.1
2025-04-21T22:33:39.3518199Z setproctitle==1.3.5
2025-04-21T22:33:39.3518425Z shapely==2.1.0
2025-04-21T22:33:39.3518648Z shellingham==1.5.4
2025-04-21T22:33:39.3518882Z simplejson==3.20.1
2025-04-21T22:33:39.3519101Z six==1.17.0
2025-04-21T22:33:39.3519312Z smart-open==7.1.0
2025-04-21T22:33:39.3519550Z smmap==5.0.2
2025-04-21T22:33:39.3519772Z sortedcontainers==2.4.0
2025-04-21T22:33:39.3520026Z soundfile==0.13.1
2025-04-21T22:33:39.3520252Z soupsieve==2.7
2025-04-21T22:33:39.3520465Z soxr==0.5.0.post1
2025-04-21T22:33:39.3520686Z spacy==3.8.5
2025-04-21T22:33:39.3520914Z spacy-legacy==3.0.12
2025-04-21T22:33:39.3521153Z spacy-loggers==1.0.5
2025-04-21T22:33:39.3521386Z srsly==2.5.1
2025-04-21T22:33:39.3521939Z stempeg==0.2.3
2025-04-21T22:33:39.3522154Z submitit==1.5.2
2025-04-21T22:33:39.3522375Z sympy==1.13.3
2025-04-21T22:33:39.3522589Z tabulate==0.9.0
2025-04-21T22:33:39.3522818Z tb-nightly==2.13.0a20230426
2025-04-21T22:33:39.3523080Z tensorboard==2.13.0
2025-04-21T22:33:39.3523336Z tensorboard-data-server==0.7.2
2025-04-21T22:33:39.3523614Z tensorboardX==2.6.2.2
2025-04-21T22:33:39.3523857Z termcolor==3.0.1
2025-04-21T22:33:39.3524083Z thinc==8.3.4
2025-04-21T22:33:39.3524297Z threadpoolctl==3.6.0
2025-04-21T22:33:39.3524699Z thriftpy2==0.5.2
2025-04-21T22:33:39.3524929Z tifffile==2025.3.30
2025-04-21T22:33:39.3525437Z timm @ git+https://github.com/huggingface/pytorch-image-models.git@730b907b4d45a4713cbc425cbf224c46089fd514
2025-04-21T22:33:39.3526009Z tlparse==0.3.30
2025-04-21T22:33:39.3526241Z tokenizers==0.15.2
2025-04-21T22:33:39.3526474Z tomli==2.2.1
2025-04-21T22:33:39.3526680Z tomlkit==0.13.2
2025-04-21T22:33:39.3527457Z torch @ file:///var/lib/jenkins/workspace/dist/torch-2.8.0a0%2Bgitb68c0ef-cp310-cp310-linux_x86_64.whl#sha256=9bbbdfa20e86b1248aa24ec871651c6a0e5290f9fa886ae1219a5942117eb26f
2025-04-21T22:33:39.3528696Z torch_geometric @ git+https://github.com/pyg-team/pytorch_geometric.git@cabcd4097442ba60aa1efa11e1619dd9bb8fb527
2025-04-21T22:33:39.3529478Z torchao @ git+https://github.com/pytorch/ao.git@4805efd03b1aea9b5b9fe84df5c5a7171b5a7f2c
2025-04-21T22:33:39.3530196Z torchaudio @ git+https://github.com/pytorch/audio.git@bccaa454a54c3c648697cc2f46a4fb0500b1f01b
2025-04-21T22:33:39.3531039Z torchmultimodal @ git+https://github.com/facebookresearch/multimodal.git@6569fcc03450c2360b50d772bf9b18ec3487fcf4
2025-04-21T22:33:39.3531875Z torchvision @ git+https://github.com/pytorch/vision.git@d23a6e1664d20707c11781299611436e1f0c104f
2025-04-21T22:33:39.3532395Z tornado==6.4.2
2025-04-21T22:33:39.3532625Z tqdm==4.67.1
2025-04-21T22:33:39.3532849Z transformers==4.38.1
2025-04-21T22:33:39.3533089Z treetable==0.2.5
2025-04-21T22:33:39.3533752Z triton @ file:///opt/triton/triton-3.3.0%2Bgit96316ce5-cp310-cp310-linux_x86_64.whl#sha256=417d95e3b977e7a78a0979b36cd88613615b5045352cbac12a89503f9febf540
2025-04-21T22:33:39.3534474Z typeguard==4.4.2
2025-04-21T22:33:39.3534692Z typer==0.15.2
2025-04-21T22:33:39.3534929Z typing-inspect==0.9.0
2025-04-21T22:33:39.3535191Z typing-inspection==0.4.0
2025-04-21T22:33:39.3535466Z typing_extensions==4.13.2
2025-04-21T22:33:39.3535720Z tzdata==2025.2
2025-04-21T22:33:39.3535941Z Unidecode==1.3.8
2025-04-21T22:33:39.3536192Z unittest-xml-reporting==3.2.0
2025-04-21T22:33:39.3536479Z uritemplate==4.1.1
2025-04-21T22:33:39.3536716Z urllib3==2.4.0
2025-04-21T22:33:39.3536940Z visdom==0.2.4
2025-04-21T22:33:39.3537148Z wandb==0.19.9
2025-04-21T22:33:39.3537364Z wasabi==1.1.3
2025-04-21T22:33:39.3537579Z wcwidth==0.2.13
2025-04-21T22:33:39.3537792Z weasel==0.4.1
2025-04-21T22:33:39.3538016Z websocket-client==1.8.0
2025-04-21T22:33:39.3538267Z Werkzeug==3.1.3
2025-04-21T22:33:39.3538480Z wrapt==1.17.2
2025-04-21T22:33:39.3538697Z xdoctest==1.1.0
2025-04-21T22:33:39.3538916Z xxhash==3.5.0
2025-04-21T22:33:39.3539131Z xyzservices==2025.1.0
2025-04-21T22:33:39.3539370Z yacs==0.1.8
2025-04-21T22:33:39.3539573Z yarl==1.20.0
2025-04-21T22:33:39.3539780Z z3-solver==4.12.6.0
2025-04-21T22:33:39.3540012Z zipp==3.21.0
2025-04-21T22:33:39.3767776Z + popd
2025-04-21T22:33:39.3768038Z ~/workspace
2025-04-21T22:33:39.3768362Z + [[ inductor_torchbench != *cpu* ]]
2025-04-21T22:33:39.3768668Z + install_torchrec_and_fbgemm
2025-04-21T22:33:39.3772480Z + local torchrec_commit
2025-04-21T22:33:39.3775941Z ++ get_pinned_commit torchrec
2025-04-21T22:33:39.3779566Z ++ cat .github/ci_commit_pins/torchrec.txt
2025-04-21T22:33:39.3803657Z + torchrec_commit=6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T22:33:39.3804170Z + local fbgemm_commit
2025-04-21T22:33:39.3808252Z ++ get_pinned_commit fbgemm
2025-04-21T22:33:39.3808553Z ++ cat .github/ci_commit_pins/fbgemm.txt
2025-04-21T22:33:39.3830677Z + fbgemm_commit=de731af65b4f04696e85c729e3282450b51b95fd
2025-04-21T22:33:39.3831154Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]]
2025-04-21T22:33:39.3831525Z + pip_uninstall torchrec-nightly
2025-04-21T22:33:39.3831845Z + pip3 uninstall -y torchrec-nightly
2025-04-21T22:33:39.7729671Z WARNING: Skipping torchrec-nightly as it is not installed.
2025-04-21T22:33:39.8099500Z + pip_uninstall fbgemm-gpu-nightly
2025-04-21T22:33:39.8099900Z + pip3 uninstall -y fbgemm-gpu-nightly
2025-04-21T22:33:40.1977455Z WARNING: Skipping fbgemm-gpu-nightly as it is not installed.
2025-04-21T22:33:40.2343613Z + pip_install setuptools-git-versioning scikit-build pyre-extensions
2025-04-21T22:33:40.2344163Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T22:33:40.2344779Z + python3 -m pip install --progress-bar off setuptools-git-versioning scikit-build pyre-extensions
2025-04-21T22:33:40.6766159Z Collecting setuptools-git-versioning
2025-04-21T22:33:40.6958245Z   Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl.metadata (6.1 kB)
2025-04-21T22:33:40.7183706Z Collecting scikit-build
2025-04-21T22:33:40.7253904Z   Downloading scikit_build-0.18.1-py3-none-any.whl.metadata (18 kB)
2025-04-21T22:33:40.7599857Z Collecting pyre-extensions
2025-04-21T22:33:40.7636463Z   Downloading pyre_extensions-0.0.32-py3-none-any.whl.metadata (4.0 kB)
2025-04-21T22:33:40.7702815Z Requirement already satisfied: packaging in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (24.2)
2025-04-21T22:33:40.7706077Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (78.1.0)
2025-04-21T22:33:40.7710536Z Requirement already satisfied: tomli>=2.0.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from setuptools-git-versioning) (2.2.1)
2025-04-21T22:33:40.7720001Z Requirement already satisfied: distro in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (1.9.0)
2025-04-21T22:33:40.7726273Z Requirement already satisfied: wheel>=0.32.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from scikit-build) (0.45.1)
2025-04-21T22:33:40.7732363Z Requirement already satisfied: typing-inspect in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (0.9.0)
2025-04-21T22:33:40.7735206Z Requirement already satisfied: typing-extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pyre-extensions) (4.13.2)
2025-04-21T22:33:40.7820607Z Requirement already satisfied: mypy-extensions>=0.3.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from typing-inspect->pyre-extensions) (1.0.0)
2025-04-21T22:33:40.7861344Z Downloading setuptools_git_versioning-2.1.0-py3-none-any.whl (10 kB)
2025-04-21T22:33:40.7945127Z Downloading scikit_build-0.18.1-py3-none-any.whl (85 kB)
2025-04-21T22:33:40.8057974Z Downloading pyre_extensions-0.0.32-py3-none-any.whl (12 kB)
2025-04-21T22:33:41.6727330Z Installing collected packages: setuptools-git-versioning, scikit-build, pyre-extensions
2025-04-21T22:33:41.7880628Z Successfully installed pyre-extensions-0.0.32 scikit-build-0.18.1 setuptools-git-versioning-2.1.0
2025-04-21T22:33:41.8982288Z + [[ linux-focal-cuda12.6-py3.10-gcc9-sm86 == *rocm* ]]
2025-04-21T22:33:41.8982774Z + CUDA_PATH=/usr/local/cuda-12.1
2025-04-21T22:33:41.8983623Z + pip_install --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2025-04-21T22:33:41.8984608Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T22:33:41.8985613Z + python3 -m pip install --progress-bar off --no-use-pep517 --user 'git+https://github.com/pytorch/FBGEMM.git@de731af65b4f04696e85c729e3282450b51b95fd#egg=fbgemm-gpu&subdirectory=fbgemm_gpu'
2025-04-21T22:33:42.2811805Z Collecting fbgemm-gpu
2025-04-21T22:33:42.2815161Z   Cloning https://github.com/pytorch/FBGEMM.git (to revision de731af65b4f04696e85c729e3282450b51b95fd) to /tmp/pip-install-vhs3599u/fbgemm-gpu_5fa2d4d8a30d4ea8b80cb16edfe820c8
2025-04-21T22:33:42.2846848Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/FBGEMM.git /tmp/pip-install-vhs3599u/fbgemm-gpu_5fa2d4d8a30d4ea8b80cb16edfe820c8
2025-04-21T22:33:43.4374005Z   Running command git rev-parse -q --verify 'sha^de731af65b4f04696e85c729e3282450b51b95fd'
2025-04-21T22:33:43.4403168Z   Running command git fetch -q https://github.com/pytorch/FBGEMM.git de731af65b4f04696e85c729e3282450b51b95fd
2025-04-21T22:33:44.2236829Z   Running command git checkout -q de731af65b4f04696e85c729e3282450b51b95fd
2025-04-21T22:33:44.7554039Z   Resolved https://github.com/pytorch/FBGEMM.git to commit de731af65b4f04696e85c729e3282450b51b95fd
2025-04-21T22:33:44.7554701Z   Running command git submodule update --init --recursive -q
2025-04-21T22:33:53.3305791Z   Preparing metadata (setup.py) ... [?25l- done
2025-04-21T22:33:53.3325585Z [?25hRequirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu) (1.22.4)
2025-04-21T22:33:53.3333125Z Building wheels for collected packages: fbgemm-gpu
2025-04-21T23:18:58.8077367Z   Building wheel for fbgemm-gpu (setup.py) ... [?25l- \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | / - \ | done
2025-04-21T23:18:58.9929828Z [?25h  Created wheel for fbgemm-gpu: filename=fbgemm_gpu-0.4.1rc0.post421-cp310-cp310-linux_x86_64.whl size=253792746 sha256=a5f7d27df21e2048bd51e0921f202859fa4804083fe7a405f9255964745842a0
2025-04-21T23:18:58.9931965Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/d8/d9/f3/f5c260aba9c9f0a533444c12621a0c2ceb8fbcffce2b5beb39
2025-04-21T23:18:58.9948587Z Successfully built fbgemm-gpu
2025-04-21T23:18:59.8688308Z Installing collected packages: fbgemm-gpu
2025-04-21T23:19:03.1152858Z Successfully installed fbgemm-gpu-0.4.1rc0.post421
2025-04-21T23:19:04.0622733Z + pip_install --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:04.0623472Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-04-21T23:19:04.0624344Z + python3 -m pip install --progress-bar off --no-use-pep517 --user git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:04.4443311Z Collecting git+https://github.com/pytorch/torchrec.git@6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:04.4447366Z   Cloning https://github.com/pytorch/torchrec.git (to revision 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2) to /tmp/pip-req-build-k69szuwn
2025-04-21T23:19:04.4477701Z   Running command git clone --filter=blob:none --quiet https://github.com/pytorch/torchrec.git /tmp/pip-req-build-k69szuwn
2025-04-21T23:19:05.2573683Z   Running command git rev-parse -q --verify 'sha^6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2'
2025-04-21T23:19:05.2602397Z   Running command git fetch -q https://github.com/pytorch/torchrec.git 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:05.6631268Z   Running command git checkout -q 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:06.0329201Z   Resolved https://github.com/pytorch/torchrec.git to commit 6cd9fd362514d14ebb9ed51314c62ac1e1e2bbf2
2025-04-21T23:19:06.3106884Z   Preparing metadata (setup.py) ... [?25l- done
2025-04-21T23:19:06.3142792Z [?25hRequirement already satisfied: iopath in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.1.9)
2025-04-21T23:19:06.3146400Z Requirement already satisfied: pandas in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (2.0.3)
2025-04-21T23:19:06.3149312Z Requirement already satisfied: tabulate in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (0.9.0)
2025-04-21T23:19:06.3574744Z Collecting torchmetrics (from torchrec==0.3.2)
2025-04-21T23:19:06.3755428Z   Downloading torchmetrics-1.7.1-py3-none-any.whl.metadata (21 kB)
2025-04-21T23:19:06.4229515Z Requirement already satisfied: tqdm in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchrec==0.3.2) (4.67.1)
2025-04-21T23:19:06.4231287Z Requirement already satisfied: fbgemm-gpu in /var/lib/jenkins/.local/lib/python3.10/site-packages (from torchrec==0.3.2) (0.4.1rc0.post421)
2025-04-21T23:19:06.4242903Z Requirement already satisfied: numpy in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from fbgemm-gpu->torchrec==0.3.2) (1.22.4)
2025-04-21T23:19:06.4252660Z Requirement already satisfied: portalocker in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from iopath->torchrec==0.3.2) (3.1.1)
2025-04-21T23:19:06.4533628Z Requirement already satisfied: python-dateutil>=2.8.2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2.9.0.post0)
2025-04-21T23:19:06.4537928Z Requirement already satisfied: pytz>=2020.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.2)
2025-04-21T23:19:06.4542032Z Requirement already satisfied: tzdata>=2022.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from pandas->torchrec==0.3.2) (2025.2)
2025-04-21T23:19:06.4566266Z Requirement already satisfied: packaging>17.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (24.2)
2025-04-21T23:19:06.4570795Z Requirement already satisfied: torch>=2.0.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torchmetrics->torchrec==0.3.2) (2.8.0a0+gitb68c0ef)
2025-04-21T23:19:06.4700511Z Collecting lightning-utilities>=0.8.0 (from torchmetrics->torchrec==0.3.2)
2025-04-21T23:19:06.4726771Z   Downloading lightning_utilities-0.14.3-py3-none-any.whl.metadata (5.6 kB)
2025-04-21T23:19:06.4814925Z Requirement already satisfied: setuptools in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (78.1.0)
2025-04-21T23:19:06.4819583Z Requirement already satisfied: typing_extensions in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from lightning-utilities>=0.8.0->torchmetrics->torchrec==0.3.2) (4.13.2)
2025-04-21T23:19:06.4841666Z Requirement already satisfied: six>=1.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas->torchrec==0.3.2) (1.17.0)
2025-04-21T23:19:06.4874704Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.16.1)
2025-04-21T23:19:06.4879797Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.13.3)
2025-04-21T23:19:06.4883323Z Requirement already satisfied: networkx in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2.8.8)
2025-04-21T23:19:06.4886905Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.1.6)
2025-04-21T23:19:06.4890505Z Requirement already satisfied: fsspec in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch>=2.0.0->torchmetrics->torchrec==0.3.2) (2024.10.0)
2025-04-21T23:19:06.4940292Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (1.3.0)
2025-04-21T23:19:06.5326862Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch>=2.0.0->torchmetrics->torchrec==0.3.2) (3.0.2)
2025-04-21T23:19:06.5475681Z Downloading torchmetrics-1.7.1-py3-none-any.whl (961 kB)
2025-04-21T23:19:06.5731995Z Downloading lightning_utilities-0.14.3-py3-none-any.whl (28 kB)
2025-04-21T23:19:06.5826190Z Building wheels for collected packages: torchrec
2025-04-21T23:19:07.1499631Z   Building wheel for torchrec (setup.py) ... [?25l- \ | done
2025-04-21T23:19:07.1510028Z [?25h  Created wheel for torchrec: filename=torchrec-0.3.2-py3-none-any.whl size=374580 sha256=87405af310bd601f0df3c33db4d8fefeadd9b18c019e052381ac676e0e175c02
2025-04-21T23:19:07.1511275Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/8e/a1/47/39ede01672ba82c08fb521bfc057cc4347e4b0e951738c8ca8
2025-04-21T23:19:07.1537988Z Successfully built torchrec
2025-04-21T23:19:07.9824535Z Installing collected packages: lightning-utilities, torchmetrics, torchrec
2025-04-21T23:19:08.6278686Z Successfully installed lightning-utilities-0.14.3 torchmetrics-1.7.1 torchrec-0.3.2
2025-04-21T23:19:08.7726117Z ++ pwd
2025-04-21T23:19:08.7728284Z + PYTHONPATH=/var/lib/jenkins/workspace/torchbench
2025-04-21T23:19:08.7728751Z + test_dynamo_benchmark torchbench 1
2025-04-21T23:19:08.7742869Z ++ pwd
2025-04-21T23:19:08.7746586Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-04-21T23:19:08.7746997Z + local suite=torchbench
2025-04-21T23:19:08.7747242Z + shift
2025-04-21T23:19:08.7747450Z + local shard_id=1
2025-04-21T23:19:08.7747697Z + shift
2025-04-21T23:19:08.7747934Z + [[ inductor_torchbench == *perf_compare* ]]
2025-04-21T23:19:08.7748280Z + [[ inductor_torchbench == *perf* ]]
2025-04-21T23:19:08.7748598Z + [[ inductor_torchbench == *cpu* ]]
2025-04-21T23:19:08.7748930Z + [[ inductor_torchbench == *aot_inductor* ]]
2025-04-21T23:19:08.7749313Z + [[ inductor_torchbench == *max_autotune_inductor* ]]
2025-04-21T23:19:08.7749812Z + test_single_dynamo_benchmark inference torchbench 1 --inference --bfloat16
2025-04-21T23:19:08.7765671Z ++ pwd
2025-04-21T23:19:08.7769120Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-04-21T23:19:08.7769684Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2025-04-21T23:19:08.7822661Z + local name=inference
2025-04-21T23:19:08.7823054Z + shift
2025-04-21T23:19:08.7823350Z + local suite=torchbench
2025-04-21T23:19:08.7823704Z + shift
2025-04-21T23:19:08.7824005Z + local shard_id=1
2025-04-21T23:19:08.7824301Z + shift
2025-04-21T23:19:08.7824595Z + partition_flags=()
2025-04-21T23:19:08.7824936Z + local partition_flags
2025-04-21T23:19:08.7825186Z + [[ -n 2 ]]
2025-04-21T23:19:08.7825405Z + [[ -n 1 ]]
2025-04-21T23:19:08.7825996Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2025-04-21T23:19:08.7826709Z + [[ inductor_torchbench == *perf_compare* ]]
2025-04-21T23:19:08.7827190Z + [[ inductor_torchbench == *perf* ]]
2025-04-21T23:19:08.7827508Z + [[ inductor_torchbench == *_avx2* ]]
2025-04-21T23:19:08.7827828Z + [[ inductor_torchbench == *_avx512* ]]
2025-04-21T23:19:08.7828987Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cuda --inference --bfloat16 --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv
2025-04-21T23:19:15.8291837Z 
2025-04-21T23:19:16.0148206Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:16.0148568Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:16.0148907Z cuda eval  llama_v2_7b_16h                    
2025-04-21T23:19:16.0154489Z Traceback (most recent call last):
2025-04-21T23:19:16.0154988Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:19:16.0155475Z     ) = runner.load_model(
2025-04-21T23:19:16.0155962Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 328, in load_model
2025-04-21T23:19:16.0156782Z     benchmark = benchmark_cls(
2025-04-21T23:19:16.0157311Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:19:16.0157894Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:19:16.0158555Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llama_v2_7b_16h/__init__.py", line 16, in __init__
2025-04-21T23:19:16.0159187Z     super().__init__(
2025-04-21T23:19:16.0159836Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 32, in __init__
2025-04-21T23:19:16.0160711Z     super().__init__(
2025-04-21T23:19:16.0161216Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 110, in __init__
2025-04-21T23:19:16.0161772Z     self._skip_by_device_name()
2025-04-21T23:19:16.0162372Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 202, in _skip_by_device_name
2025-04-21T23:19:16.0163005Z     raise NotImplementedError(
2025-04-21T23:19:16.0163550Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llama_v2_7b_16h/metadata.yaml`.
2025-04-21T23:19:16.0164015Z 
2025-04-21T23:19:16.0164106Z model_fail_to_load
2025-04-21T23:19:20.0971692Z 
2025-04-21T23:19:20.2719010Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:20.2719441Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:20.2719756Z cuda eval  llava                              
2025-04-21T23:19:20.2726845Z Traceback (most recent call last):
2025-04-21T23:19:20.2727438Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:19:20.2728030Z     ) = runner.load_model(
2025-04-21T23:19:20.2728516Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 328, in load_model
2025-04-21T23:19:20.2729132Z     benchmark = benchmark_cls(
2025-04-21T23:19:20.2729734Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:19:20.2730342Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:19:20.2731018Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 12, in __init__
2025-04-21T23:19:20.2731694Z     super().__init__(
2025-04-21T23:19:20.2732417Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 32, in __init__
2025-04-21T23:19:20.2733204Z     super().__init__(
2025-04-21T23:19:20.2733703Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 110, in __init__
2025-04-21T23:19:20.2734334Z     self._skip_by_device_name()
2025-04-21T23:19:20.2734991Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 202, in _skip_by_device_name
2025-04-21T23:19:20.2735684Z     raise NotImplementedError(
2025-04-21T23:19:20.2736180Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2025-04-21T23:19:20.2736677Z 
2025-04-21T23:19:20.2736774Z model_fail_to_load
2025-04-21T23:19:24.3691842Z 
2025-04-21T23:19:25.3638047Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:25.3638401Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:25.3644817Z cuda eval  maml                               
2025-04-21T23:19:25.3882164Z Compilation time (from dynamo_timed): 0
2025-04-21T23:19:25.3882623Z pass_due_to_skip
2025-04-21T23:19:25.3891618Z TIMING: total_wall_time:0
2025-04-21T23:19:25.3891897Z STATS: call_* op count: 0
2025-04-21T23:19:25.3892278Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2025-04-21T23:19:29.6016330Z 
2025-04-21T23:19:30.2816921Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:30.2817268Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:30.2824233Z cuda eval  maml_omniglot                      
2025-04-21T23:19:46.8053806Z Compilation time (from dynamo_timed): 14.870992299
2025-04-21T23:19:46.8054174Z pass
2025-04-21T23:19:46.8055888Z TIMING: _recursive_pre_grad_passes:0.00978 pad_mm_benchmark:0.07524 _recursive_joint_graph_passes:0.22985 _recursive_post_grad_passes:0.02914 async_compile.wait:0.61951 code_gen:1.33729 inductor_compile:1.61371 backend_compile:14.45509 entire_frame_compile:14.87099 gc:0.0002 cudagraphify.get_container:0.16279 CUDAGraphNode.record:0.15916 total_wall_time:14.87099
2025-04-21T23:19:46.8058153Z STATS: call_* op count: 14 | FakeTensorMode.__torch_dispatch__:1337 | ProxyTorchDispatchMode.__torch_dispatch__:331 | FakeTensor.__torch_dispatch__:424
2025-04-21T23:19:46.8058987Z Dynamo produced 1 graphs covering 14 ops with 0 graph breaks (0 unique)
2025-04-21T23:19:52.0587671Z 
2025-04-21T23:19:52.3811953Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:52.3812318Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:19:52.3812654Z cuda eval  microbench_unbacked_tolist_sum     
2025-04-21T23:19:57.0663549Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] Graph break from `Tensor.item()`, consider setting:
2025-04-21T23:19:57.0664580Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-21T23:19:57.0665373Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] or:
2025-04-21T23:19:57.0666142Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-21T23:19:57.0667060Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] to include these operations in the captured graph.
2025-04-21T23:19:57.0667843Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] 
2025-04-21T23:19:57.0668548Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] Graph break: from user code at:
2025-04-21T23:19:57.0669623Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in forward_pass
2025-04-21T23:19:57.0670656Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]     return mod(*inputs)
2025-04-21T23:19:57.0671867Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/microbench_unbacked_tolist_sum/__init__.py", line 18, in forward
2025-04-21T23:19:57.0673070Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0]     sum += tv[i].item()
2025-04-21T23:19:57.0673812Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] 
2025-04-21T23:19:57.0674409Z W0421 23:19:57.065000 43397 site-packages/torch/_dynamo/variables/tensor.py:928] [0/0] 
2025-04-21T23:20:00.2524700Z Compilation time (from dynamo_timed): 6.660234123
2025-04-21T23:20:00.2525090Z pass
2025-04-21T23:20:00.2525417Z TIMING: entire_frame_compile:6.66023 gc:0.00182 total_wall_time:6.66023
2025-04-21T23:20:00.2526186Z STATS: call_* op count: 0 | FakeTensor.__torch_dispatch__:2002 | FakeTensorMode.__torch_dispatch__:10006
2025-04-21T23:20:00.2526923Z Dynamo produced 0 graphs covering 0 ops with 1 graph breaks (1 unique)
2025-04-21T23:20:05.7897553Z 
2025-04-21T23:20:06.0321067Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/mnasnet1.0_top1_73.512-f206786ef8.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/mnasnet1.0_top1_73.512-f206786ef8.pth
2025-04-21T23:20:06.0492041Z 
2025-04-21T23:20:06.0492067Z 
2025-04-21T23:20:06.0971642Z   0% 0.00/16.9M [00:00<?, ?B/s]
2025-04-21T23:20:06.0971994Z 100% 16.9M/16.9M [00:00<00:00, 372MB/s]
2025-04-21T23:20:06.8540277Z 
2025-04-21T23:20:06.8540517Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:20:06.8602493Z cuda eval  mnasnet1_0                         
2025-04-21T23:20:21.3619893Z Compilation time (from dynamo_timed): 11.649890692
2025-04-21T23:20:21.3620549Z pass
2025-04-21T23:20:21.3622909Z TIMING: _recursive_pre_grad_passes:0.00793 pad_mm_benchmark:0.0621 _recursive_joint_graph_passes:0.25499 _recursive_post_grad_passes:0.17149 async_compile.wait:0.5011 code_gen:1.83194 inductor_compile:4.26513 backend_compile:8.22105 entire_frame_compile:11.64989 gc:0.00032 cudagraphify.get_container:0.24185 CUDAGraphNode.record:0.23178 total_wall_time:11.64989
2025-04-21T23:20:21.3624953Z STATS: call_* op count: 152 | FakeTensorMode.__torch_dispatch__:11232 | ProxyTorchDispatchMode.__torch_dispatch__:4024 | FakeTensor.__torch_dispatch__:6491
2025-04-21T23:20:21.3625875Z Dynamo produced 1 graphs covering 152 ops with 0 graph breaks (0 unique)
2025-04-21T23:20:26.9494192Z 
2025-04-21T23:20:27.9606045Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:20:27.9606496Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:20:27.9668620Z cuda eval  mobilenet_v2                       
2025-04-21T23:20:42.3113696Z Compilation time (from dynamo_timed): 11.672542392
2025-04-21T23:20:42.3114209Z pass
2025-04-21T23:20:42.3115653Z TIMING: _recursive_pre_grad_passes:0.00748 pad_mm_benchmark:0.00116 _recursive_joint_graph_passes:0.21307 _recursive_post_grad_passes:0.18352 async_compile.wait:0.05236 code_gen:1.39915 inductor_compile:3.9635 backend_compile:7.94885 entire_frame_compile:11.67254 gc:0.0017 cudagraphify.get_container:0.2455 CUDAGraphNode.record:0.23427 total_wall_time:11.67254
2025-04-21T23:20:42.3117440Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:11620 | FakeTensor.__torch_dispatch__:6623 | ProxyTorchDispatchMode.__torch_dispatch__:4488
2025-04-21T23:20:42.3118258Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique)
2025-04-21T23:20:47.8960683Z 
2025-04-21T23:20:48.0765061Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:20:48.0765690Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:20:48.0766323Z cuda eval  mobilenet_v2_quantized_qat         
2025-04-21T23:20:48.0771158Z Traceback (most recent call last):
2025-04-21T23:20:48.0772126Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:20:48.0773099Z     ) = runner.load_model(
2025-04-21T23:20:48.0773799Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 328, in load_model
2025-04-21T23:20:48.0774347Z     benchmark = benchmark_cls(
2025-04-21T23:20:48.0774867Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:20:48.0775429Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:20:48.0776124Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/mobilenet_v2_quantized_qat/__init__.py", line 23, in __init__
2025-04-21T23:20:48.0776919Z     raise NotImplementedError("The eval test only supports CPU.")
2025-04-21T23:20:48.0777385Z NotImplementedError: The eval test only supports CPU.
2025-04-21T23:20:48.0777650Z 
2025-04-21T23:20:48.0777750Z model_fail_to_load
2025-04-21T23:20:52.1707073Z 
2025-04-21T23:20:53.2968069Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:20:53.2968454Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:20:53.3033039Z cuda eval  mobilenet_v3_large                 
2025-04-21T23:21:10.1901382Z Compilation time (from dynamo_timed): 14.108246419
2025-04-21T23:21:10.1901928Z pass
2025-04-21T23:21:10.1905597Z TIMING: _recursive_pre_grad_passes:0.00774 pad_mm_benchmark:0.09731 _recursive_joint_graph_passes:0.30341 _recursive_post_grad_passes:0.19429 async_compile.wait:0.69957 code_gen:2.86189 inductor_compile:5.86575 backend_compile:10.07922 entire_frame_compile:14.10825 gc:0.00015 cudagraphify.get_container:0.26594 CUDAGraphNode.record:0.25406 total_wall_time:14.10825
2025-04-21T23:21:10.1908297Z STATS: call_* op count: 187 | FakeTensorMode.__torch_dispatch__:12198 | FakeTensor.__torch_dispatch__:6968 | ProxyTorchDispatchMode.__torch_dispatch__:4587
2025-04-21T23:21:10.1909542Z Dynamo produced 1 graphs covering 187 ops with 0 graph breaks (0 unique)
2025-04-21T23:21:15.8789512Z 
2025-04-21T23:21:18.0344392Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:21:18.0345040Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:21:18.0460467Z cuda eval  moco                               
2025-04-21T23:21:21.4795944Z [rank0]:W0421 23:21:21.478000 44801 site-packages/torch/_logging/_internal.py:1130] [0/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:21:31.6033602Z [rank0]:W0421 23:21:31.602000 44801 site-packages/torch/_dynamo/backends/distributed.py:106] [4/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2025-04-21T23:21:31.6283846Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2025-04-21T23:21:31.6285390Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] ┌─────────┬─────────────┬────────────────────────┐
2025-04-21T23:21:31.6286524Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2025-04-21T23:21:31.6287671Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] ├─────────┼─────────────┼────────────────────────┤
2025-04-21T23:21:31.6288788Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] │       0 │         470 │               44910720 │
2025-04-21T23:21:31.6289907Z [rank0]:W0421 23:21:31.627000 44801 site-packages/torch/_dynamo/backends/distributed.py:123] [4/0_1] └─────────┴─────────────┴────────────────────────┘
2025-04-21T23:21:50.8565156Z skipping cudagraphs due to mutated inputs (161 instances). Found from : 
2025-04-21T23:21:50.8566010Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 139, in forward
2025-04-21T23:21:50.8566723Z     self._momentum_update_key_encoder()  # update the key encoder
2025-04-21T23:21:50.8567420Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-04-21T23:21:50.8568023Z     return func(*args, **kwargs)
2025-04-21T23:21:50.8568715Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 59, in _momentum_update_key_encoder
2025-04-21T23:21:50.8569499Z     param_k.mul_(self.m).add_(param_q.mul(1.0 - self.m))
2025-04-21T23:21:50.8569757Z 
2025-04-21T23:21:53.6784910Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] Graph break from `Tensor.item()`, consider setting:
2025-04-21T23:21:53.6785981Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-21T23:21:53.6786862Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] or:
2025-04-21T23:21:53.6787670Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-21T23:21:53.6788644Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] to include these operations in the captured graph.
2025-04-21T23:21:53.6789465Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] 
2025-04-21T23:21:53.6790244Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] Graph break: from user code at:
2025-04-21T23:21:53.6791578Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 167, in torch_dynamo_resume_in_forward_at_142
2025-04-21T23:21:53.6793415Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]     self._dequeue_and_enqueue(k)
2025-04-21T23:21:53.6794611Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-04-21T23:21:53.6795797Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]     return func(*args, **kwargs)
2025-04-21T23:21:53.6797223Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 68, in _dequeue_and_enqueue
2025-04-21T23:21:53.6798480Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0]     ptr = int(self.queue_ptr)
2025-04-21T23:21:53.6799284Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] 
2025-04-21T23:21:53.6799926Z [rank0]:W0421 23:21:53.677000 44801 site-packages/torch/_dynamo/variables/tensor.py:928] [7/0] 
2025-04-21T23:22:08.4656209Z Compilation time (from dynamo_timed): 43.742235638
2025-04-21T23:22:08.4656613Z pass
2025-04-21T23:22:08.4680160Z TIMING: entire_frame_compile:43.74224 gc:0.00681 _recursive_pre_grad_passes:0.10594 _recursive_joint_graph_passes:0.4518 _recursive_post_grad_passes:0.48136 async_compile.wait:2.52277 code_gen:8.08892 inductor_compile:15.393 pad_mm_benchmark:0.12121 backend_compile:25.9951 cudagraphify.get_container:0.37486 CUDAGraphNode.record:2.5393 total_wall_time:43.74224
2025-04-21T23:22:08.4681955Z STATS: call_* op count: 877 | FakeTensorMode.__torch_dispatch__:32534 | FakeTensor.__torch_dispatch__:19405 | ProxyTorchDispatchMode.__torch_dispatch__:11928
2025-04-21T23:22:08.4682796Z Dynamo produced 7 graphs covering 877 ops with 7 graph breaks (4 unique)
2025-04-21T23:22:10.2350511Z [rank0]:[W421 23:22:10.222165425 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-04-21T23:22:15.9824063Z 
2025-04-21T23:22:16.1578334Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:22:16.1578656Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:22:16.1578968Z cuda eval  moondream                          
2025-04-21T23:22:16.1585897Z Traceback (most recent call last):
2025-04-21T23:22:16.1586418Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:22:16.1586894Z     ) = runner.load_model(
2025-04-21T23:22:16.1587373Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 328, in load_model
2025-04-21T23:22:16.1587897Z     benchmark = benchmark_cls(
2025-04-21T23:22:16.1588418Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:22:16.1588996Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:22:16.1589628Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moondream/__init__.py", line 12, in __init__
2025-04-21T23:22:16.1590237Z     super().__init__(
2025-04-21T23:22:16.1590879Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 32, in __init__
2025-04-21T23:22:16.1591579Z     super().__init__(
2025-04-21T23:22:16.1592068Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 110, in __init__
2025-04-21T23:22:16.1592634Z     self._skip_by_device_name()
2025-04-21T23:22:16.1593294Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 202, in _skip_by_device_name
2025-04-21T23:22:16.1593920Z     raise NotImplementedError(
2025-04-21T23:22:16.1594429Z NotImplementedError: The current device NVIDIA A10G is skipped by its `moondream/metadata.yaml`.
2025-04-21T23:22:16.1595102Z 
2025-04-21T23:22:16.1595191Z model_fail_to_load
2025-04-21T23:22:20.2575487Z 
2025-04-21T23:22:22.1635960Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2025-04-21T23:22:22.4146949Z num decayed parameter tensors: 50, with 124,354,560 parameters
2025-04-21T23:22:22.4147463Z num non-decayed parameter tensors: 98, with 121,344 parameters
2025-04-21T23:22:22.4150451Z using fused AdamW: True
2025-04-21T23:22:22.8549817Z 
2025-04-21T23:22:22.8550000Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:22:22.8590827Z cuda eval  nanogpt                            
2025-04-21T23:22:34.0813112Z Compilation time (from dynamo_timed): 9.245727747
2025-04-21T23:22:34.0813495Z pass
2025-04-21T23:22:34.0834220Z TIMING: _recursive_pre_grad_passes:0.00803 pad_mm_benchmark:0.12211 _recursive_joint_graph_passes:0.47564 _recursive_post_grad_passes:0.30142 async_compile.wait:0.95715 code_gen:2.38804 inductor_compile:4.14276 backend_compile:7.36144 entire_frame_compile:9.24573 gc:0.00066 cudagraphify.get_container:0.2183 CUDAGraphNode.record:0.21319 total_wall_time:9.24573
2025-04-21T23:22:34.0836037Z STATS: call_* op count: 393 | FakeTensorMode.__torch_dispatch__:12351 | FakeTensor.__torch_dispatch__:4673 | ProxyTorchDispatchMode.__torch_dispatch__:4667
2025-04-21T23:22:34.0836856Z Dynamo produced 1 graphs covering 393 ops with 0 graph breaks (0 unique)
2025-04-21T23:22:39.5136120Z 
2025-04-21T23:22:42.0287575Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:22:42.0288090Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:22:42.0291277Z cuda eval  nvidia_deeprecommender             
2025-04-21T23:22:46.0708053Z Compilation time (from dynamo_timed): 2.419919609
2025-04-21T23:22:46.0708547Z pass
2025-04-21T23:22:46.0749185Z TIMING: _recursive_pre_grad_passes:0.00323 pad_mm_benchmark:0.45217 _recursive_joint_graph_passes:0.59686 _recursive_post_grad_passes:0.06394 async_compile.precompile:0.31117 async_compile.wait:0.00019 code_gen:0.44526 inductor_compile:0.68895 backend_compile:2.16661 entire_frame_compile:2.41992 gc:0.00023 cudagraphify.get_container:0.16298 CUDAGraphNode.record:0.15655 total_wall_time:2.41992
2025-04-21T23:22:46.0751338Z STATS: call_* op count: 13 | FakeTensorMode.__torch_dispatch__:1167 | ProxyTorchDispatchMode.__torch_dispatch__:356 | FakeTensor.__torch_dispatch__:411
2025-04-21T23:22:46.0752137Z Dynamo produced 1 graphs covering 13 ops with 0 graph breaks (0 unique)
2025-04-21T23:22:51.2296764Z 
2025-04-21T23:22:52.3380044Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:22:52.3380413Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:22:52.3398385Z cuda eval  opacus_cifar10                     
2025-04-21T23:23:00.9512383Z Compilation time (from dynamo_timed): 6.871028015
2025-04-21T23:23:00.9512893Z pass
2025-04-21T23:23:00.9514625Z TIMING: _recursive_pre_grad_passes:0.00428 pad_mm_benchmark:0.06509 _recursive_joint_graph_passes:0.42175 _recursive_post_grad_passes:0.06682 async_compile.wait:0.45467 code_gen:2.26426 inductor_compile:3.32364 backend_compile:5.3066 entire_frame_compile:6.87103 gc:0.00047 cudagraphify.get_container:0.20151 CUDAGraphNode.record:0.19503 total_wall_time:6.87103
2025-04-21T23:23:00.9516437Z STATS: call_* op count: 69 | FakeTensorMode.__torch_dispatch__:3795 | ProxyTorchDispatchMode.__torch_dispatch__:1401 | FakeTensor.__torch_dispatch__:2271
2025-04-21T23:23:00.9517264Z Dynamo produced 1 graphs covering 69 ops with 0 graph breaks (0 unique)
2025-04-21T23:23:06.3918362Z 
2025-04-21T23:23:07.1725115Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:07.1725959Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:07.1786155Z cuda eval  phlippe_densenet                   
2025-04-21T23:23:24.4711702Z Compilation time (from dynamo_timed): 15.346949459
2025-04-21T23:23:24.4712115Z pass
2025-04-21T23:23:24.4713501Z TIMING: _recursive_pre_grad_passes:0.00814 pad_mm_benchmark:0.06056 _recursive_joint_graph_passes:0.25829 _recursive_post_grad_passes:0.1806 async_compile.wait:1.58977 code_gen:4.667 inductor_compile:7.54839 backend_compile:11.37567 entire_frame_compile:15.34695 gc:0.00012 cudagraphify.get_container:0.26009 CUDAGraphNode.record:0.24513 total_wall_time:15.34695
2025-04-21T23:23:24.4715857Z STATS: call_* op count: 186 | FakeTensorMode.__torch_dispatch__:12061 | ProxyTorchDispatchMode.__torch_dispatch__:4308 | FakeTensor.__torch_dispatch__:6777
2025-04-21T23:23:24.4716715Z Dynamo produced 1 graphs covering 186 ops with 0 graph breaks (0 unique)
2025-04-21T23:23:30.0944963Z 
2025-04-21T23:23:30.7772884Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:30.7773228Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:30.7798606Z cuda eval  phlippe_resnet                     
2025-04-21T23:23:37.5758825Z Compilation time (from dynamo_timed): 5.168269484
2025-04-21T23:23:37.5759377Z pass
2025-04-21T23:23:37.5761400Z TIMING: _recursive_pre_grad_passes:0.00492 pad_mm_benchmark:0.06434 _recursive_joint_graph_passes:0.37969 _recursive_post_grad_passes:0.07673 async_compile.wait:0.19248 code_gen:0.92941 inductor_compile:1.85591 backend_compile:3.83847 entire_frame_compile:5.16827 gc:0.00014 cudagraphify.get_container:0.18434 CUDAGraphNode.record:0.18051 total_wall_time:5.16827
2025-04-21T23:23:37.5763694Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:4493 | ProxyTorchDispatchMode.__torch_dispatch__:1636 | FakeTensor.__torch_dispatch__:2514
2025-04-21T23:23:37.5764512Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique)
2025-04-21T23:23:42.9396022Z 
2025-04-21T23:23:43.3337908Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:43.3338355Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:43.3339116Z cuda eval  pyhpc_equation_of_state            
2025-04-21T23:23:48.9995396Z Compilation time (from dynamo_timed): 4.202075966
2025-04-21T23:23:48.9995767Z pass
2025-04-21T23:23:48.9996995Z TIMING: _recursive_pre_grad_passes:0.00646 _recursive_joint_graph_passes:0.1472 _recursive_post_grad_passes:0.23609 async_compile.wait:0.28545 code_gen:1.45665 inductor_compile:2.38052 backend_compile:3.83826 entire_frame_compile:4.20208 gc:0.00049 cudagraphify.get_container:0.17423 CUDAGraphNode.record:0.16853 total_wall_time:4.20208
2025-04-21T23:23:48.9998742Z STATS: call_* op count: 366 | FakeTensorMode.__torch_dispatch__:4857 | ProxyTorchDispatchMode.__torch_dispatch__:1889 | FakeTensor.__torch_dispatch__:3037
2025-04-21T23:23:48.9999564Z Dynamo produced 1 graphs covering 366 ops with 0 graph breaks (0 unique)
2025-04-21T23:23:54.2123056Z 
2025-04-21T23:23:54.5627238Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:54.5627584Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:23:54.5631772Z cuda eval  pyhpc_isoneutral_mixing            
2025-04-21T23:24:05.9642452Z skipping cudagraphs due to mutated inputs (7 instances). Found from : 
2025-04-21T23:24:05.9643229Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in forward_pass
2025-04-21T23:24:05.9643835Z     return mod(*inputs)
2025-04-21T23:24:05.9644490Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/__init__.py", line 100, in forward
2025-04-21T23:24:05.9645362Z     return isoneutral_pytorch.isoneutral_diffusion_pre(
2025-04-21T23:24:05.9646541Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/pyhpc_isoneutral_mixing/isoneutral_pytorch.py", line 164, in isoneutral_diffusion_pre
2025-04-21T23:24:05.9647434Z     Ai_ez[1:-2, 2:-2, ki:, ip, kr] = taper * sxe * maskU[1:-2, 2:-2, ki:]
2025-04-21T23:24:05.9647701Z 
2025-04-21T23:24:05.9918889Z Compilation time (from dynamo_timed): 10.096416754
2025-04-21T23:24:05.9919268Z pass
2025-04-21T23:24:05.9920435Z TIMING: _recursive_pre_grad_passes:0.01092 _recursive_joint_graph_passes:0.25919 _recursive_post_grad_passes:0.9268 async_compile.wait:0.13284 code_gen:0.94385 inductor_compile:4.17055 backend_compile:8.89584 entire_frame_compile:10.09642 gc:0.00011 cudagraphify.get_container:0.24279 total_wall_time:10.09642
2025-04-21T23:24:05.9922423Z STATS: call_* op count: 744 | FakeTensorMode.__torch_dispatch__:21049 | FakeTensor.__torch_dispatch__:7452 | ProxyTorchDispatchMode.__torch_dispatch__:7055
2025-04-21T23:24:05.9923688Z Dynamo produced 1 graphs covering 744 ops with 0 graph breaks (0 unique)
2025-04-21T23:24:11.4764505Z 
2025-04-21T23:24:12.2474216Z loading model: 0it [00:00, ?it/s]WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2025-04-21T23:24:12.4083123Z 
2025-04-21T23:24:12.4083390Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:24:12.4083967Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2025-04-21T23:24:12.4084571Z cuda eval  pyhpc_turbulent_kinetic_energy     
2025-04-21T23:24:12.4539079Z WARNING:common:Model pyhpc_turbulent_kinetic_energy does not support bfloat16, running with amp instead
2025-04-21T23:24:32.7417636Z Compilation time (from dynamo_timed): 18.55390463
2025-04-21T23:24:32.7418006Z pass
2025-04-21T23:24:32.7479920Z TIMING: _recursive_pre_grad_passes:0.01335 _recursive_joint_graph_passes:0.24428 _recursive_post_grad_passes:0.57881 async_compile.wait:1.00096 code_gen:8.2823 inductor_compile:11.96952 backend_compile:17.16659 entire_frame_compile:18.5539 gc:0.00042 cudagraphify.get_container:0.3153 CUDAGraphNode.record:0.28343 total_wall_time:18.5539
2025-04-21T23:24:32.7482003Z STATS: call_* op count: 996 | FakeTensorMode.__torch_dispatch__:21539 | FakeTensor.__torch_dispatch__:7220 | ProxyTorchDispatchMode.__torch_dispatch__:7370
2025-04-21T23:24:32.7482823Z Dynamo produced 1 graphs covering 996 ops with 0 graph breaks (0 unique)
2025-04-21T23:24:38.5027380Z 
2025-04-21T23:24:39.3220093Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:24:39.3220518Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:24:39.3232582Z cuda eval  pytorch_CycleGAN_and_pix2pix       
2025-04-21T23:24:53.7119362Z Compilation time (from dynamo_timed): 7.079441624
2025-04-21T23:24:53.7119820Z pass
2025-04-21T23:24:53.7128661Z TIMING: _recursive_pre_grad_passes:0.00451 _recursive_joint_graph_passes:0.34484 _recursive_post_grad_passes:0.08874 async_compile.wait:0.73153 code_gen:2.61238 inductor_compile:3.91949 backend_compile:5.90429 entire_frame_compile:7.07944 gc:0.00013 cudagraphify.get_container:0.20871 CachingAutotuner.benchmark_all_configs:0.33685 CUDAGraphNode.record:0.20151 total_wall_time:7.07944
2025-04-21T23:24:53.7130648Z STATS: call_* op count: 91 | FakeTensorMode.__torch_dispatch__:4976 | ProxyTorchDispatchMode.__torch_dispatch__:2004 | FakeTensor.__torch_dispatch__:2700
2025-04-21T23:24:53.7131473Z Dynamo produced 1 graphs covering 91 ops with 0 graph breaks (0 unique)
2025-04-21T23:24:59.1169635Z 
2025-04-21T23:25:00.3021067Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:25:00.3022282Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:25:00.3040360Z cuda eval  pytorch_stargan                    
2025-04-21T23:25:24.3954093Z Compilation time (from dynamo_timed): 6.774750736
2025-04-21T23:25:24.3954585Z pass
2025-04-21T23:25:24.3969623Z TIMING: _recursive_pre_grad_passes:0.00468 _recursive_joint_graph_passes:0.39055 _recursive_post_grad_passes:0.07875 async_compile.wait:0.21023 code_gen:1.78554 inductor_compile:2.96609 backend_compile:5.32406 entire_frame_compile:6.77475 gc:0.00078 cudagraphify.get_container:0.21028 CUDAGraphNode.record:0.208 total_wall_time:6.77475
2025-04-21T23:25:24.3971813Z STATS: call_* op count: 56 | FakeTensorMode.__torch_dispatch__:6870 | FakeTensor.__torch_dispatch__:2836 | ProxyTorchDispatchMode.__torch_dispatch__:2542
2025-04-21T23:25:24.3972631Z Dynamo produced 1 graphs covering 56 ops with 0 graph breaks (0 unique)
2025-04-21T23:25:29.8955098Z 
2025-04-21T23:25:30.6821612Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:25:30.6821995Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:25:30.6843659Z cuda eval  pytorch_unet                       
2025-04-21T23:25:54.7395708Z Compilation time (from dynamo_timed): 8.265933293
2025-04-21T23:25:54.7396071Z pass
2025-04-21T23:25:54.7427541Z TIMING: _recursive_pre_grad_passes:0.0052 _recursive_joint_graph_passes:0.32843 _recursive_post_grad_passes:0.09547 async_compile.wait:0.32082 code_gen:2.83614 inductor_compile:4.13789 backend_compile:6.47404 entire_frame_compile:8.26593 gc:0.00094 cudagraphify.get_container:0.19909 CachingAutotuner.benchmark_all_configs:1.68797 CUDAGraphNode.record:0.2001 total_wall_time:8.26593
2025-04-21T23:25:54.7429782Z STATS: call_* op count: 71 | FakeTensorMode.__torch_dispatch__:6681 | ProxyTorchDispatchMode.__torch_dispatch__:2438 | FakeTensor.__torch_dispatch__:3314
2025-04-21T23:25:54.7430595Z Dynamo produced 1 graphs covering 71 ops with 0 graph breaks (0 unique)
2025-04-21T23:26:00.1840242Z 
2025-04-21T23:26:00.9998690Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet152-394f9c45.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet152-394f9c45.pth
2025-04-21T23:26:01.0164123Z 
2025-04-21T23:26:01.0164279Z 
2025-04-21T23:26:01.1167494Z   0% 0.00/230M [00:00<?, ?B/s]
2025-04-21T23:26:01.1167712Z 
2025-04-21T23:26:01.2170489Z  18% 41.2M/230M [00:00<00:00, 432MB/s]
2025-04-21T23:26:01.2170741Z 
2025-04-21T23:26:01.3170143Z  37% 84.5M/230M [00:00<00:00, 444MB/s]
2025-04-21T23:26:01.3170373Z 
2025-04-21T23:26:01.4171602Z  55% 128M/230M [00:00<00:00, 448MB/s] 
2025-04-21T23:26:01.4172936Z 
2025-04-21T23:26:01.5171533Z  74% 171M/230M [00:00<00:00, 450MB/s]
2025-04-21T23:26:01.5171831Z 
2025-04-21T23:26:01.5561996Z  93% 214M/230M [00:00<00:00, 450MB/s]
2025-04-21T23:26:01.5562324Z 100% 230M/230M [00:00<00:00, 448MB/s]
2025-04-21T23:26:02.6042582Z 
2025-04-21T23:26:02.6042960Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:26:02.6220237Z cuda eval  resnet152                          
2025-04-21T23:26:36.2797586Z Compilation time (from dynamo_timed): 29.015509379
2025-04-21T23:26:36.2798091Z pass
2025-04-21T23:26:36.2829961Z TIMING: _recursive_pre_grad_passes:0.01676 pad_mm_benchmark:0.06236 _recursive_joint_graph_passes:0.63148 _recursive_post_grad_passes:0.50844 async_compile.wait:0.04919 code_gen:3.30307 inductor_compile:9.91272 backend_compile:19.87123 entire_frame_compile:29.01551 gc:0.00084 cudagraphify.get_container:0.45837 CUDAGraphNode.record:0.40844 total_wall_time:29.01551
2025-04-21T23:26:36.2831834Z STATS: call_* op count: 515 | FakeTensorMode.__torch_dispatch__:30608 | ProxyTorchDispatchMode.__torch_dispatch__:11943 | FakeTensor.__torch_dispatch__:19829
2025-04-21T23:26:36.2832681Z Dynamo produced 1 graphs covering 515 ops with 0 graph breaks (0 unique)
2025-04-21T23:26:42.5598682Z 
2025-04-21T23:26:42.8651895Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet18-f37072fd.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet18-f37072fd.pth
2025-04-21T23:26:42.8795238Z 
2025-04-21T23:26:42.8795359Z 
2025-04-21T23:26:42.9796925Z   0% 0.00/44.7M [00:00<?, ?B/s]
2025-04-21T23:26:42.9797133Z 
2025-04-21T23:26:42.9854694Z  95% 42.5M/44.7M [00:00<00:00, 446MB/s]
2025-04-21T23:26:42.9855000Z 100% 44.7M/44.7M [00:00<00:00, 442MB/s]
2025-04-21T23:26:43.7360937Z 
2025-04-21T23:26:43.7361442Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:26:43.7385508Z cuda eval  resnet18                           
2025-04-21T23:26:51.7473639Z Compilation time (from dynamo_timed): 5.488369025
2025-04-21T23:26:51.7474008Z pass
2025-04-21T23:26:51.7482876Z TIMING: _recursive_pre_grad_passes:0.00513 pad_mm_benchmark:0.09708 _recursive_joint_graph_passes:0.41946 _recursive_post_grad_passes:0.0782 async_compile.wait:0.12128 code_gen:1.1371 inductor_compile:2.08044 backend_compile:4.15401 entire_frame_compile:5.48837 gc:0.00038 cudagraphify.get_container:0.19361 CUDAGraphNode.record:0.1879 total_wall_time:5.48837
2025-04-21T23:26:51.7484699Z STATS: call_* op count: 69 | FakeTensorMode.__torch_dispatch__:4547 | ProxyTorchDispatchMode.__torch_dispatch__:1601 | FakeTensor.__torch_dispatch__:2591
2025-04-21T23:26:51.7485549Z Dynamo produced 1 graphs covering 69 ops with 0 graph breaks (0 unique)
2025-04-21T23:26:57.1455233Z 
2025-04-21T23:26:57.5903231Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnet50-0676ba61.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth
2025-04-21T23:26:57.6056754Z 
2025-04-21T23:26:57.6057062Z 
2025-04-21T23:26:57.7060148Z   0% 0.00/97.8M [00:00<?, ?B/s]
2025-04-21T23:26:57.7060372Z 
2025-04-21T23:26:57.8061384Z  33% 32.1M/97.8M [00:00<00:00, 336MB/s]
2025-04-21T23:26:57.8061626Z 
2025-04-21T23:26:57.8994223Z  68% 66.1M/97.8M [00:00<00:00, 348MB/s]
2025-04-21T23:26:57.8994556Z 100% 97.8M/97.8M [00:00<00:00, 349MB/s]
2025-04-21T23:26:58.7867875Z 
2025-04-21T23:26:58.7868045Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:26:58.7934737Z cuda eval  resnet50                           
2025-04-21T23:27:13.7418150Z Compilation time (from dynamo_timed): 11.26729966
2025-04-21T23:27:13.7418517Z pass
2025-04-21T23:27:13.7435940Z TIMING: _recursive_pre_grad_passes:0.00784 pad_mm_benchmark:0.00115 _recursive_joint_graph_passes:0.19999 _recursive_post_grad_passes:0.18291 async_compile.wait:0.04813 code_gen:1.50999 inductor_compile:4.01272 backend_compile:8.02473 entire_frame_compile:11.2673 gc:0.00146 cudagraphify.get_container:0.24323 CUDAGraphNode.record:0.2327 total_wall_time:11.2673
2025-04-21T23:27:13.7437756Z STATS: call_* op count: 175 | FakeTensorMode.__torch_dispatch__:11126 | ProxyTorchDispatchMode.__torch_dispatch__:4123 | FakeTensor.__torch_dispatch__:6773
2025-04-21T23:27:13.7438585Z Dynamo produced 1 graphs covering 175 ops with 0 graph breaks (0 unique)
2025-04-21T23:27:19.3560300Z 
2025-04-21T23:27:19.5366300Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:27:19.5366637Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:27:19.5366953Z cuda eval  resnet50_quantized_qat             
2025-04-21T23:27:19.5372826Z Traceback (most recent call last):
2025-04-21T23:27:19.5373474Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:27:19.5374021Z     ) = runner.load_model(
2025-04-21T23:27:19.5374491Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 328, in load_model
2025-04-21T23:27:19.5375026Z     benchmark = benchmark_cls(
2025-04-21T23:27:19.5375543Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:27:19.5376109Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:27:19.5376792Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/resnet50_quantized_qat/__init__.py", line 22, in __init__
2025-04-21T23:27:19.5377561Z     raise NotImplementedError("The eval test only supports CPU.")
2025-04-21T23:27:19.5378025Z NotImplementedError: The eval test only supports CPU.
2025-04-21T23:27:19.5378289Z 
2025-04-21T23:27:19.5378380Z model_fail_to_load
2025-04-21T23:27:23.6178015Z 
2025-04-21T23:27:24.0627445Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/resnext50_32x4d-7cdf4587.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/resnext50_32x4d-7cdf4587.pth
2025-04-21T23:27:24.0781568Z 
2025-04-21T23:27:24.0781573Z 
2025-04-21T23:27:24.1787100Z   0% 0.00/95.8M [00:00<?, ?B/s]
2025-04-21T23:27:24.1787382Z 
2025-04-21T23:27:24.2809404Z  25% 24.0M/95.8M [00:00<00:00, 251MB/s]
2025-04-21T23:27:24.2809643Z 
2025-04-21T23:27:24.3819071Z  50% 48.0M/95.8M [00:00<00:00, 248MB/s]
2025-04-21T23:27:24.3819423Z 
2025-04-21T23:27:24.4535573Z  75% 71.8M/95.8M [00:00<00:00, 248MB/s]
2025-04-21T23:27:24.4535918Z 100% 95.8M/95.8M [00:00<00:00, 268MB/s]
2025-04-21T23:27:25.4102937Z 
2025-04-21T23:27:25.4103445Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:27:25.4165018Z cuda eval  resnext50_32x4d                    
2025-04-21T23:27:40.8489411Z Compilation time (from dynamo_timed): 11.286199676
2025-04-21T23:27:40.8489914Z pass
2025-04-21T23:27:40.8511245Z TIMING: _recursive_pre_grad_passes:0.00779 pad_mm_benchmark:0.00117 _recursive_joint_graph_passes:0.20104 _recursive_post_grad_passes:0.17862 async_compile.wait:0.04915 code_gen:1.5354 inductor_compile:4.06382 backend_compile:8.07449 entire_frame_compile:11.2862 gc:0.0003 cudagraphify.get_container:0.25356 CUDAGraphNode.record:0.23452 total_wall_time:11.2862
2025-04-21T23:27:40.8514430Z STATS: call_* op count: 175 | FakeTensorMode.__torch_dispatch__:11051 | ProxyTorchDispatchMode.__torch_dispatch__:4123 | FakeTensor.__torch_dispatch__:6773
2025-04-21T23:27:40.8515707Z Dynamo produced 1 graphs covering 175 ops with 0 graph breaks (0 unique)
2025-04-21T23:27:46.4760871Z 
2025-04-21T23:27:54.0155622Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:27:54.0155958Z loading model: 0it [00:07, ?it/s]
2025-04-21T23:27:54.0270295Z cuda eval  sam                                
2025-04-21T23:29:18.7724218Z Compilation time (from dynamo_timed): 50.425906225
2025-04-21T23:29:18.7724568Z pass
2025-04-21T23:29:18.8099328Z TIMING: _recursive_pre_grad_passes:0.029 pad_mm_benchmark:1.08888 _recursive_joint_graph_passes:2.80959 _recursive_post_grad_passes:3.22632 async_compile.wait:2.31114 code_gen:11.83717 inductor_compile:24.10994 backend_compile:41.11511 entire_frame_compile:50.42591 gc:0.00057 cudagraphify.get_container:0.74541 CachingAutotuner.benchmark_all_configs:2.20397 CUDAGraphNode.record:0.67193 total_wall_time:50.42591
2025-04-21T23:29:18.8101385Z STATS: call_* op count: 2285 | FakeTensorMode.__torch_dispatch__:87233 | FakeTensor.__torch_dispatch__:25847 | ProxyTorchDispatchMode.__torch_dispatch__:30217
2025-04-21T23:29:18.8102210Z Dynamo produced 1 graphs covering 2285 ops with 0 graph breaks (0 unique)
2025-04-21T23:29:26.1326007Z 
2025-04-21T23:30:20.7720489Z loading model: 0it [00:00, ?it/s]E0421 23:30:20.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:20.7722466Z E0421 23:30:20.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:20.7723963Z E0421 23:30:20.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:20.8852282Z E0421 23:30:20.884000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:20.8853620Z E0421 23:30:20.884000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:20.8854898Z E0421 23:30:20.884000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:21.0750294Z E0421 23:30:21.074000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:21.0751852Z E0421 23:30:21.074000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:21.0753110Z E0421 23:30:21.074000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:21.0757784Z AUTOTUNE mm(4096x1280, 1280x5120)
2025-04-21T23:30:21.0758171Z strides: [1280, 1], [1, 1280]
2025-04-21T23:30:21.0758510Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:21.0759234Z   triton_mm_87 0.8028 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:21.0760330Z   triton_mm_91 0.8028 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:21.0761412Z   triton_mm_93 0.8038 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:21.0762474Z   triton_mm_90 0.8100 ms 99.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:21.0763901Z   triton_mm_86 0.8110 ms 99.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:21.0764956Z   triton_mm_83 0.8161 ms 98.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:21.0766209Z   triton_mm_84 0.8315 ms 96.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:21.0766884Z   mm 0.8346 ms 96.2% 
2025-04-21T23:30:21.0767522Z   triton_mm_88 0.8366 ms 96.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:21.0768585Z   triton_mm_82 0.8376 ms 95.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:21.0769516Z SingleProcess AUTOTUNE benchmarking takes 0.8549 seconds and 0.0004 seconds precompiling for 20 choices
2025-04-21T23:30:22.9178898Z E0421 23:30:22.917000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:22.9180307Z E0421 23:30:22.917000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:22.9181620Z E0421 23:30:22.917000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:23.0323389Z E0421 23:30:23.031000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:23.0326105Z E0421 23:30:23.031000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:23.0327565Z E0421 23:30:23.031000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:23.2205682Z E0421 23:30:23.219000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:23.2207012Z E0421 23:30:23.219000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:23.2208277Z E0421 23:30:23.219000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:23.2213482Z AUTOTUNE mm(4096x5120, 5120x1280)
2025-04-21T23:30:23.2222579Z strides: [5120, 1], [1, 5120]
2025-04-21T23:30:23.2222997Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:23.2223380Z   mm 0.7905 ms 100.0% 
2025-04-21T23:30:23.2224123Z   triton_mm_3476 0.8018 ms 98.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:23.2225404Z   triton_mm_3469 0.8059 ms 98.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:23.2226665Z   triton_mm_3473 0.8131 ms 97.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:23.2227742Z   triton_mm_3470 0.8223 ms 96.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:23.2229080Z   triton_mm_3474 0.8264 ms 95.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:23.2230154Z   triton_mm_3471 0.8284 ms 95.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:23.2231225Z   triton_mm_3467 0.8489 ms 93.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:23.2232471Z   triton_mm_3466 0.8550 ms 92.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:23.2233617Z   triton_mm_3477 0.8796 ms 89.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:23.2234563Z SingleProcess AUTOTUNE benchmarking takes 0.6480 seconds and 0.0002 seconds precompiling for 20 choices
2025-04-21T23:30:25.4243710Z E0421 23:30:25.423000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:25.4245073Z E0421 23:30:25.423000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:25.4246457Z E0421 23:30:25.423000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:25.5223440Z E0421 23:30:25.521000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:25.6871876Z E0421 23:30:25.521000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:25.6873219Z E0421 23:30:25.521000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:25.6874091Z E0421 23:30:25.686000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:25.6875421Z E0421 23:30:25.686000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:25.6876676Z E0421 23:30:25.686000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:25.6879136Z AUTOTUNE mm(4096x1280, 1280x1280)
2025-04-21T23:30:25.6879583Z strides: [1280, 1], [1, 1280]
2025-04-21T23:30:25.6879993Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:25.6880806Z   triton_mm_828 0.2048 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:25.6881900Z   triton_mm_832 0.2058 ms 99.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:25.6882972Z   triton_mm_824 0.2079 ms 98.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:25.6884034Z   triton_mm_827 0.2079 ms 98.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:25.6885103Z   triton_mm_834 0.2079 ms 98.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:25.6886488Z   triton_mm_831 0.2099 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:25.6887550Z   triton_mm_825 0.2130 ms 96.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:25.6888216Z   mm 0.2140 ms 95.7% 
2025-04-21T23:30:25.6888987Z   triton_mm_829 0.2171 ms 94.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:25.6890048Z   triton_mm_823 0.2202 ms 93.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:25.6890982Z SingleProcess AUTOTUNE benchmarking takes 0.5755 seconds and 0.0003 seconds precompiling for 20 choices
2025-04-21T23:30:27.3562743Z AUTOTUNE convolution(1x3x1024x1024, 1280x3x16x16)
2025-04-21T23:30:27.3563167Z strides: [3145728, 1048576, 1024, 1], [768, 256, 16, 1]
2025-04-21T23:30:27.3563519Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:27.3563818Z   convolution 0.4270 ms 100.0% 
2025-04-21T23:30:27.3564657Z   triton_convolution2d_3 0.9738 ms 43.8% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:27.3566044Z   triton_convolution2d_6 1.0578 ms 40.4% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:27.3567395Z   triton_convolution2d_1 1.1407 ms 37.4% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:27.3568818Z   triton_convolution2d_0 1.2114 ms 35.2% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:27.3570161Z   triton_convolution2d_4 1.5227 ms 28.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:27.3571517Z   triton_convolution2d_5 1.6241 ms 26.3% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:27.3572863Z   triton_convolution2d_2 3.8339 ms 11.1% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=16, KERNEL_W=16, PADDING_H=0, PADDING_W=0, STRIDE_H=16, STRIDE_W=16, UNROLL=False, num_stages=1, num_warps=8
2025-04-21T23:30:27.3573928Z SingleProcess AUTOTUNE benchmarking takes 0.4138 seconds and 0.0002 seconds precompiling for 8 choices
2025-04-21T23:30:27.6957419Z E0421 23:30:27.694000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:27.6959294Z E0421 23:30:27.694000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:27.6960583Z E0421 23:30:27.694000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:27.8072808Z E0421 23:30:27.806000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:27.8074715Z E0421 23:30:27.806000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:27.8075984Z E0421 23:30:27.806000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:27.9956650Z E0421 23:30:27.994000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:27.9958248Z E0421 23:30:27.994000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:27.9959507Z E0421 23:30:27.994000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:27.9965182Z AUTOTUNE mm(4904x1280, 1280x3840)
2025-04-21T23:30:27.9965572Z strides: [1280, 1], [1, 1280]
2025-04-21T23:30:27.9965914Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:27.9966610Z   triton_mm_17 0.7281 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:27.9967683Z   triton_mm_16 0.7291 ms 99.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:27.9968739Z   triton_mm_13 0.7322 ms 99.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:27.9969776Z   triton_mm_14 0.7516 ms 96.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:27.9970825Z   triton_mm_21 0.7516 ms 96.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:27.9971870Z   triton_mm_23 0.7526 ms 96.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:27.9972915Z   triton_mm_20 0.7578 ms 96.1% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:27.9973963Z   triton_mm_18 0.7608 ms 95.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:27.9975004Z   triton_mm_12 0.7793 ms 93.4% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:27.9975664Z   mm 0.8489 ms 85.8% 
2025-04-21T23:30:27.9976155Z SingleProcess AUTOTUNE benchmarking takes 0.6390 seconds and 0.0002 seconds precompiling for 20 choices
2025-04-21T23:30:28.2266390Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2025-04-21T23:30:28.2267278Z strides: [448000, 80, 1], [1152, 1, 80]
2025-04-21T23:30:28.2268108Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:28.2269160Z   triton_bmm_38 0.0410 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:28.2270275Z   triton_bmm_27 0.0420 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2
2025-04-21T23:30:28.2271352Z   triton_bmm_39 0.0420 ms 97.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=8
2025-04-21T23:30:28.2272677Z   triton_bmm_29 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:28.2273822Z   triton_bmm_30 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:28.2274900Z   triton_bmm_33 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.2276111Z   triton_bmm_34 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.2277190Z   triton_bmm_36 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4
2025-04-21T23:30:28.2278275Z   triton_bmm_37 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.2279339Z   triton_bmm_40 0.0430 ms 95.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.2280275Z SingleProcess AUTOTUNE benchmarking takes 0.2293 seconds and 0.0003 seconds precompiling for 17 choices
2025-04-21T23:30:28.4583577Z AUTOTUNE bmm(14x5600x80, 14x80x14)
2025-04-21T23:30:28.4584059Z strides: [80, 1120, 1], [1152, 1, 80]
2025-04-21T23:30:28.4584456Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:28.4585168Z   triton_bmm_43 0.0430 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=2
2025-04-21T23:30:28.4586280Z   triton_bmm_46 0.0430 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:28.4587351Z   triton_bmm_44 0.0440 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=2
2025-04-21T23:30:28.4588403Z   triton_bmm_49 0.0440 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.4589475Z   triton_bmm_52 0.0440 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4
2025-04-21T23:30:28.4590548Z   triton_bmm_56 0.0440 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.4591613Z   triton_bmm_45 0.0451 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:28.4592673Z   triton_bmm_51 0.0451 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4
2025-04-21T23:30:28.4593804Z   triton_bmm_57 0.0451 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=128, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
2025-04-21T23:30:28.4594867Z   triton_bmm_50 0.0461 ms 93.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=16, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:28.4595792Z SingleProcess AUTOTUNE benchmarking takes 0.2312 seconds and 0.0003 seconds precompiling for 17 choices
2025-04-21T23:30:28.7723708Z E0421 23:30:28.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:28.7725543Z E0421 23:30:28.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:28.7726807Z E0421 23:30:28.771000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:28.8811060Z E0421 23:30:28.880000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:28.8812673Z E0421 23:30:28.880000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:28.8813947Z E0421 23:30:28.880000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:29.0589375Z E0421 23:30:29.058000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:29.0590710Z E0421 23:30:29.058000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:29.0591977Z E0421 23:30:29.058000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:29.0599152Z AUTOTUNE mm(4900x1280, 1280x1280)
2025-04-21T23:30:29.0599567Z strides: [1280, 1], [1, 1280]
2025-04-21T23:30:29.0599860Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:29.0600563Z   triton_mm_68 0.2550 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:29.0601662Z   triton_mm_74 0.2550 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.0602720Z   triton_mm_72 0.2560 ms 99.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:29.0603765Z   triton_mm_64 0.2570 ms 99.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:29.0604815Z   triton_mm_67 0.2570 ms 99.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.0605854Z   triton_mm_65 0.2601 ms 98.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:29.0606908Z   triton_mm_71 0.2601 ms 98.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.0607946Z   triton_mm_63 0.2642 ms 96.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:29.0608990Z   triton_mm_69 0.2662 ms 95.8% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.0609657Z   mm 0.2796 ms 91.2% 
2025-04-21T23:30:29.0610146Z SingleProcess AUTOTUNE benchmarking takes 0.6010 seconds and 0.0003 seconds precompiling for 20 choices
2025-04-21T23:30:29.4421813Z E0421 23:30:29.441000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:29.4423475Z E0421 23:30:29.441000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:29.4424756Z E0421 23:30:29.441000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:29.5538593Z E0421 23:30:29.553000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:29.5540156Z E0421 23:30:29.553000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 147456, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:29.5541422Z E0421 23:30:29.553000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:29.7390106Z E0421 23:30:29.738000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Runtime error during autotuning: 
2025-04-21T23:30:29.7391431Z E0421 23:30:29.738000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] No valid triton configs. OutOfResources: out of resource: shared memory, Required: 131072, Hardware limit: 101376. Reducing block sizes or `num_stages` may help.. 
2025-04-21T23:30:29.7392694Z E0421 23:30:29.738000 51429 site-packages/torch/_inductor/select_algorithm.py:2135] [0/0] Ignoring this choice.
2025-04-21T23:30:29.7400578Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2025-04-21T23:30:29.7400984Z strides: [0, 1], [1280, 1], [1, 1280]
2025-04-21T23:30:29.7401352Z dtypes: torch.float16, torch.float16, torch.float16
2025-04-21T23:30:29.7402132Z   triton_mm_773 0.6052 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:29.7403262Z   triton_mm_777 0.6093 ms 99.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=8
2025-04-21T23:30:29.7404342Z   triton_mm_779 0.6093 ms 99.3% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.7405425Z   triton_mm_772 0.6164 ms 98.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.7406513Z   triton_mm_776 0.6164 ms 98.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.7407584Z   triton_mm_769 0.6226 ms 97.2% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:29.7408651Z   triton_mm_770 0.6328 ms 95.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:29.7409763Z   triton_mm_774 0.6339 ms 95.5% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=128, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:29.7410458Z   bias_addmm 0.6400 ms 94.6% 
2025-04-21T23:30:29.7411116Z   triton_mm_768 0.6513 ms 92.9% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=64, EVEN_K=True, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:29.7412057Z SingleProcess AUTOTUNE benchmarking takes 0.6637 seconds and 0.0002 seconds precompiling for 21 choices
2025-04-21T23:30:30.0066922Z AUTOTUNE bmm(64x1024x80, 64x80x64)
2025-04-21T23:30:30.0067602Z strides: [81920, 80, 1], [5120, 1, 80]
2025-04-21T23:30:30.0068212Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:30.0069555Z   triton_bmm_783 0.0440 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=32, BLOCK_N=32, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:30.0070682Z   triton_bmm_786 0.0440 ms 100.0% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=32, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:30.0071785Z   triton_bmm_784 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=32, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
2025-04-21T23:30:30.0073018Z   triton_bmm_785 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=32, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=8
2025-04-21T23:30:30.0074165Z   triton_bmm_789 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=8
2025-04-21T23:30:30.0075255Z   triton_bmm_790 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=5, num_warps=4
2025-04-21T23:30:30.0076343Z   triton_bmm_793 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=64, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:30.0077432Z   triton_bmm_794 0.0451 ms 97.7% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=128, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=4, num_warps=4
2025-04-21T23:30:30.0078522Z   triton_bmm_788 0.0461 ms 95.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=2, num_warps=4
2025-04-21T23:30:30.0079650Z   triton_bmm_791 0.0461 ms 95.6% ACC_TYPE='tl.float32', ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, EVEN_K=False, GROUP_M=8, USE_FAST_ACCUM=False, num_stages=3, num_warps=4
2025-04-21T23:30:30.0080585Z SingleProcess AUTOTUNE benchmarking takes 0.2657 seconds and 0.0003 seconds precompiling for 19 choices
2025-04-21T23:30:30.2532934Z AUTOTUNE convolution(1x1280x64x64, 256x1280x1x1)
2025-04-21T23:30:30.2533419Z strides: [5242880, 4096, 64, 1], [1280, 1, 1, 1]
2025-04-21T23:30:30.2533762Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:30.2534613Z   triton_convolution2d_3483 0.0645 ms 100.0% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2025-04-21T23:30:30.2536016Z   triton_convolution2d_3484 0.0737 ms 87.5% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2025-04-21T23:30:30.2537390Z   triton_convolution2d_3485 0.0737 ms 87.5% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2025-04-21T23:30:30.2538767Z   triton_convolution2d_3479 0.0758 ms 85.1% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2025-04-21T23:30:30.2540184Z   triton_convolution2d_3482 0.0819 ms 78.8% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=8
2025-04-21T23:30:30.2541551Z   triton_convolution2d_3480 0.0860 ms 75.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=2, num_warps=4
2025-04-21T23:30:30.2542395Z   convolution 0.0891 ms 72.4% 
2025-04-21T23:30:30.2543217Z   triton_convolution2d_3481 0.1536 ms 42.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=1, KERNEL_W=1, PADDING_H=0, PADDING_W=0, STRIDE_H=1, STRIDE_W=1, UNROLL=True, num_stages=1, num_warps=8
2025-04-21T23:30:30.2544337Z   conv1x1_via_mm 0.4024 ms 16.0% 
2025-04-21T23:30:30.2544857Z SingleProcess AUTOTUNE benchmarking takes 0.1779 seconds and 0.0002 seconds precompiling for 9 choices
2025-04-21T23:30:30.5210601Z AUTOTUNE convolution(1x256x64x64, 256x256x3x3)
2025-04-21T23:30:30.5210989Z strides: [1048576, 4096, 64, 1], [2304, 9, 3, 1]
2025-04-21T23:30:30.5211321Z dtypes: torch.float16, torch.float16
2025-04-21T23:30:30.5211629Z   convolution 0.1187 ms 100.0% 
2025-04-21T23:30:30.5212718Z   triton_convolution2d_3492 0.1997 ms 59.5% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:30.5214097Z   triton_convolution2d_3487 0.2284 ms 52.0% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=256, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:30.5215478Z   triton_convolution2d_3489 0.2673 ms 44.4% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=128, BLOCK_N=128, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:30.5216831Z   triton_convolution2d_3488 0.3215 ms 36.9% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=1024, BLOCK_N=16, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=1, num_warps=8
2025-04-21T23:30:30.5218181Z   triton_convolution2d_3490 0.3901 ms 30.4% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=64, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:30.5219532Z   triton_convolution2d_3491 0.4250 ms 27.9% ALLOW_TF32=False, BLOCK_K=32, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=8
2025-04-21T23:30:30.5220920Z   triton_convolution2d_3486 0.5089 ms 23.3% ALLOW_TF32=False, BLOCK_K=16, BLOCK_M=64, BLOCK_N=256, GROUPS=1, KERNEL_H=3, KERNEL_W=3, PADDING_H=1, PADDING_W=1, STRIDE_H=1, STRIDE_W=1, UNROLL=False, num_stages=2, num_warps=4
2025-04-21T23:30:30.5222427Z SingleProcess AUTOTUNE benchmarking takes 0.2672 seconds and 0.0002 seconds precompiling for 8 choices
2025-04-21T23:31:01.3237235Z Warning: Custom flash attention kernels were written specifically for A100.
2025-04-21T23:31:01.3238352Z We will try to read previously created kernel configurations from /var/lib/jenkins/workspace/flash_4_configs.p.
2025-04-21T23:31:01.3239411Z You can disable this kernel by setting SEGMENT_ANYTHING_FAST_USE_FLASH_4=0
2025-04-21T23:31:01.3241220Z key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))  not found. Running autotune. This might take a while.
2025-04-21T23:31:01.3242810Z all configs len:  60
2025-04-21T23:31:05.1238197Z (64, 64, 1, 1)  : 39995.52540008153
2025-04-21T23:31:07.7459535Z (64, 64, 2, 1)  : 6995.476049996796
2025-04-21T23:31:10.5893948Z (64, 64, 2, 2)  : 7904.694319986447
2025-04-21T23:31:11.8193987Z (64, 64, 4, 1)  : 2644.365939995623
2025-04-21T23:31:13.2916720Z (64, 64, 4, 2)  : 3761.1643399941386
2025-04-21T23:31:14.6228242Z (64, 64, 4, 3)  : 3007.9510299947287
2025-04-21T23:31:15.3438196Z (64, 64, 4, 4)  : None
2025-04-21T23:31:16.6725498Z (64, 64, 8, 1)  : 4035.330970000359
2025-04-21T23:31:17.9689118Z (64, 64, 8, 2)  : 3867.725190011697
2025-04-21T23:31:19.1644429Z (64, 64, 8, 3)  : 3294.004520012095
2025-04-21T23:31:19.6882561Z (64, 64, 8, 4)  : None
2025-04-21T23:31:20.2216378Z (64, 64, 8, 5)  : None
2025-04-21T23:31:20.7647154Z (64, 64, 8, 6)  : None
2025-04-21T23:31:21.3278217Z (64, 64, 8, 7)  : None
2025-04-21T23:31:21.9174981Z (64, 64, 8, 8)  : None
2025-04-21T23:31:28.5639470Z (64, 128, 1, 1)  : 68922.37840002053
2025-04-21T23:31:30.7324100Z (64, 128, 2, 1)  : 10440.559450034925
2025-04-21T23:31:32.9948629Z (64, 128, 2, 2)  : 11018.206000062492
2025-04-21T23:31:34.7123830Z (64, 128, 4, 1)  : 3860.5994299905433
2025-04-21T23:31:36.6311568Z (64, 128, 4, 2)  : 4523.418310000125
2025-04-21T23:31:37.6770526Z (64, 128, 4, 3)  : None
2025-04-21T23:31:38.7625486Z (64, 128, 4, 4)  : None
2025-04-21T23:31:40.1112972Z (64, 128, 8, 1)  : 3627.1677300101146
2025-04-21T23:31:41.4280820Z (64, 128, 8, 2)  : 3299.237680002989
2025-04-21T23:31:42.0597378Z (64, 128, 8, 3)  : None
2025-04-21T23:31:42.7430050Z (64, 128, 8, 4)  : None
2025-04-21T23:31:43.4474424Z (64, 128, 8, 5)  : None
2025-04-21T23:31:44.1885420Z (64, 128, 8, 6)  : None
2025-04-21T23:31:44.9567161Z (64, 128, 8, 7)  : None
2025-04-21T23:31:45.8033495Z (64, 128, 8, 8)  : None
2025-04-21T23:31:54.3505442Z (128, 64, 1, 1)  : 69743.12260008446
2025-04-21T23:31:57.2970764Z (128, 64, 2, 1)  : 23266.050800157245
2025-04-21T23:32:00.3088881Z (128, 64, 2, 2)  : 23757.679799928155
2025-04-21T23:32:02.3739119Z (128, 64, 4, 1)  : 4908.6776100011775
2025-04-21T23:32:03.8917359Z (128, 64, 4, 2)  : 13726.107900038187
2025-04-21T23:32:05.0160007Z (128, 64, 4, 3)  : None
2025-04-21T23:32:06.1686374Z (128, 64, 4, 4)  : None
2025-04-21T23:32:07.4523303Z (128, 64, 8, 1)  : 2859.492930001579
2025-04-21T23:32:08.9701267Z (128, 64, 8, 2)  : 4038.0376499888375
2025-04-21T23:32:10.5667743Z (128, 64, 8, 3)  : 4333.495580012823
2025-04-21T23:32:11.2726613Z (128, 64, 8, 4)  : None
2025-04-21T23:32:11.9950976Z (128, 64, 8, 5)  : None
2025-04-21T23:32:12.7109103Z (128, 64, 8, 6)  : None
2025-04-21T23:32:13.4777701Z (128, 64, 8, 7)  : None
2025-04-21T23:32:14.2470166Z (128, 64, 8, 8)  : None
2025-04-21T23:32:29.1378582Z (128, 128, 1, 1)  : 86435.14333380153
2025-04-21T23:32:34.4023957Z (128, 128, 2, 1)  : 104178.59449989919
2025-04-21T23:32:39.8626140Z (128, 128, 2, 2)  : 88029.32166630247
2025-04-21T23:32:41.5315625Z (128, 128, 4, 1)  : None
2025-04-21T23:32:43.4558427Z (128, 128, 4, 2)  : None
2025-04-21T23:32:45.4845527Z (128, 128, 4, 3)  : None
2025-04-21T23:32:47.5558446Z (128, 128, 4, 4)  : None
2025-04-21T23:32:49.4550040Z (128, 128, 8, 1)  : 4369.448699999339
2025-04-21T23:32:51.2306348Z (128, 128, 8, 2)  : 33520.39750006952
2025-04-21T23:32:52.3313710Z (128, 128, 8, 3)  : None
2025-04-21T23:32:53.4642498Z (128, 128, 8, 4)  : None
2025-04-21T23:32:54.6408293Z (128, 128, 8, 5)  : None
2025-04-21T23:32:55.8597633Z (128, 128, 8, 6)  : None
2025-04-21T23:32:57.1086525Z (128, 128, 8, 7)  : None
2025-04-21T23:32:58.4028272Z (128, 128, 8, 8)  : None
2025-04-21T23:32:58.4029520Z Found best_config  (64, 64, 4, 1)  with time  2644.365939995623  for key  (torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), torch.Size([1, 16, 4096, 128]), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1), (8388608, 524288, 128, 1))
2025-04-21T23:32:58.4030922Z Warning: Custom flash attention kernels were written specifically for A100.
2025-04-21T23:32:58.4031617Z Storing configs for NVIDIA A10G locally under /var/lib/jenkins/workspace/flash_4_configs.p
2025-04-21T23:32:58.4032272Z Saving best configs to file /var/lib/jenkins/workspace/flash_4_configs.p
2025-04-21T23:33:06.2059257Z 
2025-04-21T23:33:06.2059717Z loading model: 0it [03:40, ?it/s]
2025-04-21T23:33:06.2102707Z cuda eval  sam_fast                           
2025-04-21T23:33:39.1963147Z AUTOTUNE addmm(4096x3840, 4096x1280, 1280x3840)
2025-04-21T23:33:39.1963522Z strides: [0, 1], [1280, 1], [1, 1280]
2025-04-21T23:33:39.1963868Z dtypes: torch.float64, torch.float64, torch.float64
2025-04-21T23:33:39.1964217Z   bias_addmm 85.9218 ms 100.0% 
2025-04-21T23:33:39.1964492Z   addmm 86.2433 ms 99.6% 
2025-04-21T23:33:39.1965340Z SingleProcess AUTOTUNE benchmarking takes 1.2328 seconds and 0.0002 seconds precompiling for 2 choices
2025-04-21T23:36:36.1892149Z Compilation time (from dynamo_timed): 51.536752596
2025-04-21T23:36:36.1893925Z pass
2025-04-21T23:36:36.2295977Z TIMING: _recursive_pre_grad_passes:0.02625 pad_mm_benchmark:0.74617 _recursive_joint_graph_passes:2.66991 _recursive_post_grad_passes:1.67034 async_compile.wait:2.64212 code_gen:14.38564 inductor_compile:24.79499 backend_compile:42.20314 entire_frame_compile:51.53675 gc:0.00074 cudagraphify.get_container:1.68475 CachingAutotuner.benchmark_all_configs:1.65287 CUDAGraphNode.record:1.51779 total_wall_time:51.53675
2025-04-21T23:36:36.2298749Z STATS: call_* op count: 6495 | FakeTensorMode.__torch_dispatch__:234669 | FakeTensor.__torch_dispatch__:75229 | ProxyTorchDispatchMode.__torch_dispatch__:83610
2025-04-21T23:36:36.2299687Z Dynamo produced 0 graphs covering 257 ops with 0 graph breaks (0 unique)
2025-04-21T23:36:48.3033071Z 
2025-04-21T23:36:48.5106531Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/shufflenetv2_x1-5666bf0f80.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/shufflenetv2_x1-5666bf0f80.pth
2025-04-21T23:36:48.5264029Z 
2025-04-21T23:36:48.5264243Z 
2025-04-21T23:36:48.5499415Z   0% 0.00/8.79M [00:00<?, ?B/s]
2025-04-21T23:36:48.5499766Z 100% 8.79M/8.79M [00:00<00:00, 396MB/s]
2025-04-21T23:36:49.2796073Z 
2025-04-21T23:36:49.2796578Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:36:49.2862746Z cuda eval  shufflenet_v2_x1_0                 
2025-04-21T23:37:04.5489723Z Compilation time (from dynamo_timed): 13.116338793
2025-04-21T23:37:04.5490093Z pass
2025-04-21T23:37:04.5491475Z TIMING: _recursive_pre_grad_passes:0.00876 pad_mm_benchmark:0.06318 _recursive_joint_graph_passes:0.29611 _recursive_post_grad_passes:0.19263 async_compile.wait:0.06498 code_gen:2.28903 inductor_compile:5.16593 backend_compile:9.41579 entire_frame_compile:13.11634 gc:0.00139 cudagraphify.get_container:0.26393 CUDAGraphNode.record:0.24879 total_wall_time:13.11634
2025-04-21T23:37:04.5493354Z STATS: call_* op count: 271 | FakeTensorMode.__torch_dispatch__:12793 | ProxyTorchDispatchMode.__torch_dispatch__:4744 | FakeTensor.__torch_dispatch__:7309
2025-04-21T23:37:04.5494199Z Dynamo produced 1 graphs covering 271 ops with 0 graph breaks (0 unique)
2025-04-21T23:37:10.2184608Z 
2025-04-21T23:37:11.2645312Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:37:11.2645671Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:37:11.2648358Z cuda eval  soft_actor_critic                  
2025-04-21T23:37:14.3521663Z Compilation time (from dynamo_timed): 1.655546243
2025-04-21T23:37:14.3522084Z pass
2025-04-21T23:37:14.3524114Z TIMING: _recursive_pre_grad_passes:0.00346 pad_mm_benchmark:0.16331 _recursive_joint_graph_passes:0.30056 _recursive_post_grad_passes:0.03027 async_compile.precompile:0.18032 async_compile.wait:0.00019 code_gen:0.24 inductor_compile:0.36314 backend_compile:1.46074 entire_frame_compile:1.65555 gc:0.00042 cudagraphify.get_container:0.15809 CUDAGraphNode.record:0.15583 total_wall_time:1.65555
2025-04-21T23:37:14.3526019Z STATS: call_* op count: 16 | FakeTensorMode.__torch_dispatch__:751 | ProxyTorchDispatchMode.__torch_dispatch__:169 | FakeTensor.__torch_dispatch__:116
2025-04-21T23:37:14.3526816Z Dynamo produced 1 graphs covering 16 ops with 0 graph breaks (0 unique)
2025-04-21T23:37:19.7164977Z 
2025-04-21T23:37:21.2377519Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:37:21.2377893Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:37:21.2442578Z cuda eval  speech_transformer                 
2025-04-21T23:37:32.3603097Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg13_1). Found from : 
2025-04-21T23:37:32.3604846Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 72, in torch_dynamo_resume_in_forward_at_63
2025-04-21T23:37:32.3606229Z     enc_output, enc_slf_attn = enc_layer(
2025-04-21T23:37:32.3607427Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 95, in forward
2025-04-21T23:37:32.3609190Z     enc_output, enc_slf_attn = self.slf_attn(
2025-04-21T23:37:32.3610434Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 53, in forward
2025-04-21T23:37:32.3611696Z     output, attn = self.attention(q, k, v, mask=mask)
2025-04-21T23:37:32.3612998Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2025-04-21T23:37:32.3614473Z     attn = attn / self.temperature
2025-04-21T23:37:32.3614775Z 
2025-04-21T23:37:48.3318553Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg11_1). Found from : 
2025-04-21T23:37:48.3319897Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 131, in torch_dynamo_resume_in_forward_at_120
2025-04-21T23:37:48.3320901Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2025-04-21T23:37:48.3322140Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 291, in forward
2025-04-21T23:37:48.3323122Z     dec_output, dec_slf_attn = self.slf_attn(
2025-04-21T23:37:48.3324038Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 53, in forward
2025-04-21T23:37:48.3324877Z     output, attn = self.attention(q, k, v, mask=mask)
2025-04-21T23:37:48.3325733Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2025-04-21T23:37:48.3326545Z     attn = attn / self.temperature
2025-04-21T23:37:48.3326742Z 
2025-04-21T23:37:50.5103143Z Compilation time (from dynamo_timed): 25.815162688
2025-04-21T23:37:50.5103536Z pass
2025-04-21T23:37:50.5130942Z TIMING: entire_frame_compile:25.81516 gc:0.00724 _recursive_pre_grad_passes:0.0215 _recursive_joint_graph_passes:1.4031 _recursive_post_grad_passes:1.01026 async_compile.precompile:0.08358 async_compile.wait:1.04221 code_gen:4.64612 inductor_compile:9.09729 backend_compile:19.3551 cudagraphify.get_container:0.15389 pad_mm_benchmark:0.72871 CachingAutotuner.benchmark_all_configs:0.01556 CUDAGraphNode.record:1.26291 total_wall_time:25.81516
2025-04-21T23:37:50.5133130Z STATS: call_* op count: 1454 | FakeTensorMode.__torch_dispatch__:40012 | FakeTensor.__torch_dispatch__:8320 | ProxyTorchDispatchMode.__torch_dispatch__:17586
2025-04-21T23:37:50.5133974Z Dynamo produced 7 graphs covering 1454 ops with 10 graph breaks (2 unique)
2025-04-21T23:37:56.5203109Z 
2025-04-21T23:37:56.7220459Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/squeezenet1_1-b8a52dc0.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/squeezenet1_1-b8a52dc0.pth
2025-04-21T23:37:56.7390487Z 
2025-04-21T23:37:56.7390635Z 
2025-04-21T23:37:56.7540578Z   0% 0.00/4.73M [00:00<?, ?B/s]
2025-04-21T23:37:56.7540885Z 100% 4.73M/4.73M [00:00<00:00, 335MB/s]
2025-04-21T23:37:57.4494221Z 
2025-04-21T23:37:57.4494441Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:37:57.4509379Z cuda eval  squeezenet1_1                      
2025-04-21T23:38:03.8307688Z Compilation time (from dynamo_timed): 4.352833311
2025-04-21T23:38:03.8308067Z pass
2025-04-21T23:38:03.8310055Z TIMING: _recursive_pre_grad_passes:0.00442 _recursive_joint_graph_passes:0.13805 _recursive_post_grad_passes:0.02507 async_compile.wait:0.10943 code_gen:1.36806 inductor_compile:2.07544 backend_compile:3.44673 entire_frame_compile:4.35283 gc:0.00055 cudagraphify.get_container:0.1832 CUDAGraphNode.record:0.17903 total_wall_time:4.35283
2025-04-21T23:38:03.8311770Z STATS: call_* op count: 66 | FakeTensorMode.__torch_dispatch__:2193 | ProxyTorchDispatchMode.__torch_dispatch__:550 | FakeTensor.__torch_dispatch__:562
2025-04-21T23:38:03.8312812Z Dynamo produced 1 graphs covering 66 ops with 0 graph breaks (0 unique)
2025-04-21T23:38:09.2172741Z 
2025-04-21T23:38:10.2916532Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:38:10.2916858Z 
2025-04-21T23:38:10.4906169Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2025-04-21T23:38:10.4906776Z 
2025-04-21T23:38:11.5208239Z Loading pipeline components...:  67% 4/6 [00:00<00:00, 20.14it/s]
2025-04-21T23:38:11.5208796Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.88it/s]
2025-04-21T23:38:13.1035393Z 
2025-04-21T23:38:13.1035964Z loading model: 0it [00:03, ?it/s]
2025-04-21T23:38:13.1124212Z cuda eval  stable_diffusion_text_encoder      
2025-04-21T23:38:13.1806749Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2025-04-21T23:38:37.1943723Z Compilation time (from dynamo_timed): 21.717267513
2025-04-21T23:38:37.1944080Z pass
2025-04-21T23:38:37.2072358Z TIMING: _recursive_pre_grad_passes:0.0198 pad_mm_benchmark:0.51573 _recursive_joint_graph_passes:1.39688 _recursive_post_grad_passes:1.2889 async_compile.wait:0.20877 code_gen:4.12548 inductor_compile:9.15205 backend_compile:16.95086 entire_frame_compile:21.71727 gc:0.00172 cudagraphify.get_container:0.48387 CUDAGraphNode.record:0.37669 total_wall_time:21.71727
2025-04-21T23:38:37.2076151Z STATS: call_* op count: 824 | FakeTensorMode.__torch_dispatch__:30873 | FakeTensor.__torch_dispatch__:11686 | ProxyTorchDispatchMode.__torch_dispatch__:12189
2025-04-21T23:38:37.2077388Z Dynamo produced 1 graphs covering 824 ops with 0 graph breaks (0 unique)
2025-04-21T23:38:43.1433830Z 
2025-04-21T23:38:44.2137058Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:38:44.2137315Z 
2025-04-21T23:38:44.3983028Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2025-04-21T23:38:44.3983352Z 
2025-04-21T23:38:45.4300656Z Loading pipeline components...:  67% 4/6 [00:00<00:00, 21.69it/s]
2025-04-21T23:38:45.4301182Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.93it/s]
2025-04-21T23:38:47.2308384Z 
2025-04-21T23:38:47.2308606Z loading model: 0it [00:04, ?it/s]
2025-04-21T23:38:47.2483538Z cuda eval  stable_diffusion_unet              
2025-04-21T23:38:47.2719655Z Compilation time (from dynamo_timed): 0
2025-04-21T23:38:47.2720020Z pass_due_to_skip
2025-04-21T23:38:47.4307576Z TIMING: total_wall_time:0
2025-04-21T23:38:47.4307866Z STATS: call_* op count: 0
2025-04-21T23:38:47.4308247Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2025-04-21T23:38:51.7464154Z 
2025-04-21T23:38:52.8923072Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:38:52.8923331Z 
2025-04-21T23:38:52.9703093Z model.safetensors:   0% 0.00/21.4M [00:00<?, ?B/s]
2025-04-21T23:38:52.9703532Z model.safetensors: 100% 21.4M/21.4M [00:00<00:00, 274MB/s]
2025-04-21T23:38:53.7988371Z 
2025-04-21T23:38:53.7988764Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:38:53.8068971Z cuda eval  timm_efficientnet                  
2025-04-21T23:39:12.4412950Z Compilation time (from dynamo_timed): 15.442009923
2025-04-21T23:39:12.4413423Z pass
2025-04-21T23:39:12.4416838Z TIMING: _recursive_pre_grad_passes:0.01512 pad_mm_benchmark:0.00133 _recursive_joint_graph_passes:0.21828 _recursive_post_grad_passes:0.20319 async_compile.wait:0.13379 code_gen:2.99749 inductor_compile:6.38411 backend_compile:11.06722 entire_frame_compile:15.44201 gc:0.00014 cudagraphify.get_container:0.32761 CUDAGraphNode.record:0.30352 total_wall_time:15.44201
2025-04-21T23:39:12.4418682Z STATS: call_* op count: 239 | FakeTensorMode.__torch_dispatch__:13567 | FakeTensor.__torch_dispatch__:7458 | ProxyTorchDispatchMode.__torch_dispatch__:4941
2025-04-21T23:39:12.4419537Z Dynamo produced 1 graphs covering 239 ops with 0 graph breaks (0 unique)
2025-04-21T23:39:18.1742950Z 
2025-04-21T23:39:19.7887351Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:39:19.7887601Z 
2025-04-21T23:39:19.8996770Z model.safetensors:   0% 0.00/208M [00:00<?, ?B/s]
2025-04-21T23:39:19.8997052Z 
2025-04-21T23:39:20.0102270Z model.safetensors:  20% 41.9M/208M [00:00<00:00, 378MB/s]
2025-04-21T23:39:20.0102576Z 
2025-04-21T23:39:20.1238142Z model.safetensors:  45% 94.4M/208M [00:00<00:00, 435MB/s]
2025-04-21T23:39:20.1238456Z 
2025-04-21T23:39:20.2294320Z model.safetensors:  71% 147M/208M [00:00<00:00, 447MB/s] 
2025-04-21T23:39:20.2294613Z 
2025-04-21T23:39:20.2518968Z model.safetensors:  96% 199M/208M [00:00<00:00, 466MB/s]
2025-04-21T23:39:20.2519409Z model.safetensors: 100% 208M/208M [00:00<00:00, 449MB/s]
2025-04-21T23:39:21.3059381Z 
2025-04-21T23:39:21.3060151Z loading model: 0it [00:03, ?it/s]
2025-04-21T23:39:21.3155348Z cuda eval  timm_regnet                        
2025-04-21T23:39:43.6189185Z Compilation time (from dynamo_timed): 16.522966508
2025-04-21T23:39:43.6189705Z pass
2025-04-21T23:39:43.6216482Z TIMING: _recursive_pre_grad_passes:0.00977 pad_mm_benchmark:0.06409 _recursive_joint_graph_passes:0.28176 _recursive_post_grad_passes:0.46933 async_compile.wait:0.32785 code_gen:2.36904 inductor_compile:5.81978 backend_compile:11.01941 entire_frame_compile:16.52297 gc:0.00018 cudagraphify.get_container:0.33183 CUDAGraphNode.record:0.30903 total_wall_time:16.52297
2025-04-21T23:39:43.6219344Z STATS: call_* op count: 319 | FakeTensorMode.__torch_dispatch__:15647 | ProxyTorchDispatchMode.__torch_dispatch__:5814 | FakeTensor.__torch_dispatch__:8775
2025-04-21T23:39:43.6220594Z Dynamo produced 1 graphs covering 319 ops with 0 graph breaks (0 unique)
2025-04-21T23:39:49.3470799Z 
2025-04-21T23:39:50.5210354Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:39:50.5210599Z 
2025-04-21T23:39:50.6373667Z model.safetensors:   0% 0.00/42.6M [00:00<?, ?B/s]
2025-04-21T23:39:50.6373942Z 
2025-04-21T23:39:50.6685933Z model.safetensors:  74% 31.5M/42.6M [00:00<00:00, 271MB/s]
2025-04-21T23:39:50.6686382Z model.safetensors: 100% 42.6M/42.6M [00:00<00:00, 289MB/s]
2025-04-21T23:39:51.5164881Z 
2025-04-21T23:39:51.5165061Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:39:51.5198491Z cuda eval  timm_resnest                       
2025-04-21T23:40:03.8549740Z Compilation time (from dynamo_timed): 8.857009414
2025-04-21T23:40:03.8550273Z pass
2025-04-21T23:40:03.8553153Z TIMING: _recursive_pre_grad_passes:0.00585 pad_mm_benchmark:0.00138 _recursive_joint_graph_passes:0.19369 _recursive_post_grad_passes:0.15236 async_compile.wait:0.26759 code_gen:3.07769 inductor_compile:4.77194 backend_compile:6.94642 entire_frame_compile:8.85701 gc:0.00024 cudagraphify.get_container:0.24476 CUDAGraphNode.record:0.23533 total_wall_time:8.85701
2025-04-21T23:40:03.8556037Z STATS: call_* op count: 131 | FakeTensorMode.__torch_dispatch__:6526 | ProxyTorchDispatchMode.__torch_dispatch__:2205 | FakeTensor.__torch_dispatch__:3283
2025-04-21T23:40:03.8557173Z Dynamo produced 1 graphs covering 131 ops with 0 graph breaks (0 unique)
2025-04-21T23:40:09.3462972Z 
2025-04-21T23:40:10.6588857Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:40:10.6589107Z 
2025-04-21T23:40:10.7774633Z model.safetensors:   0% 0.00/88.2M [00:00<?, ?B/s]
2025-04-21T23:40:10.7774932Z 
2025-04-21T23:40:10.8840194Z model.safetensors:  48% 41.9M/88.2M [00:00<00:00, 354MB/s]
2025-04-21T23:40:10.8840486Z 
2025-04-21T23:40:10.8867239Z model.safetensors: 100% 88.2M/88.2M [00:00<00:00, 399MB/s]
2025-04-21T23:40:10.8868094Z model.safetensors: 100% 88.2M/88.2M [00:00<00:00, 387MB/s]
2025-04-21T23:40:11.5118575Z 
2025-04-21T23:40:11.5118927Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:40:11.5159464Z cuda eval  timm_vision_transformer            
2025-04-21T23:40:22.3900232Z Compilation time (from dynamo_timed): 8.714675431
2025-04-21T23:40:22.3900592Z pass
2025-04-21T23:40:22.3908760Z TIMING: _recursive_pre_grad_passes:0.00677 pad_mm_benchmark:0.3869 _recursive_joint_graph_passes:0.72322 _recursive_post_grad_passes:0.33082 async_compile.wait:0.11069 code_gen:1.45583 inductor_compile:3.44851 backend_compile:6.75366 entire_frame_compile:8.71468 gc:0.0002 cudagraphify.get_container:0.25063 CUDAGraphNode.record:0.24681 total_wall_time:8.71468
2025-04-21T23:40:22.3910937Z STATS: call_* op count: 263 | FakeTensorMode.__torch_dispatch__:10955 | FakeTensor.__torch_dispatch__:4406 | ProxyTorchDispatchMode.__torch_dispatch__:4222
2025-04-21T23:40:22.3911768Z Dynamo produced 1 graphs covering 263 ops with 0 graph breaks (0 unique)
2025-04-21T23:40:27.8911876Z 
2025-04-21T23:40:40.7012400Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:40:40.7012753Z loading model: 0it [00:12, ?it/s]
2025-04-21T23:40:40.7143996Z cuda eval  timm_vision_transformer_large      
2025-04-21T23:40:40.7380605Z Compilation time (from dynamo_timed): 0
2025-04-21T23:40:40.7380941Z pass_due_to_skip
2025-04-21T23:40:40.8469566Z TIMING: total_wall_time:0
2025-04-21T23:40:40.8470275Z STATS: call_* op count: 0
2025-04-21T23:40:40.8470672Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2025-04-21T23:40:45.1689755Z 
2025-04-21T23:40:47.2017324Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:40:47.2017829Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:40:47.2068809Z cuda eval  timm_vovnet                        
2025-04-21T23:41:03.0454262Z Compilation time (from dynamo_timed): 10.191916345
2025-04-21T23:41:03.0454627Z pass
2025-04-21T23:41:03.0469968Z TIMING: _recursive_pre_grad_passes:0.00665 pad_mm_benchmark:0.00138 _recursive_joint_graph_passes:0.39182 _recursive_post_grad_passes:0.1386 async_compile.wait:0.07493 code_gen:2.44202 inductor_compile:4.23457 backend_compile:7.37894 entire_frame_compile:10.19192 gc:0.00019 cudagraphify.get_container:0.26373 CUDAGraphNode.record:0.25182 total_wall_time:10.19192
2025-04-21T23:41:03.0471814Z STATS: call_* op count: 130 | FakeTensorMode.__torch_dispatch__:8519 | ProxyTorchDispatchMode.__torch_dispatch__:3092 | FakeTensor.__torch_dispatch__:5023
2025-04-21T23:41:03.0472636Z Dynamo produced 1 graphs covering 130 ops with 0 graph breaks (0 unique)
2025-04-21T23:41:08.7980719Z 
2025-04-21T23:41:09.0300435Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:41:09.0300679Z 
2025-04-21T23:41:09.0373725Z clip_merges.bpe: 0.00B [00:00, ?B/s]
2025-04-21T23:41:09.0374083Z clip_merges.bpe: 532kB [00:00, 64.8MB/s]
2025-04-21T23:41:10.6828082Z 
2025-04-21T23:41:10.6828477Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:41:10.6906580Z cuda eval  torch_multimodal_clip              
2025-04-21T23:41:31.3892954Z Compilation time (from dynamo_timed): 17.41798863
2025-04-21T23:41:31.3893346Z pass
2025-04-21T23:41:31.3925460Z TIMING: _recursive_pre_grad_passes:0.0096 pad_mm_benchmark:0.11687 _recursive_joint_graph_passes:0.60727 _recursive_post_grad_passes:0.60206 async_compile.wait:0.71451 code_gen:4.381 inductor_compile:7.93659 backend_compile:13.18915 entire_frame_compile:17.41799 gc:0.0016 cudagraphify.get_container:0.32145 CachingAutotuner.benchmark_all_configs:0.04833 CUDAGraphNode.record:0.30135 total_wall_time:17.41799
2025-04-21T23:41:31.3927483Z STATS: call_* op count: 370 | FakeTensorMode.__torch_dispatch__:20654 | FakeTensor.__torch_dispatch__:8965 | ProxyTorchDispatchMode.__torch_dispatch__:7987
2025-04-21T23:41:31.3928305Z Dynamo produced 1 graphs covering 370 ops with 0 graph breaks (0 unique)
2025-04-21T23:41:37.2722585Z 
2025-04-21T23:41:37.6741402Z loading model: 0it [00:00, ?it/s]WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2025-04-21T23:41:37.9052131Z 
2025-04-21T23:41:37.9052322Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:41:37.9052794Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2025-04-21T23:41:37.9053289Z cuda eval  tts_angular                        
2025-04-21T23:41:37.9743729Z WARNING:common:Model tts_angular does not support bfloat16, running with amp instead
2025-04-21T23:41:40.3960704Z Compilation time (from dynamo_timed): 1.282236259
2025-04-21T23:41:40.3961108Z pass
2025-04-21T23:41:40.3985213Z TIMING: _recursive_pre_grad_passes:0.00339 _recursive_joint_graph_passes:0.13485 inductor_compile:0.2431 backend_compile:1.15354 entire_frame_compile:1.28224 gc:0.00237 _recursive_post_grad_passes:0.01743 async_compile.precompile:0.10563 async_compile.wait:0.00018 code_gen:0.15679 total_wall_time:1.28224
2025-04-21T23:41:40.3987254Z STATS: call_* op count: 6 | FakeTensorMode.__torch_dispatch__:376 | ProxyTorchDispatchMode.__torch_dispatch__:58 | FakeTensor.__torch_dispatch__:33
2025-04-21T23:41:40.3988078Z Dynamo produced 3 graphs covering 6 ops with 2 graph breaks (1 unique)
2025-04-21T23:41:46.2376977Z 
2025-04-21T23:41:47.2710884Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/vgg16-397923af.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/vgg16-397923af.pth
2025-04-21T23:41:47.2864427Z 
2025-04-21T23:41:47.2864565Z 
2025-04-21T23:41:47.3869359Z   0% 0.00/528M [00:00<?, ?B/s]
2025-04-21T23:41:47.3869577Z 
2025-04-21T23:41:47.4869971Z   7% 37.9M/528M [00:00<00:01, 396MB/s]
2025-04-21T23:41:47.4870194Z 
2025-04-21T23:41:47.5870673Z  15% 77.8M/528M [00:00<00:01, 409MB/s]
2025-04-21T23:41:47.5870893Z 
2025-04-21T23:41:47.6872395Z  22% 118M/528M [00:00<00:01, 414MB/s] 
2025-04-21T23:41:47.6872607Z 
2025-04-21T23:41:47.7878299Z  30% 158M/528M [00:00<00:00, 416MB/s]
2025-04-21T23:41:47.7878617Z 
2025-04-21T23:41:47.8878829Z  37% 198M/528M [00:00<00:00, 415MB/s]
2025-04-21T23:41:47.8879136Z 
2025-04-21T23:41:47.9880782Z  45% 238M/528M [00:00<00:00, 418MB/s]
2025-04-21T23:41:47.9881094Z 
2025-04-21T23:41:48.2011771Z  53% 279M/528M [00:00<00:00, 422MB/s]
2025-04-21T23:41:48.2012121Z 
2025-04-21T23:41:48.3145008Z  61% 320M/528M [00:00<00:00, 311MB/s]
2025-04-21T23:41:48.3145370Z 
2025-04-21T23:41:48.4147328Z  67% 353M/528M [00:01<00:00, 311MB/s]
2025-04-21T23:41:48.4147689Z 
2025-04-21T23:41:48.5149444Z  75% 397M/528M [00:01<00:00, 350MB/s]
2025-04-21T23:41:48.5149786Z 
2025-04-21T23:41:48.6151577Z  84% 442M/528M [00:01<00:00, 380MB/s]
2025-04-21T23:41:48.6151937Z 
2025-04-21T23:41:48.7292102Z  92% 486M/528M [00:01<00:00, 404MB/s]
2025-04-21T23:41:48.7292359Z 
2025-04-21T23:41:48.7354285Z 100% 526M/528M [00:01<00:00, 393MB/s]
2025-04-21T23:41:48.7354915Z 100% 528M/528M [00:01<00:00, 382MB/s]
2025-04-21T23:41:49.8476675Z 
2025-04-21T23:41:49.8477232Z loading model: 0it [00:03, ?it/s]
2025-04-21T23:41:49.8485221Z cuda eval  vgg16                              
2025-04-21T23:42:02.6534352Z Compilation time (from dynamo_timed): 3.838404102
2025-04-21T23:42:02.6534738Z pass
2025-04-21T23:42:02.6536203Z TIMING: _recursive_pre_grad_passes:0.00394 pad_mm_benchmark:0.22571 _recursive_joint_graph_passes:0.36652 _recursive_post_grad_passes:0.03392 async_compile.wait:0.35493 code_gen:1.35777 inductor_compile:1.73739 backend_compile:3.2578 entire_frame_compile:3.8384 gc:0.00044 cudagraphify.get_container:0.17418 CachingAutotuner.benchmark_all_configs:0.27674 CUDAGraphNode.record:0.17167 total_wall_time:3.8384
2025-04-21T23:42:02.6538190Z STATS: call_* op count: 40 | FakeTensorMode.__torch_dispatch__:1861 | ProxyTorchDispatchMode.__torch_dispatch__:426 | FakeTensor.__torch_dispatch__:330
2025-04-21T23:42:02.6538996Z Dynamo produced 1 graphs covering 40 ops with 0 graph breaks (0 unique)
2025-04-21T23:42:07.9261351Z 
2025-04-21T23:42:08.5221940Z loading model: 0it [00:00, ?it/s]Downloading: "https://download.pytorch.org/models/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth" to /var/lib/jenkins/.cache/torch/hub/checkpoints/maskrcnn_resnet50_fpn_coco-bf2d0c1e.pth
2025-04-21T23:42:08.5372402Z 
2025-04-21T23:42:08.5372510Z 
2025-04-21T23:42:08.6377271Z   0% 0.00/170M [00:00<?, ?B/s]
2025-04-21T23:42:08.6377518Z 
2025-04-21T23:42:08.7377960Z  25% 41.8M/170M [00:00<00:00, 437MB/s]
2025-04-21T23:42:08.7378208Z 
2025-04-21T23:42:08.8377426Z  50% 85.0M/170M [00:00<00:00, 447MB/s]
2025-04-21T23:42:08.8377742Z 
2025-04-21T23:42:08.9346563Z  76% 128M/170M [00:00<00:00, 450MB/s] 
2025-04-21T23:42:08.9346943Z 100% 170M/170M [00:00<00:00, 448MB/s]
2025-04-21T23:42:10.5754426Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2025-04-21T23:42:11.1177852Z 
2025-04-21T23:42:11.1178207Z loading model: 0it [00:03, ?it/s]
2025-04-21T23:42:11.1178732Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2025-04-21T23:42:11.1241983Z cuda eval  vision_maskrcnn                    
2025-04-21T23:42:13.1425348Z WARNING:common:Model vision_maskrcnn does not support bfloat16, running with float16 instead
2025-04-21T23:42:35.4495763Z W0421 23:42:35.448000 62833 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-21T23:42:35.4631090Z W0421 23:42:35.462000 62833 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-21T23:42:35.4769102Z W0421 23:42:35.476000 62833 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-21T23:42:35.4901528Z W0421 23:42:35.489000 62833 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-21T23:42:35.5031274Z W0421 23:42:35.502000 62833 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-21T23:42:36.9435273Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg11_1). Found from : 
2025-04-21T23:42:36.9436293Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2025-04-21T23:42:36.9437211Z     anchors = self.anchor_generator(images, features)
2025-04-21T23:42:36.9438196Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2025-04-21T23:42:36.9438950Z     self.set_cell_anchors(dtype, device)
2025-04-21T23:42:36.9439648Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2025-04-21T23:42:36.9440553Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2025-04-21T23:42:36.9441422Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2025-04-21T23:42:36.9442295Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2025-04-21T23:42:36.9442715Z 
2025-04-21T23:42:38.1099472Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] Graph break from `Tensor.item()`, consider setting:
2025-04-21T23:42:38.1101045Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-21T23:42:38.1101856Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] or:
2025-04-21T23:42:38.1102671Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-21T23:42:38.1103609Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] to include these operations in the captured graph.
2025-04-21T23:42:38.1104393Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] 
2025-04-21T23:42:38.1105139Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] Graph break: from user code at:
2025-04-21T23:42:38.1106515Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104
2025-04-21T23:42:38.1108126Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
2025-04-21T23:42:38.1109685Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward
2025-04-21T23:42:38.1111016Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     box_features = self.box_roi_pool(features, proposals, image_shapes)
2025-04-21T23:42:38.1112571Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward
2025-04-21T23:42:38.1113848Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     self.scales, self.map_levels = _setup_scales(
2025-04-21T23:42:38.1115066Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales
2025-04-21T23:42:38.1116514Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2025-04-21T23:42:38.1117805Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in <listcomp>
2025-04-21T23:42:38.1119087Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2025-04-21T23:42:38.1120365Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale
2025-04-21T23:42:38.1121837Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0]     scale = 2 ** float(torch.tensor(approx_scale).log2().round())
2025-04-21T23:42:38.1122662Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] 
2025-04-21T23:42:38.1123267Z W0421 23:42:38.108000 62833 site-packages/torch/_dynamo/variables/tensor.py:928] [12/0] 
2025-04-21T23:43:21.3021887Z Compilation time (from dynamo_timed): 65.740793272
2025-04-21T23:43:21.3022456Z pass
2025-04-21T23:43:21.3044184Z TIMING: entire_frame_compile:65.74079 gc:0.02105 _recursive_pre_grad_passes:0.0277 _recursive_joint_graph_passes:0.91057 _recursive_post_grad_passes:0.40171 async_compile.wait:4.36137 code_gen:12.02541 inductor_compile:20.07046 backend_compile:33.32335 cudagraphify.get_container:0.29977 pad_mm_benchmark:0.00265 CUDAGraphNode.record:0.45535 total_wall_time:65.74079
2025-04-21T23:43:21.3046850Z STATS: call_* op count: 1358 | FakeTensorMode.__torch_dispatch__:60923 | FakeTensor.__torch_dispatch__:16821 | ProxyTorchDispatchMode.__torch_dispatch__:18377 | attempt fast:152 | fast is_contiguous:96 | slow both tensors nontrivially broadcast:56
2025-04-21T23:43:21.3048030Z Dynamo produced 25 graphs covering 1358 ops with 18 graph breaks (2 unique)
2025-04-21T23:43:28.0507093Z 
2025-04-21T23:43:30.0098936Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:43:30.0099399Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:43:30.0184275Z cuda eval  yolov3                             
2025-04-21T23:43:50.9224835Z W0421 23:43:50.921000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:51.0769942Z W0421 23:43:51.076000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:51.2287979Z W0421 23:43:51.228000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:51.2967547Z W0421 23:43:51.296000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:51.2978121Z W0421 23:43:51.297000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:51.2988722Z W0421 23:43:51.298000 63567 site-packages/torch/_inductor/utils.py:1881] [0/0] DeviceCopy in input program
2025-04-21T23:43:59.6365826Z skipping cudagraphs due to skipping cudagraphs due to cpu device (arg298_1). Found from : 
2025-04-21T23:43:59.6366588Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 460, in forward_pass
2025-04-21T23:43:59.6367467Z     return mod(*inputs)
2025-04-21T23:43:59.6368042Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 300, in forward
2025-04-21T23:43:59.6368662Z     return self.forward_once(x)
2025-04-21T23:43:59.6369280Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 363, in forward_once
2025-04-21T23:43:59.6369925Z     yolo_out.append(module(x, out))
2025-04-21T23:43:59.6370527Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 237, in forward
2025-04-21T23:43:59.6371156Z     self.create_grids((nx, ny), p.device)
2025-04-21T23:43:59.6372001Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 204, in create_grids
2025-04-21T23:43:59.6372701Z     self.anchor_vec = self.anchor_vec.to(device)
2025-04-21T23:43:59.6372935Z 
2025-04-21T23:44:19.7903749Z Compilation time (from dynamo_timed): 40.467465431
2025-04-21T23:44:19.7906077Z pass
2025-04-21T23:44:19.8018308Z TIMING: _recursive_pre_grad_passes:0.01867 _recursive_joint_graph_passes:0.62966 _recursive_post_grad_passes:0.61063 async_compile.wait:1.93814 code_gen:12.11566 inductor_compile:19.38767 backend_compile:30.56 entire_frame_compile:40.46747 gc:0.00072 cudagraphify.get_container:0.48939 total_wall_time:40.46747
2025-04-21T23:44:19.8020802Z STATS: call_* op count: 649 | FakeTensorMode.__torch_dispatch__:34170 | FakeTensor.__torch_dispatch__:21276 | ProxyTorchDispatchMode.__torch_dispatch__:13986
2025-04-21T23:44:19.8022291Z Dynamo produced 2 graphs covering 649 ops with 0 graph breaks (0 unique)
2025-04-21T23:44:23.2473931Z accuracy            pass_rate=82.61%
2025-04-21T23:44:23.2478478Z calls_captured      gmean=0.00x mean=305.457x
2025-04-21T23:44:23.2482127Z unique_graphs       gmean=0.00x mean=1.630x
2025-04-21T23:44:23.2485812Z graph_breaks        gmean=0.00x mean=0.826x
2025-04-21T23:44:23.2489685Z unique_graph_breaks gmean=0.00x mean=0.217x
2025-04-21T23:44:23.2493287Z autograd_captures   gmean=0.00x mean=0.000x
2025-04-21T23:44:23.2496921Z autograd_compiles   gmean=0.00x mean=0.000x
2025-04-21T23:44:23.2500891Z cudagraph_skips     gmean=0.00x mean=0.174x
2025-04-21T23:44:23.2501745Z compilation_latency mean=13.280 seconds
2025-04-21T23:44:24.9647562Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2025-04-21T23:44:25.2991314Z llama_v2_7b_16h                     XFAIL
2025-04-21T23:44:25.2995618Z llava                               XFAIL
2025-04-21T23:44:25.2999955Z maml                                XFAIL
2025-04-21T23:44:25.3004704Z maml_omniglot                       PASS
2025-04-21T23:44:25.3009497Z microbench_unbacked_tolist_sum      PASS
2025-04-21T23:44:25.3014103Z mnasnet1_0                          PASS
2025-04-21T23:44:25.3018933Z mobilenet_v2                        PASS
2025-04-21T23:44:25.3023953Z mobilenet_v2_quantized_qat          XFAIL
2025-04-21T23:44:25.3028948Z mobilenet_v3_large                  PASS
2025-04-21T23:44:25.3033337Z moco                                PASS
2025-04-21T23:44:25.3038038Z moondream                           XFAIL
2025-04-21T23:44:25.3042596Z nanogpt                             PASS
2025-04-21T23:44:25.3047281Z nvidia_deeprecommender              PASS
2025-04-21T23:44:25.3051838Z opacus_cifar10                      PASS
2025-04-21T23:44:25.3056475Z phlippe_densenet                    PASS
2025-04-21T23:44:25.3061061Z phlippe_resnet                      PASS
2025-04-21T23:44:25.3065697Z pyhpc_equation_of_state             PASS
2025-04-21T23:44:25.3070342Z pyhpc_isoneutral_mixing             PASS
2025-04-21T23:44:25.3075217Z pyhpc_turbulent_kinetic_energy      PASS
2025-04-21T23:44:25.3079769Z pytorch_CycleGAN_and_pix2pix        PASS
2025-04-21T23:44:25.3084288Z pytorch_stargan                     PASS
2025-04-21T23:44:25.3088933Z pytorch_unet                        PASS
2025-04-21T23:44:25.3093488Z resnet152                           PASS
2025-04-21T23:44:25.3098048Z resnet18                            PASS
2025-04-21T23:44:25.3102747Z resnet50                            PASS
2025-04-21T23:44:25.3107218Z resnet50_quantized_qat              XFAIL
2025-04-21T23:44:25.3111816Z resnext50_32x4d                     PASS
2025-04-21T23:44:25.3116518Z sam                                 PASS
2025-04-21T23:44:25.3121188Z sam_fast                            PASS
2025-04-21T23:44:25.3126248Z shufflenet_v2_x1_0                  PASS
2025-04-21T23:44:25.3130913Z soft_actor_critic                   PASS
2025-04-21T23:44:25.3135581Z speech_transformer                  PASS
2025-04-21T23:44:25.3139974Z squeezenet1_1                       PASS
2025-04-21T23:44:25.3144574Z stable_diffusion_text_encoder       PASS
2025-04-21T23:44:25.3149109Z stable_diffusion_unet               XFAIL
2025-04-21T23:44:25.3153787Z timm_efficientnet                   PASS
2025-04-21T23:44:25.3158553Z timm_regnet                         PASS
2025-04-21T23:44:25.3163076Z timm_resnest                        PASS
2025-04-21T23:44:25.3167765Z timm_vision_transformer             PASS
2025-04-21T23:44:25.3172250Z timm_vision_transformer_large       XFAIL
2025-04-21T23:44:25.3176774Z timm_vovnet                         PASS
2025-04-21T23:44:25.3181374Z torch_multimodal_clip               PASS
2025-04-21T23:44:25.3185951Z tts_angular                         PASS
2025-04-21T23:44:25.3190547Z vgg16                               PASS
2025-04-21T23:44:25.3195291Z vision_maskrcnn                     PASS
2025-04-21T23:44:25.3199869Z yolov3                              PASS
2025-04-21T23:44:25.3739130Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_inference.csv
2025-04-21T23:44:25.7035543Z llama_v2_7b_16h                     PASS
2025-04-21T23:44:25.7039521Z llava                               PASS
2025-04-21T23:44:25.7044694Z maml                                PASS
2025-04-21T23:44:25.7050453Z maml_omniglot                       PASS
2025-04-21T23:44:25.7055820Z microbench_unbacked_tolist_sum      PASS
2025-04-21T23:44:25.7061073Z mnasnet1_0                          PASS
2025-04-21T23:44:25.7066424Z mobilenet_v2                        PASS
2025-04-21T23:44:25.7071574Z mobilenet_v2_quantized_qat          PASS
2025-04-21T23:44:25.7077078Z mobilenet_v3_large                  PASS
2025-04-21T23:44:25.7082323Z moco                                PASS
2025-04-21T23:44:25.7087569Z moondream                           PASS
2025-04-21T23:44:25.7092804Z nanogpt                             PASS
2025-04-21T23:44:25.7097913Z nvidia_deeprecommender              PASS
2025-04-21T23:44:25.7103208Z opacus_cifar10                      PASS
2025-04-21T23:44:25.7108435Z phlippe_densenet                    PASS
2025-04-21T23:44:25.7113813Z phlippe_resnet                      PASS
2025-04-21T23:44:25.7119177Z pyhpc_equation_of_state             PASS
2025-04-21T23:44:25.7125094Z pyhpc_isoneutral_mixing             PASS
2025-04-21T23:44:25.7130306Z pyhpc_turbulent_kinetic_energy      PASS
2025-04-21T23:44:25.7135683Z pytorch_CycleGAN_and_pix2pix        PASS
2025-04-21T23:44:25.7140716Z pytorch_stargan                     PASS
2025-04-21T23:44:25.7145958Z pytorch_unet                        PASS
2025-04-21T23:44:25.7151087Z resnet152                           PASS
2025-04-21T23:44:25.7156661Z resnet18                            PASS
2025-04-21T23:44:25.7161754Z resnet50                            PASS
2025-04-21T23:44:25.7167023Z resnet50_quantized_qat              PASS
2025-04-21T23:44:25.7172149Z resnext50_32x4d                     PASS
2025-04-21T23:44:25.7177382Z sam                                 PASS
2025-04-21T23:44:25.7182652Z sam_fast                            PASS
2025-04-21T23:44:25.7187884Z shufflenet_v2_x1_0                  PASS
2025-04-21T23:44:25.7193089Z soft_actor_critic                   PASS
2025-04-21T23:44:25.7198722Z speech_transformer                  PASS
2025-04-21T23:44:25.7203823Z squeezenet1_1                       PASS
2025-04-21T23:44:25.7209066Z stable_diffusion_text_encoder       PASS
2025-04-21T23:44:25.7214203Z stable_diffusion_unet               PASS
2025-04-21T23:44:25.7219467Z timm_efficientnet                   PASS
2025-04-21T23:44:25.7225852Z timm_regnet                         PASS
2025-04-21T23:44:25.7230977Z timm_resnest                        PASS
2025-04-21T23:44:25.7236421Z timm_vision_transformer             PASS
2025-04-21T23:44:25.7241643Z timm_vision_transformer_large       PASS
2025-04-21T23:44:25.7246950Z timm_vovnet                         PASS
2025-04-21T23:44:25.7252031Z torch_multimodal_clip               PASS
2025-04-21T23:44:25.7257520Z tts_angular                         PASS
2025-04-21T23:44:25.7262733Z vgg16                               PASS
2025-04-21T23:44:25.7267934Z vision_maskrcnn                     PASS
2025-04-21T23:44:25.7273309Z yolov3                              PASS_BUT_FLAKY
2025-04-21T23:44:25.7808119Z + test_single_dynamo_benchmark training torchbench 1 --training --amp
2025-04-21T23:44:25.7810866Z ++ pwd
2025-04-21T23:44:25.7813703Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports
2025-04-21T23:44:25.7814170Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports
2025-04-21T23:44:25.7840915Z + local name=training
2025-04-21T23:44:25.7841152Z + shift
2025-04-21T23:44:25.7841382Z + local suite=torchbench
2025-04-21T23:44:25.7841626Z + shift
2025-04-21T23:44:25.7841817Z + local shard_id=1
2025-04-21T23:44:25.7842040Z + shift
2025-04-21T23:44:25.7842245Z + partition_flags=()
2025-04-21T23:44:25.7842507Z + local partition_flags
2025-04-21T23:44:25.7842750Z + [[ -n 2 ]]
2025-04-21T23:44:25.7843011Z + [[ -n 1 ]]
2025-04-21T23:44:25.7843409Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id")
2025-04-21T23:44:25.7843914Z + [[ inductor_torchbench == *perf_compare* ]]
2025-04-21T23:44:25.7844266Z + [[ inductor_torchbench == *perf* ]]
2025-04-21T23:44:25.7844580Z + [[ inductor_torchbench == *_avx2* ]]
2025-04-21T23:44:25.7844889Z + [[ inductor_torchbench == *_avx512* ]]
2025-04-21T23:44:25.7846047Z + python benchmarks/dynamo/torchbench.py --ci --accuracy --timing --explain --print-compilation-time --inductor --device cuda --training --amp --total-partitions 2 --partition-id 1 --output /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv
2025-04-21T23:44:32.3576061Z 
2025-04-21T23:44:32.5334548Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:32.5335021Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:32.5335478Z cuda train llava                              
2025-04-21T23:44:32.5341722Z Traceback (most recent call last):
2025-04-21T23:44:32.5342418Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:44:32.5343055Z     ) = runner.load_model(
2025-04-21T23:44:32.5343536Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 321, in load_model
2025-04-21T23:44:32.5344084Z     benchmark = benchmark_cls(
2025-04-21T23:44:32.5344604Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 43, in __call__
2025-04-21T23:44:32.5345164Z     obj = type.__call__(cls, *args, **kwargs)
2025-04-21T23:44:32.5345772Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/llava/__init__.py", line 12, in __init__
2025-04-21T23:44:32.5346361Z     super().__init__(
2025-04-21T23:44:32.5346994Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/framework/huggingface/model_factory.py", line 32, in __init__
2025-04-21T23:44:32.5347686Z     super().__init__(
2025-04-21T23:44:32.5348177Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 110, in __init__
2025-04-21T23:44:32.5348724Z     self._skip_by_device_name()
2025-04-21T23:44:32.5349300Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/util/model.py", line 202, in _skip_by_device_name
2025-04-21T23:44:32.5350188Z     raise NotImplementedError(
2025-04-21T23:44:32.5350672Z NotImplementedError: The current device NVIDIA A10G is skipped by its `llava/metadata.yaml`.
2025-04-21T23:44:32.5351079Z 
2025-04-21T23:44:32.5351168Z model_fail_to_load
2025-04-21T23:44:36.6368028Z 
2025-04-21T23:44:37.3871990Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:37.3872346Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:37.3872659Z cuda train maml_omniglot                      
2025-04-21T23:44:46.7042345Z W0421 23:44:46.703000 64289 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:44:51.2469193Z Compilation time (from dynamo_timed): 11.767931285
2025-04-21T23:44:51.2477970Z pass
2025-04-21T23:44:51.2479396Z TIMING: entire_frame_compile:7.79088 gc:0.00315 _recursive_pre_grad_passes:0.00846 pad_mm_benchmark:0.47694 _recursive_joint_graph_passes:0.64344 _recursive_post_grad_passes:0.21128 async_compile.wait:3.81263 code_gen:6.97865 inductor_compile:8.11224 backend_compile:6.68113 cudagraphify.get_container:0.16768 entire_backward_compile:3.97705 CUDAGraphNode.record:0.59824 total_wall_time:11.76793
2025-04-21T23:44:51.2481359Z STATS: call_* op count: 107 | FakeTensorMode.__torch_dispatch__:4298 | ProxyTorchDispatchMode.__torch_dispatch__:1641 | FakeTensor.__torch_dispatch__:2437
2025-04-21T23:44:51.2482181Z Dynamo produced 3 graphs covering 107 ops with 7 graph breaks (5 unique)
2025-04-21T23:44:56.6912176Z 
2025-04-21T23:44:57.0193184Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:57.0193577Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:44:57.0193915Z cuda train microbench_unbacked_tolist_sum     
2025-04-21T23:45:01.7821171Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] Graph break from `Tensor.item()`, consider setting:
2025-04-21T23:45:01.7822335Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-21T23:45:01.7823185Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] or:
2025-04-21T23:45:01.7823971Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-21T23:45:01.7824923Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] to include these operations in the captured graph.
2025-04-21T23:45:01.7825706Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 
2025-04-21T23:45:01.7826450Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] Graph break: from user code at:
2025-04-21T23:45:01.7827719Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-21T23:45:01.7828973Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]     pred = mod(*cloned_inputs)
2025-04-21T23:45:01.7830239Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/microbench_unbacked_tolist_sum/__init__.py", line 18, in forward
2025-04-21T23:45:01.7831482Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0]     sum += tv[i].item()
2025-04-21T23:45:01.7832181Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 
2025-04-21T23:45:01.7832799Z W0421 23:45:01.781000 64935 site-packages/torch/_dynamo/variables/tensor.py:928] [2/0] 
2025-04-21T23:45:06.1350942Z W0421 23:45:06.134000 64935 site-packages/torch/_logging/_internal.py:1130] [9/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:45:07.5818287Z Compilation time (from dynamo_timed): 8.703898764
2025-04-21T23:45:07.5819066Z pass
2025-04-21T23:45:07.5820390Z TIMING: entire_frame_compile:8.7039 gc:0.0041 _recursive_pre_grad_passes:0.0068 _recursive_joint_graph_passes:0.34714 inductor_compile:0.60431 backend_compile:1.78157 _recursive_post_grad_passes:0.01808 async_compile.wait:0.43063 code_gen:0.48933 cudagraphify.get_container:0.27535 CUDAGraphNode.record:0.26735 total_wall_time:8.7039
2025-04-21T23:45:07.5822462Z STATS: call_* op count: 30 | FakeTensor.__torch_dispatch__:2134 | FakeTensorMode.__torch_dispatch__:10522 | ProxyTorchDispatchMode.__torch_dispatch__:93
2025-04-21T23:45:07.5823262Z Dynamo produced 4 graphs covering 30 ops with 8 graph breaks (6 unique)
2025-04-21T23:45:13.2326512Z 
2025-04-21T23:45:15.6954986Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:45:15.6955348Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:45:15.6955654Z cuda train mnasnet1_0                         
2025-04-21T23:45:50.1846738Z W0421 23:45:50.183000 65070 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:46:24.3162587Z Compilation time (from dynamo_timed): 60.751465587
2025-04-21T23:46:24.3184731Z pass
2025-04-21T23:46:24.3214371Z TIMING: entire_frame_compile:49.30425 gc:0.00295 _recursive_pre_grad_passes:0.02812 pad_mm_benchmark:0.20818 _recursive_joint_graph_passes:0.63053 _recursive_post_grad_passes:0.44174 async_compile.wait:6.34886 code_gen:20.31477 inductor_compile:31.62998 backend_compile:36.52198 cudagraphify.get_container:0.29233 entire_backward_compile:11.44722 CUDAGraphNode.record:2.1037 total_wall_time:60.75147
2025-04-21T23:46:24.3216331Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:42017 | ProxyTorchDispatchMode.__torch_dispatch__:18737 | FakeTensor.__torch_dispatch__:29755
2025-04-21T23:46:24.3217159Z Dynamo produced 3 graphs covering 965 ops with 7 graph breaks (5 unique)
2025-04-21T23:46:31.8206130Z 
2025-04-21T23:46:33.8023406Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:46:33.8023781Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:46:33.8024097Z cuda train mobilenet_v2                       
2025-04-21T23:47:07.5276149Z Compilation time (from dynamo_timed): 28.777914767
2025-04-21T23:47:07.5297147Z pass
2025-04-21T23:47:07.5349052Z TIMING: entire_frame_compile:17.80213 gc:0.00214 _recursive_pre_grad_passes:0.00791 pad_mm_benchmark:0.0369 _recursive_joint_graph_passes:0.33435 _recursive_post_grad_passes:0.44762 async_compile.wait:0.8658 code_gen:8.8175 inductor_compile:15.37277 backend_compile:11.94935 cudagraphify.get_container:0.29798 entire_backward_compile:10.97578 CUDAGraphNode.record:0.8466 total_wall_time:28.77791
2025-04-21T23:47:07.5350981Z STATS: call_* op count: 157 | FakeTensorMode.__torch_dispatch__:24208 | FakeTensor.__torch_dispatch__:16975 | ProxyTorchDispatchMode.__torch_dispatch__:10380
2025-04-21T23:47:13.8643360Z Dynamo produced 2 graphs covering 157 ops with 6 graph breaks (5 unique)
2025-04-21T23:47:13.8643717Z 
2025-04-21T23:47:15.7991702Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:47:15.7992214Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:47:15.7992668Z cuda train mobilenet_v2_quantized_qat         
2025-04-21T23:47:15.7993094Z Traceback (most recent call last):
2025-04-21T23:47:15.7993665Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model
2025-04-21T23:47:15.7994211Z     self.model_iter_fn(model, example_inputs)
2025-04-21T23:47:15.7994824Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in forward_and_backward_pass
2025-04-21T23:47:15.7995422Z     pred = mod(*cloned_inputs)
2025-04-21T23:47:15.7995994Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 833, in call_wrapped
2025-04-21T23:47:15.7996618Z     return self._wrapped_call(self, *args, **kwargs)
2025-04-21T23:47:15.7997209Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 409, in __call__
2025-04-21T23:47:15.7998121Z     raise e
2025-04-21T23:47:15.7998584Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 396, in __call__
2025-04-21T23:47:15.7999260Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2025-04-21T23:47:15.7999997Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-04-21T23:47:15.8000631Z     return self._call_impl(*args, **kwargs)
2025-04-21T23:47:15.8001211Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-04-21T23:47:15.8001802Z     return forward_call(*args, **kwargs)
2025-04-21T23:47:15.8002317Z   File "<eval_with_key>.3", line 207, in forward
2025-04-21T23:47:15.8002852Z     activation_post_process_101 = self.activation_post_process_101(classifier_1);  classifier_1 = None
2025-04-21T23:47:15.8003684Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-04-21T23:47:15.8004325Z     return self._call_impl(*args, **kwargs)
2025-04-21T23:47:15.8004919Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-04-21T23:47:15.8005506Z     return forward_call(*args, **kwargs)
2025-04-21T23:47:15.8006129Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward
2025-04-21T23:47:15.8006789Z     return torch.fused_moving_avg_obs_fake_quant(
2025-04-21T23:47:15.8007189Z RuntimeError: expected scalar type Float but found Half
2025-04-21T23:47:15.8007468Z 
2025-04-21T23:47:15.8007683Z The above exception was the direct cause of the following exception:
2025-04-21T23:47:15.8008001Z 
2025-04-21T23:47:15.8008113Z Traceback (most recent call last):
2025-04-21T23:47:15.8008576Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:47:15.8009044Z     ) = runner.load_model(
2025-04-21T23:47:15.8009528Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model
2025-04-21T23:47:15.8010083Z     self.validate_model(model, example_inputs)
2025-04-21T23:47:15.8010632Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model
2025-04-21T23:47:15.8011183Z     raise RuntimeError("Eager run failed") from e
2025-04-21T23:47:15.8011530Z RuntimeError: Eager run failed
2025-04-21T23:47:15.8011713Z 
2025-04-21T23:47:15.8011818Z eager_fail_to_run
2025-04-21T23:47:20.0724898Z 
2025-04-21T23:47:22.4543299Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:47:22.4543654Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:47:22.4543994Z cuda train mobilenet_v3_large                 
2025-04-21T23:48:01.8410725Z W0421 23:48:01.840000 68058 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:48:39.4988331Z E0421 23:48:39.497000 68058 site-packages/torch/_dynamo/utils.py:2939] RMSE (res-fp64): 0.00287, (ref-fp64): 0.00018 and shape=torch.Size([168]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.001000, use_larger_multiplier_for_smaller_tensor: 0
2025-04-21T23:48:39.4990439Z E0421 23:48:39.498000 68058 site-packages/torch/_dynamo/utils.py:2800] Accuracy failed for key name features.13.block.2.fc1.bias.grad
2025-04-21T23:48:39.5232578Z Compilation time (from dynamo_timed): 68.864238674
2025-04-21T23:48:39.5255880Z pass
2025-04-21T23:48:39.5295443Z TIMING: entire_frame_compile:55.61216 gc:0.00469 _recursive_pre_grad_passes:0.0309 pad_mm_benchmark:0.21143 _recursive_joint_graph_passes:0.68558 _recursive_post_grad_passes:0.89962 async_compile.wait:4.30692 code_gen:22.04584 inductor_compile:35.88907 backend_compile:40.37065 cudagraphify.get_container:0.30621 entire_backward_compile:13.25208 CUDAGraphNode.record:2.42389 total_wall_time:68.86424
2025-04-21T23:48:39.5298524Z STATS: call_* op count: 1080 | FakeTensorMode.__torch_dispatch__:47611 | FakeTensor.__torch_dispatch__:36006 | ProxyTorchDispatchMode.__torch_dispatch__:20796
2025-04-21T23:48:39.5300364Z Dynamo produced 3 graphs covering 1080 ops with 7 graph breaks (5 unique)
2025-04-21T23:48:47.4917713Z 
2025-04-21T23:48:49.8324480Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:48:49.8324830Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:48:49.8325136Z cuda train moco                               
2025-04-21T23:48:55.3396387Z [rank0]:W0421 23:48:55.338000 70134 site-packages/torch/_logging/_internal.py:1130] [2/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:49:04.8146086Z [rank0]:W0421 23:49:04.813000 70134 site-packages/torch/_dynamo/backends/distributed.py:106] [6/0_1] Some buckets were extended beyond their requested parameter capacities in order to ensure each subgraph has an output node, required for fx graph partitioning. This can be the case when a subgraph would have only contained nodes performing inplace mutation, and returning no logical outputs. This should not be a problem, unless it results in too few graph partitions for optimal DDP performance.
2025-04-21T23:49:04.8401452Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] DDPOptimizer extended these buckets to ensure per-subgraph output nodes:
2025-04-21T23:49:04.8402959Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] ┌─────────┬─────────────┬────────────────────────┐
2025-04-21T23:49:04.8404093Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] │   Index │   Extra Ops │   Extra Param Size (b) │
2025-04-21T23:49:04.8405256Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] ├─────────┼─────────────┼────────────────────────┤
2025-04-21T23:49:04.8406352Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] │       0 │         480 │               94032128 │
2025-04-21T23:49:04.8407493Z [rank0]:W0421 23:49:04.839000 70134 site-packages/torch/_dynamo/backends/distributed.py:123] [6/0_1] └─────────┴─────────────┴────────────────────────┘
2025-04-21T23:49:26.6118390Z skipping cudagraphs due to mutated inputs (161 instances)
2025-04-21T23:49:29.4088325Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] Graph break from `Tensor.item()`, consider setting:
2025-04-21T23:49:29.4089398Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-21T23:49:29.4090290Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] or:
2025-04-21T23:49:29.4091111Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-21T23:49:29.4092092Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] to include these operations in the captured graph.
2025-04-21T23:49:29.4092925Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] 
2025-04-21T23:49:29.4093696Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] Graph break: from user code at:
2025-04-21T23:49:29.4095050Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 167, in torch_dynamo_resume_in_forward_at_142
2025-04-21T23:49:29.4096419Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]     self._dequeue_and_enqueue(k)
2025-04-21T23:49:29.4097626Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
2025-04-21T23:49:29.4099206Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]     return func(*args, **kwargs)
2025-04-21T23:49:29.4100474Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/moco/moco/builder.py", line 68, in _dequeue_and_enqueue
2025-04-21T23:49:29.4101737Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0]     ptr = int(self.queue_ptr)
2025-04-21T23:49:29.4102698Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] 
2025-04-21T23:49:29.4103339Z [rank0]:W0421 23:49:29.407000 70134 site-packages/torch/_dynamo/variables/tensor.py:928] [9/0] 
2025-04-21T23:50:01.3548536Z Compilation time (from dynamo_timed): 59.700333423
2025-04-21T23:50:01.3600300Z pass
2025-04-21T23:50:01.3944391Z TIMING: entire_frame_compile:46.63414 gc:0.00834 _recursive_pre_grad_passes:0.108 _recursive_joint_graph_passes:0.72374 inductor_compile:27.31255 backend_compile:30.3757 _recursive_post_grad_passes:0.63738 async_compile.wait:3.63984 code_gen:16.78044 pad_mm_benchmark:0.2932 cudagraphify.get_container:0.44741 entire_backward_compile:13.06619 CUDAGraphNode.record:4.88356 total_wall_time:59.70033
2025-04-21T23:50:01.3948257Z STATS: call_* op count: 887 | FakeTensorMode.__torch_dispatch__:48543 | ProxyTorchDispatchMode.__torch_dispatch__:20264 | FakeTensor.__torch_dispatch__:26615
2025-04-21T23:50:01.3949892Z Dynamo produced 10 graphs covering 887 ops with 13 graph breaks (9 unique)
2025-04-21T23:50:03.6084327Z [rank0]:[W421 23:50:03.595692483 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
2025-04-21T23:50:09.4991634Z 
2025-04-21T23:50:11.3925879Z loading model: 0it [00:00, ?it/s]number of parameters: 123.69M
2025-04-21T23:50:11.6398016Z num decayed parameter tensors: 50, with 124,354,560 parameters
2025-04-21T23:50:11.6398501Z num non-decayed parameter tensors: 98, with 121,344 parameters
2025-04-21T23:50:11.6400012Z using fused AdamW: True
2025-04-21T23:50:12.2005969Z 
2025-04-21T23:50:12.2006491Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:50:12.2006862Z cuda train nanogpt                            
2025-04-21T23:50:36.5735209Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-21T23:50:36.5736071Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-21T23:50:36.5736766Z     pred = mod(*cloned_inputs)
2025-04-21T23:50:36.5737343Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/nanogpt/model.py", line 229, in forward
2025-04-21T23:50:36.5738069Z     tok_emb = self.transformer.wte(idx)  # token embeddings of shape (b, t, n_embd)
2025-04-21T23:50:36.5738436Z 
2025-04-21T23:50:36.6577038Z W0421 23:50:36.656000 71995 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:51:04.3466663Z Compilation time (from dynamo_timed): 47.90818445
2025-04-21T23:51:04.3485094Z pass
2025-04-21T23:51:04.4104084Z TIMING: entire_frame_compile:41.05301 gc:0.00473 _recursive_pre_grad_passes:0.02817 pad_mm_benchmark:0.29304 _recursive_joint_graph_passes:1.32067 _recursive_post_grad_passes:0.59093 async_compile.wait:3.8468 code_gen:15.34542 inductor_compile:25.54341 backend_compile:32.61142 cudagraphify.get_container:0.3117 entire_backward_compile:6.85517 CUDAGraphNode.record:1.1918 total_wall_time:47.90818
2025-04-21T23:51:04.4106036Z STATS: call_* op count: 1156 | FakeTensorMode.__torch_dispatch__:47484 | FakeTensor.__torch_dispatch__:28154 | ProxyTorchDispatchMode.__torch_dispatch__:20485
2025-04-21T23:51:04.4106872Z Dynamo produced 3 graphs covering 1156 ops with 7 graph breaks (5 unique)
2025-04-21T23:51:11.5805079Z 
2025-04-21T23:51:14.1247734Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:51:14.1248092Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:51:14.1248414Z cuda train nvidia_deeprecommender             
2025-04-21T23:51:20.1297091Z W0421 23:51:20.128000 72648 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:51:24.3104121Z Compilation time (from dynamo_timed): 7.415830237000001
2025-04-21T23:51:24.3104494Z pass
2025-04-21T23:51:24.3396534Z TIMING: entire_frame_compile:6.78542 gc:0.00266 _recursive_pre_grad_passes:0.00797 pad_mm_benchmark:0.98436 _recursive_joint_graph_passes:1.19065 _recursive_post_grad_passes:0.22384 async_compile.wait:1.73809 code_gen:2.62328 inductor_compile:3.6586 backend_compile:5.88062 cudagraphify.get_container:0.17682 entire_backward_compile:0.63041 CUDAGraphNode.record:0.56535 total_wall_time:7.41583
2025-04-21T23:51:24.3398490Z STATS: call_* op count: 96 | FakeTensorMode.__torch_dispatch__:4093 | ProxyTorchDispatchMode.__torch_dispatch__:1596 | FakeTensor.__torch_dispatch__:2400
2025-04-21T23:51:24.3399311Z Dynamo produced 3 graphs covering 96 ops with 7 graph breaks (5 unique)
2025-04-21T23:51:29.7017011Z 
2025-04-21T23:51:30.8416499Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:51:30.8416836Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:51:30.8417148Z cuda train opacus_cifar10                     
2025-04-21T23:51:30.8432990Z Traceback (most recent call last):
2025-04-21T23:51:30.8433768Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model
2025-04-21T23:51:30.8434361Z     self.model_iter_fn(model, example_inputs)
2025-04-21T23:51:30.8434980Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 471, in forward_and_backward_pass
2025-04-21T23:51:30.8435593Z     self.grad_scaler.scale(loss).backward()
2025-04-21T23:51:30.8436130Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward
2025-04-21T23:51:30.8436673Z     torch.autograd.backward(
2025-04-21T23:51:30.8437214Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 354, in backward
2025-04-21T23:51:30.8437777Z     _engine_run_backward(
2025-04-21T23:51:30.8438337Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 824, in _engine_run_backward
2025-04-21T23:51:30.8439158Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-04-21T23:51:30.8439945Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 88, in __call__
2025-04-21T23:51:30.8440519Z     return self.hook(module, *args, **kwargs)
2025-04-21T23:51:30.8441207Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 327, in capture_backprops_hook
2025-04-21T23:51:30.8441962Z     activations, backprops = self.rearrange_grad_samples(
2025-04-21T23:51:30.8442691Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/opacus/grad_sample/grad_sample_module.py", line 383, in rearrange_grad_samples
2025-04-21T23:51:30.8443355Z     raise ValueError(
2025-04-21T23:51:30.8443891Z ValueError: No activations detected for <class 'torch.nn.modules.linear.Linear'>, run forward after add_hooks(model)
2025-04-21T23:51:30.8444377Z 
2025-04-21T23:51:30.8444595Z The above exception was the direct cause of the following exception:
2025-04-21T23:51:30.8444903Z 
2025-04-21T23:51:30.8445022Z Traceback (most recent call last):
2025-04-21T23:51:30.8445483Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:51:30.8445955Z     ) = runner.load_model(
2025-04-21T23:51:30.8446427Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model
2025-04-21T23:51:30.8446965Z     self.validate_model(model, example_inputs)
2025-04-21T23:51:30.8447808Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model
2025-04-21T23:51:30.8448356Z     raise RuntimeError("Eager run failed") from e
2025-04-21T23:51:30.8448693Z RuntimeError: Eager run failed
2025-04-21T23:51:30.8448872Z 
2025-04-21T23:51:30.8448959Z eager_fail_to_run
2025-04-21T23:51:35.1081020Z 
2025-04-21T23:51:36.1083966Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:51:36.1084641Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:51:36.1085315Z cuda train phlippe_densenet                   
2025-04-21T23:52:07.2016167Z Compilation time (from dynamo_timed): 28.336575082
2025-04-21T23:52:07.2037002Z pass
2025-04-21T23:52:07.2045834Z TIMING: entire_frame_compile:19.63656 gc:0.00282 _recursive_pre_grad_passes:0.00944 pad_mm_benchmark:0.20896 _recursive_joint_graph_passes:0.52121 _recursive_post_grad_passes:0.3738 async_compile.wait:0.53025 code_gen:8.92358 inductor_compile:14.17618 backend_compile:13.44564 cudagraphify.get_container:0.301 entire_backward_compile:8.70001 CUDAGraphNode.record:0.7888 total_wall_time:28.33658
2025-04-21T23:52:07.2047828Z STATS: call_* op count: 190 | FakeTensorMode.__torch_dispatch__:24797 | ProxyTorchDispatchMode.__torch_dispatch__:11251 | FakeTensor.__torch_dispatch__:14285
2025-04-21T23:52:07.2048649Z Dynamo produced 2 graphs covering 190 ops with 6 graph breaks (5 unique)
2025-04-21T23:52:13.3235417Z 
2025-04-21T23:52:14.1667326Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:52:14.1667681Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:52:14.1667998Z cuda train phlippe_resnet                     
2025-04-21T23:52:26.2406802Z Compilation time (from dynamo_timed): 10.009754507
2025-04-21T23:52:26.2413335Z pass
2025-04-21T23:52:26.2415390Z TIMING: entire_frame_compile:7.1198 gc:0.00189 _recursive_pre_grad_passes:0.00527 pad_mm_benchmark:0.2131 _recursive_joint_graph_passes:0.41621 _recursive_post_grad_passes:0.14878 async_compile.wait:0.70713 code_gen:2.66848 inductor_compile:4.71428 backend_compile:5.11262 cudagraphify.get_container:0.19819 entire_backward_compile:2.88995 CUDAGraphNode.record:0.44689 total_wall_time:10.00975
2025-04-21T23:52:26.2417471Z STATS: call_* op count: 75 | FakeTensorMode.__torch_dispatch__:9126 | ProxyTorchDispatchMode.__torch_dispatch__:4232 | FakeTensor.__torch_dispatch__:5296
2025-04-21T23:52:26.2426519Z Dynamo produced 2 graphs covering 75 ops with 6 graph breaks (5 unique)
2025-04-21T23:52:31.7865994Z 
2025-04-21T23:52:33.0038359Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:52:33.0038715Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:52:33.0039050Z cuda train pytorch_CycleGAN_and_pix2pix       
2025-04-21T23:53:02.2646626Z Compilation time (from dynamo_timed): 19.03681279
2025-04-21T23:53:02.2651494Z pass
2025-04-21T23:53:02.2749794Z TIMING: entire_frame_compile:10.43297 gc:0.00245 _recursive_pre_grad_passes:0.0049 _recursive_joint_graph_passes:0.39762 _recursive_post_grad_passes:0.29255 async_compile.wait:1.83653 code_gen:7.94273 inductor_compile:11.84657 backend_compile:8.59031 cudagraphify.get_container:0.28606 CachingAutotuner.benchmark_all_configs:0.60046 entire_backward_compile:8.60385 CUDAGraphNode.record:0.6203 total_wall_time:19.03681
2025-04-21T23:53:02.2751796Z STATS: call_* op count: 95 | FakeTensorMode.__torch_dispatch__:14498 | ProxyTorchDispatchMode.__torch_dispatch__:7141 | FakeTensor.__torch_dispatch__:10665
2025-04-21T23:53:02.2752613Z Dynamo produced 2 graphs covering 95 ops with 6 graph breaks (5 unique)
2025-04-21T23:53:08.1791446Z 
2025-04-21T23:53:09.6785015Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:53:09.6785414Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:53:09.6785742Z cuda train pytorch_stargan                    
2025-04-21T23:53:48.3796235Z Compilation time (from dynamo_timed): 11.331705534000001
2025-04-21T23:53:48.3804701Z pass
2025-04-21T23:53:48.3887066Z TIMING: entire_frame_compile:8.21521 gc:0.00243 _recursive_pre_grad_passes:0.00503 _recursive_joint_graph_passes:0.35876 _recursive_post_grad_passes:0.14433 async_compile.wait:0.78045 code_gen:3.27342 inductor_compile:5.63406 backend_compile:6.30542 cudagraphify.get_container:0.22418 entire_backward_compile:3.1165 CUDAGraphNode.record:0.47751 total_wall_time:11.33171
2025-04-21T23:53:48.3889225Z STATS: call_* op count: 60 | FakeTensorMode.__torch_dispatch__:12519 | FakeTensor.__torch_dispatch__:5664 | ProxyTorchDispatchMode.__torch_dispatch__:5363
2025-04-21T23:53:48.3890040Z Dynamo produced 2 graphs covering 60 ops with 6 graph breaks (5 unique)
2025-04-21T23:53:54.0073870Z 
2025-04-21T23:53:55.0351134Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:53:55.0351666Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:53:55.0352128Z cuda train pytorch_unet                       
2025-04-21T23:54:43.6784675Z W0421 23:54:43.677000 75818 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:55:00.3150041Z Compilation time (from dynamo_timed): 34.76489246
2025-04-21T23:55:00.3157666Z pass_due_to_skip
2025-04-21T23:55:00.3206050Z TIMING: entire_frame_compile:25.55572 gc:0.00412 _recursive_pre_grad_passes:0.01667 _recursive_joint_graph_passes:0.28247 _recursive_post_grad_passes:0.24624 async_compile.wait:2.84732 code_gen:15.26512 inductor_compile:21.10241 backend_compile:19.69181 cudagraphify.get_container:0.25287 CachingAutotuner.benchmark_all_configs:3.24574 entire_backward_compile:9.20917 CUDAGraphNode.record:1.19902 total_wall_time:34.76489
2025-04-21T23:55:00.3208063Z STATS: call_* op count: 464 | FakeTensorMode.__torch_dispatch__:21983 | ProxyTorchDispatchMode.__torch_dispatch__:9646 | FakeTensor.__torch_dispatch__:14877
2025-04-21T23:55:00.3208905Z Dynamo produced 3 graphs covering 464 ops with 7 graph breaks (5 unique)
2025-04-21T23:55:06.7044213Z 
2025-04-21T23:55:09.1009605Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:55:09.1010249Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:55:09.1010648Z cuda train resnet152                          
2025-04-21T23:56:25.8105556Z W0421 23:56:25.809000 77488 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-21T23:58:11.3606939Z Compilation time (from dynamo_timed): 164.399449513
2025-04-21T23:58:11.3667164Z pass
2025-04-21T23:58:11.4059812Z TIMING: entire_frame_compile:141.40589 gc:0.00562 _recursive_pre_grad_passes:0.07318 pad_mm_benchmark:0.21037 _recursive_joint_graph_passes:1.22379 _recursive_post_grad_passes:1.32744 async_compile.wait:5.48692 code_gen:49.56244 inductor_compile:80.69656 backend_compile:103.551 cudagraphify.get_container:0.64314 entire_backward_compile:22.99356 CUDAGraphNode.record:6.27484 total_wall_time:164.39945
2025-04-21T23:58:11.4061871Z STATS: call_* op count: 2873 | FakeTensorMode.__torch_dispatch__:120899 | ProxyTorchDispatchMode.__torch_dispatch__:56937 | FakeTensor.__torch_dispatch__:88655
2025-04-21T23:58:11.4062748Z Dynamo produced 3 graphs covering 2873 ops with 7 graph breaks (5 unique)
2025-04-21T23:58:23.3422797Z 
2025-04-21T23:58:24.7067250Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:58:24.7067717Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:58:24.7068143Z cuda train resnet18                           
2025-04-21T23:58:41.5052358Z Compilation time (from dynamo_timed): 13.231153767
2025-04-21T23:58:41.5060305Z pass
2025-04-21T23:58:41.5133877Z TIMING: entire_frame_compile:7.8213 gc:0.00234 _recursive_pre_grad_passes:0.00547 pad_mm_benchmark:0.21966 _recursive_joint_graph_passes:0.42179 _recursive_post_grad_passes:0.14883 async_compile.wait:0.53199 code_gen:5.27105 inductor_compile:7.79107 backend_compile:5.84167 cudagraphify.get_container:0.20798 entire_backward_compile:5.40986 CUDAGraphNode.record:0.49365 total_wall_time:13.23115
2025-04-21T23:58:41.5135826Z STATS: call_* op count: 73 | FakeTensorMode.__torch_dispatch__:9366 | ProxyTorchDispatchMode.__torch_dispatch__:4228 | FakeTensor.__torch_dispatch__:5367
2025-04-21T23:58:41.5136654Z Dynamo produced 2 graphs covering 73 ops with 6 graph breaks (5 unique)
2025-04-21T23:58:47.1994914Z 
2025-04-21T23:58:49.0429113Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:58:49.0429586Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:58:49.0430010Z cuda train resnet50                           
2025-04-21T23:59:22.6000290Z Compilation time (from dynamo_timed): 26.85579008
2025-04-21T23:59:22.6020878Z pass
2025-04-21T23:59:22.6192925Z TIMING: entire_frame_compile:17.30846 gc:0.00224 _recursive_pre_grad_passes:0.00826 pad_mm_benchmark:0.03763 _recursive_joint_graph_passes:0.33675 _recursive_post_grad_passes:0.3601 async_compile.wait:0.22977 code_gen:8.51966 inductor_compile:13.82514 backend_compile:11.75488 cudagraphify.get_container:0.29855 entire_backward_compile:9.54733 CUDAGraphNode.record:0.79382 total_wall_time:26.85579
2025-04-21T23:59:22.6194948Z STATS: call_* op count: 179 | FakeTensorMode.__torch_dispatch__:23391 | ProxyTorchDispatchMode.__torch_dispatch__:11015 | FakeTensor.__torch_dispatch__:13979
2025-04-21T23:59:22.6195788Z Dynamo produced 2 graphs covering 179 ops with 6 graph breaks (5 unique)
2025-04-21T23:59:28.9370720Z 
2025-04-21T23:59:31.1781622Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:59:31.1781975Z loading model: 0it [00:02, ?it/s]
2025-04-21T23:59:31.1782311Z cuda train resnet50_quantized_qat             
2025-04-21T23:59:31.1782768Z Traceback (most recent call last):
2025-04-21T23:59:31.1783361Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model
2025-04-21T23:59:31.1783967Z     self.model_iter_fn(model, example_inputs)
2025-04-21T23:59:31.1784673Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in forward_and_backward_pass
2025-04-21T23:59:31.1785353Z     pred = mod(*cloned_inputs)
2025-04-21T23:59:31.1786017Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 833, in call_wrapped
2025-04-21T23:59:31.1786784Z     return self._wrapped_call(self, *args, **kwargs)
2025-04-21T23:59:31.1787408Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 409, in __call__
2025-04-21T23:59:31.1788048Z     raise e
2025-04-21T23:59:31.1788600Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/fx/graph_module.py", line 396, in __call__
2025-04-21T23:59:31.1789460Z     return super(self.cls, obj).__call__(*args, **kwargs)  # type: ignore[misc]
2025-04-21T23:59:31.1790274Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-04-21T23:59:31.1790941Z     return self._call_impl(*args, **kwargs)
2025-04-21T23:59:31.1791614Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-04-21T23:59:31.1792311Z     return forward_call(*args, **kwargs)
2025-04-21T23:59:31.1792687Z   File "<eval_with_key>.3", line 167, in forward
2025-04-21T23:59:31.1793180Z     activation_post_process_73 = self.activation_post_process_73(fc);  fc = None
2025-04-21T23:59:31.1794138Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
2025-04-21T23:59:31.1794922Z     return self._call_impl(*args, **kwargs)
2025-04-21T23:59:31.1795564Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
2025-04-21T23:59:31.1796175Z     return forward_call(*args, **kwargs)
2025-04-21T23:59:31.1796871Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/ao/quantization/fake_quantize.py", line 408, in forward
2025-04-21T23:59:31.1797595Z     return torch.fused_moving_avg_obs_fake_quant(
2025-04-21T23:59:31.1798019Z RuntimeError: expected scalar type Float but found Half
2025-04-21T23:59:31.1798350Z 
2025-04-21T23:59:31.1798561Z The above exception was the direct cause of the following exception:
2025-04-21T23:59:31.1798920Z 
2025-04-21T23:59:31.1799073Z Traceback (most recent call last):
2025-04-21T23:59:31.1799541Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-21T23:59:31.1800465Z     ) = runner.load_model(
2025-04-21T23:59:31.1801026Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model
2025-04-21T23:59:31.1801652Z     self.validate_model(model, example_inputs)
2025-04-21T23:59:31.1802196Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model
2025-04-21T23:59:31.1802821Z     raise RuntimeError("Eager run failed") from e
2025-04-21T23:59:31.1803201Z RuntimeError: Eager run failed
2025-04-21T23:59:31.1803428Z 
2025-04-21T23:59:31.1803528Z eager_fail_to_run
2025-04-21T23:59:35.4649503Z 
2025-04-21T23:59:37.4616547Z loading model: 0it [00:00, ?it/s]
2025-04-21T23:59:37.4617751Z loading model: 0it [00:01, ?it/s]
2025-04-21T23:59:37.4618389Z cuda train resnext50_32x4d                    
2025-04-22T00:00:10.8311207Z W0422 00:00:10.830000 79468 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:00:44.7875643Z Compilation time (from dynamo_timed): 58.249861228
2025-04-22T00:00:44.7896983Z pass
2025-04-22T00:00:44.8023842Z TIMING: entire_frame_compile:48.2055 gc:0.00593 _recursive_pre_grad_passes:0.02917 pad_mm_benchmark:0.03651 _recursive_joint_graph_passes:0.47397 _recursive_post_grad_passes:0.45524 async_compile.wait:4.33469 code_gen:18.90163 inductor_compile:29.48094 backend_compile:35.62095 cudagraphify.get_container:0.2949 entire_backward_compile:10.04436 CachingAutotuner.benchmark_all_configs:0.05649 CUDAGraphNode.record:2.07745 total_wall_time:58.24986
2025-04-22T00:00:44.8029010Z STATS: call_* op count: 1003 | FakeTensorMode.__torch_dispatch__:42066 | ProxyTorchDispatchMode.__torch_dispatch__:19571 | FakeTensor.__torch_dispatch__:30507
2025-04-22T00:00:44.8030400Z Dynamo produced 3 graphs covering 1003 ops with 7 graph breaks (5 unique)
2025-04-22T00:00:52.2924789Z 
2025-04-22T00:01:00.1130081Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:01:00.1130449Z loading model: 0it [00:07, ?it/s]
2025-04-22T00:01:00.1130798Z cuda train sam                                
2025-04-22T00:01:00.1141352Z Traceback (most recent call last):
2025-04-22T00:01:00.1142057Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1902, in validate_model
2025-04-22T00:01:00.1142616Z     self.model_iter_fn(model, example_inputs)
2025-04-22T00:01:00.1143260Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 471, in forward_and_backward_pass
2025-04-22T00:01:00.1143875Z     self.grad_scaler.scale(loss).backward()
2025-04-22T00:01:00.1144427Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_tensor.py", line 648, in backward
2025-04-22T00:01:00.1144973Z     torch.autograd.backward(
2025-04-22T00:01:00.1145517Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/__init__.py", line 354, in backward
2025-04-22T00:01:00.1146092Z     _engine_run_backward(
2025-04-22T00:01:00.1146657Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/autograd/graph.py", line 824, in _engine_run_backward
2025-04-22T00:01:00.1147504Z     return Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2025-04-22T00:01:00.1148192Z RuntimeError: element 0 of tensors does not require grad and does not have a grad_fn
2025-04-22T00:01:00.1148557Z 
2025-04-22T00:01:00.1148791Z The above exception was the direct cause of the following exception:
2025-04-22T00:01:00.1149102Z 
2025-04-22T00:01:00.1149216Z Traceback (most recent call last):
2025-04-22T00:01:00.1149682Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 3956, in run
2025-04-22T00:01:00.1150158Z     ) = runner.load_model(
2025-04-22T00:01:00.1150641Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 381, in load_model
2025-04-22T00:01:00.1151187Z     self.validate_model(model, example_inputs)
2025-04-22T00:01:00.1151730Z   File "/var/lib/jenkins/workspace/benchmarks/dynamo/common.py", line 1904, in validate_model
2025-04-22T00:01:00.1152606Z     raise RuntimeError("Eager run failed") from e
2025-04-22T00:01:00.1152946Z RuntimeError: Eager run failed
2025-04-22T00:01:00.1153135Z 
2025-04-22T00:01:00.1153226Z eager_fail_to_run
2025-04-22T00:01:04.4310693Z 
2025-04-22T00:01:05.8042105Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:01:05.8042436Z loading model: 0it [00:01, ?it/s]
2025-04-22T00:01:05.8042749Z cuda train shufflenet_v2_x1_0                 
2025-04-22T00:01:44.3870569Z Compilation time (from dynamo_timed): 34.639805545
2025-04-22T00:01:44.3895872Z pass
2025-04-22T00:01:44.3909896Z TIMING: entire_frame_compile:20.10228 gc:0.00424 _recursive_pre_grad_passes:0.00921 pad_mm_benchmark:0.21009 _recursive_joint_graph_passes:0.57472 _recursive_post_grad_passes:0.43643 async_compile.wait:0.40409 code_gen:13.40435 inductor_compile:20.29957 backend_compile:14.08744 cudagraphify.get_container:0.32082 entire_backward_compile:14.53753 CUDAGraphNode.record:0.91361 total_wall_time:34.63981
2025-04-22T00:01:44.3911866Z STATS: call_* op count: 275 | FakeTensorMode.__torch_dispatch__:26685 | ProxyTorchDispatchMode.__torch_dispatch__:11979 | FakeTensor.__torch_dispatch__:16771
2025-04-22T00:01:44.3912694Z Dynamo produced 2 graphs covering 275 ops with 6 graph breaks (5 unique)
2025-04-22T00:01:50.9060937Z 
2025-04-22T00:01:51.8940129Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:01:51.8940465Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:01:51.8940783Z cuda train soft_actor_critic                  
2025-04-22T00:01:56.8609014Z W0422 00:01:56.860000 81484 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:01:59.2749724Z Compilation time (from dynamo_timed): 5.1210100259999995
2025-04-22T00:01:59.2750104Z pass
2025-04-22T00:01:59.2754682Z TIMING: entire_frame_compile:4.36677 gc:0.00439 _recursive_pre_grad_passes:0.00764 pad_mm_benchmark:0.32918 _recursive_joint_graph_passes:0.48973 _recursive_post_grad_passes:0.05111 async_compile.precompile:0.31487 async_compile.wait:1.36827 code_gen:2.19166 inductor_compile:2.69774 backend_compile:3.73818 cudagraphify.get_container:0.1682 entire_backward_compile:0.75424 CUDAGraphNode.record:0.88304 total_wall_time:5.12101
2025-04-22T00:01:59.2756700Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:2485 | ProxyTorchDispatchMode.__torch_dispatch__:945 | FakeTensor.__torch_dispatch__:1125
2025-04-22T00:01:59.2757506Z Dynamo produced 4 graphs covering 70 ops with 6 graph breaks (5 unique)
2025-04-22T00:02:04.6015630Z 
2025-04-22T00:02:05.9921375Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:02:05.9921972Z loading model: 0it [00:01, ?it/s]
2025-04-22T00:02:05.9922356Z cuda train speech_transformer                 
2025-04-22T00:02:23.2232457Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_14). Found from : 
2025-04-22T00:02:23.2233804Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 72, in torch_dynamo_resume_in_forward_at_63
2025-04-22T00:02:23.2234749Z     enc_output, enc_slf_attn = enc_layer(
2025-04-22T00:02:23.2235539Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/encoder.py", line 95, in forward
2025-04-22T00:02:23.2236336Z     enc_output, enc_slf_attn = self.slf_attn(
2025-04-22T00:02:23.2237159Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 53, in forward
2025-04-22T00:02:23.2237996Z     output, attn = self.attention(q, k, v, mask=mask)
2025-04-22T00:02:23.2238897Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2025-04-22T00:02:23.2239690Z     attn = attn / self.temperature
2025-04-22T00:02:23.2239890Z 
2025-04-22T00:02:47.2712606Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 
2025-04-22T00:02:47.2714296Z    File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 131, in torch_dynamo_resume_in_forward_at_120
2025-04-22T00:02:47.2715260Z     dec_output, dec_slf_attn, dec_enc_attn = dec_layer(
2025-04-22T00:02:47.2716097Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/decoder.py", line 291, in forward
2025-04-22T00:02:47.2716905Z     dec_output, dec_slf_attn = self.slf_attn(
2025-04-22T00:02:47.2718628Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 53, in forward
2025-04-22T00:02:47.2719467Z     output, attn = self.attention(q, k, v, mask=mask)
2025-04-22T00:02:47.2720299Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/speech_transformer/speech_transformer/transformer/attention.py", line 77, in forward
2025-04-22T00:02:47.2721107Z     attn = attn / self.temperature
2025-04-22T00:02:47.2721301Z 
2025-04-22T00:03:04.3103292Z Compilation time (from dynamo_timed): 52.365682572
2025-04-22T00:03:04.3132375Z pass
2025-04-22T00:03:04.3533394Z TIMING: entire_frame_compile:38.76075 gc:0.01034 _recursive_pre_grad_passes:0.02263 _recursive_joint_graph_passes:3.23207 inductor_compile:24.40346 backend_compile:31.94417 _recursive_post_grad_passes:1.34264 async_compile.precompile:0.08356 async_compile.wait:3.0842 code_gen:15.18541 cudagraphify.get_container:0.16588 pad_mm_benchmark:1.7246 CachingAutotuner.benchmark_all_configs:0.08423 entire_backward_compile:13.60493 CUDAGraphNode.record:2.10982 total_wall_time:52.36568
2025-04-22T00:03:04.3536102Z STATS: call_* op count: 1460 | FakeTensorMode.__torch_dispatch__:86204 | ProxyTorchDispatchMode.__torch_dispatch__:48932 | FakeTensor.__torch_dispatch__:19744
2025-04-22T00:03:04.3536941Z Dynamo produced 10 graphs covering 1460 ops with 16 graph breaks (7 unique)
2025-04-22T00:03:11.3433741Z 
2025-04-22T00:03:12.5992297Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:03:12.5992762Z loading model: 0it [00:01, ?it/s]
2025-04-22T00:03:12.5993113Z cuda train squeezenet1_1                      
2025-04-22T00:03:30.2562694Z Compilation time (from dynamo_timed): 14.829240146
2025-04-22T00:03:30.2566538Z pass
2025-04-22T00:03:30.2588737Z TIMING: entire_frame_compile:6.05833 gc:0.00193 _recursive_pre_grad_passes:0.00496 _recursive_joint_graph_passes:0.33728 _recursive_post_grad_passes:0.08718 async_compile.wait:0.60917 code_gen:9.58878 inductor_compile:11.08525 backend_compile:4.62686 cudagraphify.get_container:0.20309 entire_backward_compile:8.77091 CUDAGraphNode.record:0.4677 total_wall_time:14.82924
2025-04-22T00:03:30.2590594Z STATS: call_* op count: 70 | FakeTensorMode.__torch_dispatch__:6419 | ProxyTorchDispatchMode.__torch_dispatch__:2534 | FakeTensor.__torch_dispatch__:2192
2025-04-22T00:03:30.2591418Z Dynamo produced 2 graphs covering 70 ops with 6 graph breaks (5 unique)
2025-04-22T00:03:35.8954384Z 
2025-04-22T00:03:36.9337581Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:03:36.9337842Z 
2025-04-22T00:03:37.3334710Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2025-04-22T00:03:37.3335007Z 
2025-04-22T00:03:37.4996789Z Loading pipeline components...:  17% 1/6 [00:00<00:01,  2.50it/s]
2025-04-22T00:03:37.4997154Z 
2025-04-22T00:03:38.1095511Z Loading pipeline components...:  67% 4/6 [00:00<00:00,  8.30it/s]
2025-04-22T00:03:38.1095850Z 
2025-04-22T00:03:38.1096122Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  4.98it/s]
2025-04-22T00:03:38.1096614Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.10it/s]
2025-04-22T00:03:39.8149560Z 
2025-04-22T00:03:39.8149988Z loading model: 0it [00:03, ?it/s]
2025-04-22T00:03:39.8150392Z cuda train stable_diffusion_text_encoder      
2025-04-22T00:03:39.8559959Z WARNING:common:fp64 golden ref were not generated for stable_diffusion_text_encoder. Setting accuracy check to cosine
2025-04-22T00:04:30.2710869Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:04:30.2711736Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-22T00:04:30.2712461Z     pred = mod(*cloned_inputs)
2025-04-22T00:04:30.2713093Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 806, in forward
2025-04-22T00:04:30.2713796Z     return self.text_model(
2025-04-22T00:04:30.2714400Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 699, in forward
2025-04-22T00:04:30.2715624Z     hidden_states = self.embeddings(input_ids=input_ids, position_ids=position_ids)
2025-04-22T00:04:30.2716426Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/clip/modeling_clip.py", line 224, in forward
2025-04-22T00:04:30.2717107Z     inputs_embeds = self.token_embedding(input_ids)
2025-04-22T00:04:30.2717364Z 
2025-04-22T00:04:31.5593443Z Compilation time (from dynamo_timed): 47.964657137
2025-04-22T00:04:31.5617258Z pass
2025-04-22T00:04:31.6399623Z TIMING: entire_frame_compile:36.65621 gc:0.00404 _recursive_pre_grad_passes:0.01347 pad_mm_benchmark:1.43615 _recursive_joint_graph_passes:3.43719 _recursive_post_grad_passes:1.77056 async_compile.wait:0.07549 code_gen:11.39256 inductor_compile:22.21195 backend_compile:28.14578 cudagraphify.get_container:0.74352 entire_backward_compile:11.30844 CUDAGraphNode.record:0.7127 total_wall_time:47.96466
2025-04-22T00:04:31.6402783Z STATS: call_* op count: 828 | FakeTensorMode.__torch_dispatch__:76817 | FakeTensor.__torch_dispatch__:29216 | ProxyTorchDispatchMode.__torch_dispatch__:32624
2025-04-22T00:04:31.6404099Z Dynamo produced 2 graphs covering 828 ops with 5 graph breaks (5 unique)
2025-04-22T00:04:38.6477372Z 
2025-04-22T00:04:39.7060387Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:04:39.7060932Z 
2025-04-22T00:04:40.1067032Z Loading pipeline components...:   0% 0/6 [00:00<?, ?it/s]
2025-04-22T00:04:40.1067919Z 
2025-04-22T00:04:40.6748267Z Loading pipeline components...:  17% 1/6 [00:00<00:02,  2.50it/s]
2025-04-22T00:04:40.6748929Z 
2025-04-22T00:04:40.8402136Z Loading pipeline components...:  33% 2/6 [00:00<00:01,  2.00it/s]
2025-04-22T00:04:40.8402511Z 
2025-04-22T00:04:40.8402793Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  6.83it/s]
2025-04-22T00:04:40.8403284Z Loading pipeline components...: 100% 6/6 [00:01<00:00,  5.29it/s]
2025-04-22T00:04:43.2652780Z 
2025-04-22T00:04:43.2655933Z loading model: 0it [00:04, ?it/s]
2025-04-22T00:04:43.2656431Z cuda train stable_diffusion_unet              
2025-04-22T00:04:43.2888252Z Compilation time (from dynamo_timed): 0
2025-04-22T00:04:43.2888714Z pass_due_to_skip
2025-04-22T00:04:43.6132321Z TIMING: total_wall_time:0
2025-04-22T00:04:43.6132669Z STATS: call_* op count: 0
2025-04-22T00:04:43.6133070Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2025-04-22T00:04:47.9296113Z 
2025-04-22T00:04:51.3959518Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:04:51.3959871Z loading model: 0it [00:03, ?it/s]
2025-04-22T00:04:51.3960193Z cuda train timm_efficientnet                  
2025-04-22T00:05:35.7080127Z W0422 00:05:35.706000 83890 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:06:21.8067709Z E0422 00:06:21.805000 83890 site-packages/torch/_dynamo/utils.py:2939] RMSE (res-fp64): 1.86121, (ref-fp64): 0.53464 and shape=torch.Size([10, 240, 1, 1]). res.dtype: torch.float32, multiplier: 3.000000, tol: 0.080000, use_larger_multiplier_for_smaller_tensor: 1
2025-04-22T00:06:21.8069467Z E0422 00:06:21.806000 83890 site-packages/torch/_dynamo/utils.py:2800] Accuracy failed for key name blocks.2.1.se.conv_reduce.weight.grad
2025-04-22T00:06:21.8315102Z Compilation time (from dynamo_timed): 80.24996500900001
2025-04-22T00:06:21.8343298Z fail_accuracy
2025-04-22T00:06:21.8396657Z TIMING: entire_frame_compile:66.24699 gc:0.0048 _recursive_pre_grad_passes:0.03558 pad_mm_benchmark:0.03699 _recursive_joint_graph_passes:0.5664 _recursive_post_grad_passes:0.7614 async_compile.wait:3.60227 code_gen:25.34701 inductor_compile:41.52986 backend_compile:48.32477 cudagraphify.get_container:0.3881 entire_backward_compile:14.00298 CUDAGraphNode.record:3.00066 total_wall_time:80.24997
2025-04-22T00:06:21.8398572Z STATS: call_* op count: 1327 | FakeTensorMode.__torch_dispatch__:58268 | FakeTensor.__torch_dispatch__:44066 | ProxyTorchDispatchMode.__torch_dispatch__:25724
2025-04-22T00:06:21.8399393Z Dynamo produced 3 graphs covering 1327 ops with 7 graph breaks (5 unique)
2025-04-22T00:06:30.0461734Z 
2025-04-22T00:06:33.2381604Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:06:33.2381947Z loading model: 0it [00:03, ?it/s]
2025-04-22T00:06:33.2382263Z cuda train timm_regnet                        
2025-04-22T00:07:22.4688268Z W0422 00:07:22.467000 85997 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:08:19.1989067Z W0422 00:08:19.197000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.1995184Z W0422 00:08:19.199000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2004564Z W0422 00:08:19.199000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2011159Z W0422 00:08:19.200000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2017388Z W0422 00:08:19.201000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2024000Z W0422 00:08:19.201000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2030635Z W0422 00:08:19.202000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2036830Z W0422 00:08:19.203000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2043085Z W0422 00:08:19.203000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2057330Z W0422 00:08:19.205000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2063647Z W0422 00:08:19.205000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2070189Z W0422 00:08:19.206000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2076632Z W0422 00:08:19.207000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2082936Z W0422 00:08:19.207000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2089913Z W0422 00:08:19.208000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2096112Z W0422 00:08:19.209000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2102453Z W0422 00:08:19.209000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2108761Z W0422 00:08:19.210000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2123841Z W0422 00:08:19.211000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2130250Z W0422 00:08:19.212000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2137177Z W0422 00:08:19.213000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2144106Z W0422 00:08:19.213000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2150730Z W0422 00:08:19.214000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2157750Z W0422 00:08:19.215000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2177271Z W0422 00:08:19.217000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2184496Z W0422 00:08:19.217000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2707379Z W0422 00:08:19.269000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2711878Z W0422 00:08:19.270000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2719253Z W0422 00:08:19.271000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2726448Z W0422 00:08:19.272000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2733063Z W0422 00:08:19.272000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2740262Z W0422 00:08:19.273000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2746972Z W0422 00:08:19.274000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2753848Z W0422 00:08:19.274000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2760978Z W0422 00:08:19.275000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2776471Z W0422 00:08:19.277000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2783128Z W0422 00:08:19.277000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2790222Z W0422 00:08:19.278000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2797226Z W0422 00:08:19.279000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2804023Z W0422 00:08:19.279000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2811253Z W0422 00:08:19.280000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2818218Z W0422 00:08:19.281000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2825568Z W0422 00:08:19.282000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2832508Z W0422 00:08:19.282000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2841832Z W0422 00:08:19.283000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2852683Z W0422 00:08:19.284000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2859523Z W0422 00:08:19.285000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2866951Z W0422 00:08:19.286000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2874319Z W0422 00:08:19.286000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2881223Z W0422 00:08:19.287000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2888304Z W0422 00:08:19.288000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2909409Z W0422 00:08:19.290000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.2916183Z W0422 00:08:19.291000 85997 site-packages/torch/_dynamo/utils.py:2884] Found nan in reference. Consider running in higher precision.
2025-04-22T00:08:19.4047850Z Compilation time (from dynamo_timed): 91.230308173
2025-04-22T00:08:19.4080669Z pass
2025-04-22T00:08:19.4404278Z TIMING: entire_frame_compile:78.71309 gc:0.0057 _recursive_pre_grad_passes:0.04273 pad_mm_benchmark:0.21082 _recursive_joint_graph_passes:0.82958 _recursive_post_grad_passes:0.77329 async_compile.wait:4.28506 code_gen:25.7804 inductor_compile:43.42379 backend_compile:56.33227 cudagraphify.get_container:0.38556 entire_backward_compile:12.51722 CachingAutotuner.benchmark_all_configs:0.01888 CUDAGraphNode.record:3.4214 total_wall_time:91.23031
2025-04-22T00:08:19.4406543Z STATS: call_* op count: 1662 | FakeTensorMode.__torch_dispatch__:67732 | ProxyTorchDispatchMode.__torch_dispatch__:31122 | FakeTensor.__torch_dispatch__:49557
2025-04-22T00:08:19.4407394Z Dynamo produced 3 graphs covering 1662 ops with 7 graph breaks (5 unique)
2025-04-22T00:08:28.0662929Z 
2025-04-22T00:08:30.4412849Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:08:30.4413359Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:08:30.4413852Z cuda train timm_resnest                       
2025-04-22T00:09:00.7689645Z Compilation time (from dynamo_timed): 24.516387762
2025-04-22T00:09:00.7700165Z pass
2025-04-22T00:09:00.7789815Z TIMING: entire_frame_compile:12.36676 gc:0.00267 _recursive_pre_grad_passes:0.00628 pad_mm_benchmark:0.03756 _recursive_joint_graph_passes:0.30521 _recursive_post_grad_passes:0.32994 async_compile.wait:0.39994 code_gen:13.47608 inductor_compile:17.35556 backend_compile:9.54986 cudagraphify.get_container:0.27381 entire_backward_compile:12.14962 CUDAGraphNode.record:0.67782 total_wall_time:24.51639
2025-04-22T00:09:00.7792449Z STATS: call_* op count: 135 | FakeTensorMode.__torch_dispatch__:14198 | ProxyTorchDispatchMode.__torch_dispatch__:6029 | FakeTensor.__torch_dispatch__:7888
2025-04-22T00:09:00.7793372Z Dynamo produced 2 graphs covering 135 ops with 6 graph breaks (5 unique)
2025-04-22T00:09:06.6958772Z 
2025-04-22T00:09:08.7247791Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:09:08.7248269Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:09:08.7248705Z cuda train timm_vision_transformer            
2025-04-22T00:09:33.5988333Z Compilation time (from dynamo_timed): 21.381978338
2025-04-22T00:09:33.6004273Z pass
2025-04-22T00:09:33.6192999Z TIMING: entire_frame_compile:14.27395 gc:0.00255 _recursive_pre_grad_passes:0.00713 pad_mm_benchmark:1.03558 _recursive_joint_graph_passes:1.70876 _recursive_post_grad_passes:0.48656 async_compile.wait:0.16229 code_gen:6.35527 inductor_compile:10.98016 backend_compile:11.02596 cudagraphify.get_container:0.33031 entire_backward_compile:7.10803 CUDAGraphNode.record:0.80538 total_wall_time:21.38198
2025-04-22T00:09:33.6195939Z STATS: call_* op count: 267 | FakeTensorMode.__torch_dispatch__:26364 | FakeTensor.__torch_dispatch__:12073 | ProxyTorchDispatchMode.__torch_dispatch__:10947
2025-04-22T00:09:33.6196768Z Dynamo produced 2 graphs covering 267 ops with 6 graph breaks (5 unique)
2025-04-22T00:09:39.6286736Z 
2025-04-22T00:09:52.6877540Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:09:52.6877899Z loading model: 0it [00:13, ?it/s]
2025-04-22T00:09:52.6878221Z cuda train timm_vision_transformer_large      
2025-04-22T00:09:52.7118776Z Compilation time (from dynamo_timed): 0
2025-04-22T00:09:52.7119137Z pass_due_to_skip
2025-04-22T00:09:52.9753379Z TIMING: total_wall_time:0
2025-04-22T00:09:52.9753674Z STATS: call_* op count: 0
2025-04-22T00:09:52.9754062Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique)
2025-04-22T00:09:57.3473578Z 
2025-04-22T00:09:59.7780629Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:09:59.7781467Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:09:59.7782086Z cuda train timm_vovnet                        
2025-04-22T00:10:36.7242825Z Compilation time (from dynamo_timed): 27.684324398999998
2025-04-22T00:10:36.7257079Z pass
2025-04-22T00:10:36.7442883Z TIMING: entire_frame_compile:15.27769 gc:0.00278 _recursive_pre_grad_passes:0.00727 pad_mm_benchmark:0.03794 _recursive_joint_graph_passes:0.30341 _recursive_post_grad_passes:0.29975 async_compile.wait:0.29182 code_gen:12.86126 inductor_compile:17.26765 backend_compile:10.8555 cudagraphify.get_container:0.30266 entire_backward_compile:12.40663 CUDAGraphNode.record:0.76703 total_wall_time:27.68432
2025-04-22T00:10:36.7444808Z STATS: call_* op count: 134 | FakeTensorMode.__torch_dispatch__:17994 | ProxyTorchDispatchMode.__torch_dispatch__:8530 | FakeTensor.__torch_dispatch__:11379
2025-04-22T00:10:36.7445638Z Dynamo produced 2 graphs covering 134 ops with 6 graph breaks (5 unique)
2025-04-22T00:10:43.0725798Z 
2025-04-22T00:10:45.1295029Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:10:45.1295386Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:10:45.1295699Z cuda train torch_multimodal_clip              
2025-04-22T00:11:37.7490948Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:11:37.7491989Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-22T00:11:37.7492686Z     pred = mod(*cloned_inputs)
2025-04-22T00:11:37.7500956Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torchmultimodal/models/clip/model.py", line 72, in forward
2025-04-22T00:11:37.7501663Z     embeddings_b = self.encoder_b(features_b)
2025-04-22T00:11:37.7502341Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torchmultimodal/models/clip/text_encoder.py", line 119, in forward
2025-04-22T00:11:37.7503011Z     embeddings = self.token_embedding(text)
2025-04-22T00:11:37.7503240Z 
2025-04-22T00:11:37.8797984Z W0422 00:11:37.878000 89825 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:12:34.8531114Z Compilation time (from dynamo_timed): 100.85121063
2025-04-22T00:12:34.8567293Z pass
2025-04-22T00:12:34.9439679Z TIMING: entire_frame_compile:86.3407 gc:0.0063 _recursive_pre_grad_passes:0.04625 pad_mm_benchmark:0.40268 _recursive_joint_graph_passes:2.0709 _recursive_post_grad_passes:1.30661 async_compile.wait:4.25895 code_gen:33.38441 inductor_compile:53.55063 backend_compile:67.49836 cudagraphify.get_container:0.47617 entire_backward_compile:14.51051 CUDAGraphNode.record:2.36793 total_wall_time:100.85121
2025-04-22T00:12:34.9443500Z STATS: call_* op count: 1922 | FakeTensorMode.__torch_dispatch__:98961 | FakeTensor.__torch_dispatch__:56315 | ProxyTorchDispatchMode.__torch_dispatch__:42349
2025-04-22T00:12:34.9444651Z Dynamo produced 3 graphs covering 1922 ops with 7 graph breaks (5 unique)
2025-04-22T00:12:44.3631889Z 
2025-04-22T00:12:45.1359416Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:12:45.1359903Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:12:45.1360318Z cuda train tts_angular                        
2025-04-22T00:12:48.2519517Z W0422 00:12:48.250000 90795 site-packages/torch/_logging/_internal.py:1130] [10/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:12:52.2397975Z Compilation time (from dynamo_timed): 5.637606283
2025-04-22T00:12:52.2398348Z pass
2025-04-22T00:12:52.2473262Z TIMING: entire_frame_compile:5.30324 gc:0.00548 _recursive_pre_grad_passes:0.0088 _recursive_joint_graph_passes:0.15428 inductor_compile:2.99584 backend_compile:4.32835 _recursive_post_grad_passes:0.03351 async_compile.precompile:0.32734 async_compile.wait:1.37466 code_gen:2.37213 entire_backward_compile:0.33437 total_wall_time:5.63761
2025-04-22T00:12:52.2475055Z STATS: call_* op count: 102 | FakeTensorMode.__torch_dispatch__:2392 | ProxyTorchDispatchMode.__torch_dispatch__:994 | FakeTensor.__torch_dispatch__:1672
2025-04-22T00:12:52.2475876Z Dynamo produced 5 graphs covering 102 ops with 9 graph breaks (6 unique)
2025-04-22T00:12:57.6250987Z 
2025-04-22T00:13:00.3514993Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:13:00.3515418Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:13:00.3515785Z cuda train vgg16                              
2025-04-22T00:13:31.3426298Z Compilation time (from dynamo_timed): 10.461967834
2025-04-22T00:13:31.3429741Z pass
2025-04-22T00:13:31.3644590Z TIMING: entire_frame_compile:5.32778 gc:0.0026 _recursive_pre_grad_passes:0.00454 pad_mm_benchmark:0.57003 _recursive_joint_graph_passes:0.75731 _recursive_post_grad_passes:0.07254 async_compile.wait:0.73895 code_gen:5.88619 inductor_compile:6.98221 backend_compile:4.40393 cudagraphify.get_container:0.19346 CachingAutotuner.benchmark_all_configs:1.12879 entire_backward_compile:5.13419 CUDAGraphNode.record:0.41847 total_wall_time:10.46197
2025-04-22T00:13:31.3647302Z STATS: call_* op count: 44 | FakeTensorMode.__torch_dispatch__:4640 | ProxyTorchDispatchMode.__torch_dispatch__:1690 | FakeTensor.__torch_dispatch__:1302
2025-04-22T00:13:31.3648124Z Dynamo produced 2 graphs covering 44 ops with 6 graph breaks (5 unique)
2025-04-22T00:13:36.8859565Z 
2025-04-22T00:13:40.3979500Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:13:40.3980008Z loading model: 0it [00:03, ?it/s]
2025-04-22T00:13:40.3980336Z cuda train vision_maskrcnn                    
2025-04-22T00:14:13.3146815Z W0422 00:14:13.313000 91676 site-packages/torch/_inductor/utils.py:1881] [4/0_1] DeviceCopy in input program
2025-04-22T00:14:13.3156959Z W0422 00:14:13.315000 91676 site-packages/torch/_inductor/utils.py:1881] [4/0_1] DeviceCopy in input program
2025-04-22T00:14:13.3166856Z W0422 00:14:13.316000 91676 site-packages/torch/_inductor/utils.py:1881] [4/0_1] DeviceCopy in input program
2025-04-22T00:14:13.3176519Z W0422 00:14:13.317000 91676 site-packages/torch/_inductor/utils.py:1881] [4/0_1] DeviceCopy in input program
2025-04-22T00:14:13.3186412Z W0422 00:14:13.318000 91676 site-packages/torch/_inductor/utils.py:1881] [4/0_1] DeviceCopy in input program
2025-04-22T00:14:14.7821120Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_12). Found from : 
2025-04-22T00:14:14.7822290Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/rpn.py", line 362, in forward
2025-04-22T00:14:14.7822951Z     anchors = self.anchor_generator(images, features)
2025-04-22T00:14:14.7823648Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 126, in forward
2025-04-22T00:14:14.7824359Z     self.set_cell_anchors(dtype, device)
2025-04-22T00:14:14.7825045Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in set_cell_anchors
2025-04-22T00:14:14.7825934Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2025-04-22T00:14:14.7827140Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/anchor_utils.py", line 77, in <listcomp>
2025-04-22T00:14:14.7828014Z     self.cell_anchors = [cell_anchor.to(dtype=dtype, device=device) for cell_anchor in self.cell_anchors]
2025-04-22T00:14:14.7828428Z 
2025-04-22T00:14:16.5264186Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] Graph break from `Tensor.item()`, consider setting:
2025-04-22T00:14:16.5265248Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     torch._dynamo.config.capture_scalar_outputs = True
2025-04-22T00:14:16.5266382Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] or:
2025-04-22T00:14:16.5267181Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     env TORCHDYNAMO_CAPTURE_SCALAR_OUTPUTS=1
2025-04-22T00:14:16.5268138Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] to include these operations in the captured graph.
2025-04-22T00:14:16.5268951Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] 
2025-04-22T00:14:16.5269696Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] Graph break: from user code at:
2025-04-22T00:14:16.5271089Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 105, in torch_dynamo_resume_in_forward_at_104
2025-04-22T00:14:16.5272741Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     detections, detector_losses = self.roi_heads(features, proposals, images.image_sizes, targets)
2025-04-22T00:14:16.5274303Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 761, in forward
2025-04-22T00:14:16.5275681Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     box_features = self.box_roi_pool(features, proposals, image_shapes)
2025-04-22T00:14:16.5276962Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 310, in forward
2025-04-22T00:14:16.5278176Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     self.scales, self.map_levels = _setup_scales(
2025-04-22T00:14:16.5279418Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in _setup_scales
2025-04-22T00:14:16.5280730Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2025-04-22T00:14:16.5282045Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 122, in <listcomp>
2025-04-22T00:14:16.5283350Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     scales = [_infer_scale(feat, original_input_shape) for feat in features]
2025-04-22T00:14:16.5284657Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/poolers.py", line 104, in _infer_scale
2025-04-22T00:14:16.5285930Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0]     scale = 2 ** float(torch.tensor(approx_scale).log2().round())
2025-04-22T00:14:16.5286757Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] 
2025-04-22T00:14:16.5287558Z W0422 00:14:16.525000 91676 site-packages/torch/_dynamo/variables/tensor.py:928] [17/0] 
2025-04-22T00:14:32.8780564Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:14:32.8781293Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:14:32.8782117Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:14:32.8782540Z 
2025-04-22T00:14:38.9755354Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:14:38.9756423Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:14:38.9757262Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:14:38.9757687Z 
2025-04-22T00:14:45.1307480Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:14:45.1308242Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:14:45.1309048Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:14:45.1309464Z 
2025-04-22T00:15:23.3424769Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:23.3425509Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:15:23.3426350Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:15:23.3426766Z 
2025-04-22T00:15:32.5925968Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:32.5926706Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:15:32.5927561Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:15:32.5927986Z 
2025-04-22T00:15:47.2712736Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:47.2713547Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:15:47.2714354Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:15:47.2714765Z 
2025-04-22T00:15:52.0525904Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:52.0526660Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:15:52.0527465Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:15:52.0527881Z 
2025-04-22T00:15:58.4109991Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:58.4111885Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2025-04-22T00:15:58.4113470Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2025-04-22T00:15:58.4113813Z 
2025-04-22T00:15:59.3994134Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:15:59.3995104Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 415, in torch_dynamo_resume_in_paste_mask_in_image_at_407
2025-04-22T00:15:59.3996070Z     mask = F.interpolate(mask, size=(h, w), mode="bilinear", align_corners=False)
2025-04-22T00:15:59.3996407Z 
2025-04-22T00:16:00.3646817Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:16:00.3647745Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 821, in torch_dynamo_resume_in_forward_at_804
2025-04-22T00:16:00.3648921Z     masks_probs = maskrcnn_inference(mask_logits, labels)
2025-04-22T00:16:00.3649683Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/roi_heads.py", line 76, in maskrcnn_inference
2025-04-22T00:16:00.3650424Z     mask_prob = mask_prob[index, labels][:, None]
2025-04-22T00:16:00.3650669Z 
2025-04-22T00:16:02.0141222Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:16:02.0142256Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py", line 236, in roi_align
2025-04-22T00:16:02.0143461Z     return _roi_align(input, rois, spatial_scale, output_size[0], output_size[1], sampling_ratio, aligned)
2025-04-22T00:16:02.0143898Z 
2025-04-22T00:16:06.9253700Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:16:06.9254553Z    File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/generalized_rcnn.py", line 101, in forward
2025-04-22T00:16:06.9255305Z     features = self.backbone(images.tensors)
2025-04-22T00:16:06.9255977Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/models/detection/backbone_utils.py", line 58, in forward
2025-04-22T00:16:06.9256639Z     x = self.fpn(x)
2025-04-22T00:16:06.9257220Z   File "/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/feature_pyramid_network.py", line 194, in forward
2025-04-22T00:16:06.9257988Z     inner_top_down = F.interpolate(last_inner, size=feat_shape, mode="nearest")
2025-04-22T00:16:06.9258325Z 
2025-04-22T00:16:11.7629230Z W0422 00:16:11.761000 91676 site-packages/torch/_dynamo/convert_frame.py:990] [29/8] torch._dynamo hit config.recompile_limit (8)
2025-04-22T00:16:11.7630393Z W0422 00:16:11.761000 91676 site-packages/torch/_dynamo/convert_frame.py:990] [29/8]    function: 'roi_align' (/var/lib/jenkins/.local/lib/python3.10/site-packages/torchvision/ops/roi_align.py:186)
2025-04-22T00:16:11.7631645Z W0422 00:16:11.761000 91676 site-packages/torch/_dynamo/convert_frame.py:990] [29/8]    last reason: 29/7: tensor 'boxes' requires_grad mismatch. expected requires_grad=1
2025-04-22T00:16:11.7632701Z W0422 00:16:11.761000 91676 site-packages/torch/_dynamo/convert_frame.py:990] [29/8] To log all recompilation reasons, use TORCH_LOGS="recompiles".
2025-04-22T00:16:11.7633948Z W0422 00:16:11.761000 91676 site-packages/torch/_dynamo/convert_frame.py:990] [29/8] To diagnose recompilation issues, see https://pytorch.org/docs/main/torch.compiler_troubleshooting.html.
2025-04-22T00:16:12.5837829Z Compilation time (from dynamo_timed): 141.500042122
2025-04-22T00:16:12.5867921Z pass
2025-04-22T00:16:12.6487916Z TIMING: entire_frame_compile:133.6172 gc:0.03317 _recursive_pre_grad_passes:0.03527 _recursive_joint_graph_passes:2.02555 inductor_compile:65.28144 backend_compile:94.27441 _recursive_post_grad_passes:0.90952 async_compile.wait:14.00684 code_gen:38.77259 cudagraphify.get_container:0.34569 pad_mm_benchmark:0.36277 entire_backward_compile:7.88284 CUDAGraphNode.record:1.05621 total_wall_time:141.50004
2025-04-22T00:16:12.6490785Z STATS: call_* op count: 1463 | FakeTensorMode.__torch_dispatch__:98394 | FakeTensor.__torch_dispatch__:29201 | ProxyTorchDispatchMode.__torch_dispatch__:31466 | attempt fast:1501 | slow no contiguity match:174 | fast is_contiguous:1050 | slow both tensors nontrivially broadcast:277
2025-04-22T00:16:12.6492257Z Dynamo produced 42 graphs covering 1463 ops with 35 graph breaks (8 unique)
2025-04-22T00:16:22.2677617Z 
2025-04-22T00:16:25.0192266Z loading model: 0it [00:00, ?it/s]
2025-04-22T00:16:25.0192616Z loading model: 0it [00:02, ?it/s]
2025-04-22T00:16:25.0192953Z cuda train yolov3                             
2025-04-22T00:17:05.7592808Z W0422 00:17:05.758000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:05.7596974Z W0422 00:17:05.759000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:05.9764047Z W0422 00:17:05.975000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:05.9769079Z W0422 00:17:05.976000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:06.2118551Z W0422 00:17:06.211000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:06.2122220Z W0422 00:17:06.211000 94368 site-packages/torch/_inductor/utils.py:1881] [2/0_1] DeviceCopy in input program
2025-04-22T00:17:20.8188403Z skipping cudagraphs due to skipping cudagraphs due to cpu device (primals_358). Found from : 
2025-04-22T00:17:20.8189685Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-22T00:17:20.8190403Z     pred = mod(*cloned_inputs)
2025-04-22T00:17:20.8191009Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 300, in forward
2025-04-22T00:17:20.8191655Z     return self.forward_once(x)
2025-04-22T00:17:20.8192286Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 363, in forward_once
2025-04-22T00:17:20.8192940Z     yolo_out.append(module(x, out))
2025-04-22T00:17:20.8193661Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 237, in forward
2025-04-22T00:17:20.8194301Z     self.create_grids((nx, ny), p.device)
2025-04-22T00:17:20.8194952Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 204, in create_grids
2025-04-22T00:17:20.8195633Z     self.anchor_vec = self.anchor_vec.to(device)
2025-04-22T00:17:20.8195876Z 
2025-04-22T00:17:48.4997624Z W0422 00:17:48.498000 94368 site-packages/torch/_logging/_internal.py:1130] [6/0] Profiler function <class 'torch.autograd.profiler.record_function'> will be ignored
2025-04-22T00:19:32.5961778Z skipping cudagraphs due to deterministic index put. Found from : 
2025-04-22T00:19:32.5963476Z    File "/var/lib/jenkins/workspace/benchmarks/dynamo/torchbench.py", line 469, in torch_dynamo_resume_in_forward_and_backward_pass_at_464
2025-04-22T00:19:32.5964868Z     pred = mod(*cloned_inputs)
2025-04-22T00:19:32.5965868Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 300, in forward
2025-04-22T00:19:32.5966520Z     return self.forward_once(x)
2025-04-22T00:19:32.5967126Z   File "/var/lib/jenkins/workspace/torchbench/torchbenchmark/models/yolov3/yolo_models.py", line 365, in forward_once
2025-04-22T00:19:32.5967756Z     x = module(x)
2025-04-22T00:19:32.5967905Z 
2025-04-22T00:19:34.9399543Z Compilation time (from dynamo_timed): 168.380539635
2025-04-22T00:19:34.9448396Z pass
2025-04-22T00:19:34.9839394Z TIMING: entire_frame_compile:119.95642 gc:0.00575 _recursive_pre_grad_passes:0.04724 _recursive_joint_graph_passes:1.89639 _recursive_post_grad_passes:1.83454 async_compile.wait:5.15148 code_gen:73.20463 inductor_compile:103.14319 backend_compile:92.18745 CachingAutotuner.benchmark_all_configs:0.12639 entire_backward_compile:48.42412 cudagraphify.get_container:1.21496 CUDAGraphNode.record:1.89751 total_wall_time:168.38054
2025-04-22T00:19:34.9841495Z STATS: call_* op count: 1829 | FakeTensorMode.__torch_dispatch__:101919 | FakeTensor.__torch_dispatch__:87448 | ProxyTorchDispatchMode.__torch_dispatch__:46242
2025-04-22T00:19:34.9842335Z Dynamo produced 4 graphs covering 1829 ops with 8 graph breaks (5 unique)
2025-04-22T00:19:44.0435660Z accuracy            pass_rate=76.92%
2025-04-22T00:19:44.0439291Z calls_captured      gmean=0.00x mean=540.462x
2025-04-22T00:19:44.0442947Z unique_graphs       gmean=0.00x mean=3.590x
2025-04-22T00:19:44.0446554Z graph_breaks        gmean=0.00x mean=6.538x
2025-04-22T00:19:44.0450565Z unique_graph_breaks gmean=0.00x mean=4.385x
2025-04-22T00:19:44.0453770Z autograd_captures   gmean=0.00x mean=0.000x
2025-04-22T00:19:44.0457503Z autograd_compiles   gmean=0.00x mean=0.000x
2025-04-22T00:19:44.0461133Z cudagraph_skips     gmean=0.00x mean=0.590x
2025-04-22T00:19:44.0461942Z compilation_latency mean=38.126 seconds
2025-04-22T00:19:46.5377491Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2025-04-22T00:19:46.8746073Z llava                               XFAIL
2025-04-22T00:19:46.8751902Z maml_omniglot                       PASS
2025-04-22T00:19:46.8755129Z microbench_unbacked_tolist_sum      PASS
2025-04-22T00:19:46.8761169Z mnasnet1_0                          PASS
2025-04-22T00:19:46.8767162Z mobilenet_v2                        PASS
2025-04-22T00:19:46.8772316Z mobilenet_v2_quantized_qat          XFAIL
2025-04-22T00:19:46.8777905Z mobilenet_v3_large                  PASS
2025-04-22T00:19:46.8783329Z moco                                PASS
2025-04-22T00:19:46.8788729Z nanogpt                             PASS
2025-04-22T00:19:46.8794227Z nvidia_deeprecommender              PASS
2025-04-22T00:19:46.8799671Z opacus_cifar10                      XFAIL
2025-04-22T00:19:46.8805015Z phlippe_densenet                    PASS
2025-04-22T00:19:46.8810430Z phlippe_resnet                      PASS
2025-04-22T00:19:46.8815776Z pytorch_CycleGAN_and_pix2pix        PASS
2025-04-22T00:19:46.8821271Z pytorch_stargan                     PASS
2025-04-22T00:19:46.8827220Z pytorch_unet                        XFAIL
2025-04-22T00:19:46.8832524Z resnet152                           PASS
2025-04-22T00:19:46.8838246Z resnet18                            PASS
2025-04-22T00:19:46.8843610Z resnet50                            PASS
2025-04-22T00:19:46.8848869Z resnet50_quantized_qat              XFAIL
2025-04-22T00:19:46.8854260Z resnext50_32x4d                     PASS
2025-04-22T00:19:46.8859694Z sam                                 XFAIL
2025-04-22T00:19:46.8865188Z shufflenet_v2_x1_0                  PASS
2025-04-22T00:19:46.8870629Z soft_actor_critic                   PASS
2025-04-22T00:19:46.8876110Z speech_transformer                  PASS
2025-04-22T00:19:46.8881547Z squeezenet1_1                       PASS
2025-04-22T00:19:46.8886909Z stable_diffusion_text_encoder       PASS
2025-04-22T00:19:46.8903095Z stable_diffusion_unet               XFAIL
2025-04-22T00:19:46.8903694Z timm_efficientnet                   XFAIL
2025-04-22T00:19:46.8904858Z timm_regnet                         PASS
2025-04-22T00:19:46.8911253Z timm_resnest                        PASS
2025-04-22T00:19:46.8916792Z timm_vision_transformer             PASS
2025-04-22T00:19:46.8922393Z timm_vision_transformer_large       XFAIL
2025-04-22T00:19:46.8928040Z timm_vovnet                         PASS
2025-04-22T00:19:46.8933373Z torch_multimodal_clip               PASS
2025-04-22T00:19:46.8938811Z tts_angular                         PASS
2025-04-22T00:19:46.8944223Z vgg16                               PASS
2025-04-22T00:19:46.8949562Z vision_maskrcnn                     PASS
2025-04-22T00:19:46.8955202Z yolov3                              PASS
2025-04-22T00:19:46.9496136Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/training_torchbench.csv --expected benchmarks/dynamo/ci_expected_accuracy/inductor_torchbench_training.csv
2025-04-22T00:19:47.2798475Z llava                               PASS
2025-04-22T00:19:47.2803808Z maml_omniglot                       PASS
2025-04-22T00:19:47.2809401Z microbench_unbacked_tolist_sum      PASS
2025-04-22T00:19:47.2815289Z mnasnet1_0                          PASS
2025-04-22T00:19:47.2821067Z mobilenet_v2                        PASS
2025-04-22T00:19:47.2827104Z mobilenet_v2_quantized_qat          PASS
2025-04-22T00:19:47.2832742Z mobilenet_v3_large                  PASS
2025-04-22T00:19:47.2838584Z moco                                PASS
2025-04-22T00:19:47.2844035Z nanogpt                             PASS
2025-04-22T00:19:47.2849414Z nvidia_deeprecommender              PASS
2025-04-22T00:19:47.2854380Z opacus_cifar10                      PASS
2025-04-22T00:19:47.2859165Z phlippe_densenet                    PASS
2025-04-22T00:19:47.2864002Z phlippe_resnet                      PASS
2025-04-22T00:19:47.2868768Z pytorch_CycleGAN_and_pix2pix        PASS
2025-04-22T00:19:47.2873291Z pytorch_stargan                     PASS
2025-04-22T00:19:47.2878517Z pytorch_unet                        PASS
2025-04-22T00:19:47.2883035Z resnet152                           PASS
2025-04-22T00:19:47.2887830Z resnet18                            PASS
2025-04-22T00:19:47.2892526Z resnet50                            PASS
2025-04-22T00:19:47.2897210Z resnet50_quantized_qat              PASS
2025-04-22T00:19:47.2902236Z resnext50_32x4d                     PASS
2025-04-22T00:19:47.2906663Z sam                                 PASS
2025-04-22T00:19:47.2911493Z shufflenet_v2_x1_0                  PASS
2025-04-22T00:19:47.2916131Z soft_actor_critic                   PASS
2025-04-22T00:19:47.2920906Z speech_transformer                  PASS
2025-04-22T00:19:47.2926743Z squeezenet1_1                       PASS
2025-04-22T00:19:47.2931246Z stable_diffusion_text_encoder       PASS
2025-04-22T00:19:47.2935975Z stable_diffusion_unet               PASS
2025-04-22T00:19:47.2940493Z timm_efficientnet                   PASS
2025-04-22T00:19:47.2946885Z timm_regnet                         PASS
2025-04-22T00:19:47.2951691Z timm_resnest                        PASS
2025-04-22T00:19:47.2956778Z timm_vision_transformer             PASS
2025-04-22T00:19:47.2961364Z timm_vision_transformer_large       PASS
2025-04-22T00:19:47.2966311Z timm_vovnet                         PASS
2025-04-22T00:19:47.2970743Z torch_multimodal_clip               PASS
2025-04-22T00:19:47.2975529Z tts_angular                         PASS
2025-04-22T00:19:47.2980173Z vgg16                               PASS
2025-04-22T00:19:47.2984947Z vision_maskrcnn                     PASS
2025-04-22T00:19:47.2989566Z yolov3                              PASS_BUT_FLAKY
2025-04-22T00:19:47.3536356Z + cleanup_workspace
2025-04-22T00:19:47.3536896Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.'
2025-04-22T00:19:47.3537661Z sudo may print the following warning message that can be ignored. The chown command will still run.
2025-04-22T00:19:47.3538267Z + echo '    sudo: setrlimit(RLIMIT_STACK): Operation not permitted'
2025-04-22T00:19:47.3538721Z     sudo: setrlimit(RLIMIT_STACK): Operation not permitted
2025-04-22T00:19:47.3539238Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42'
2025-04-22T00:19:47.3539815Z For more details refer to https://github.com/sudo-project/sudo/issues/42
2025-04-22T00:19:47.3540284Z + sudo chown -R 1000 /var/lib/jenkins/workspace
2025-04-22T00:19:50.2218385Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-04-22T00:19:50.2218861Z with:
2025-04-22T00:19:50.2219119Z   benchmark-results-dir: test/test-reports
2025-04-22T00:19:50.2219445Z   dry-run: false
2025-04-22T00:19:50.2219692Z   schema-version: v3
2025-04-22T00:19:50.2220131Z   github-token: ***
2025-04-22T00:19:50.2220360Z env:
2025-04-22T00:19:50.2220571Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:50.2220901Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:50.2221674Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:50.2222170Z ##[endgroup]
2025-04-22T00:19:50.2248748Z ##[group]Run set -eux
2025-04-22T00:19:50.2249002Z set -eux
2025-04-22T00:19:50.2249260Z python3 -mpip install boto3==1.35.33
2025-04-22T00:19:50.2263130Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:50.2263496Z env:
2025-04-22T00:19:50.2263733Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:50.2264059Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:50.2264588Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:50.2265070Z ##[endgroup]
2025-04-22T00:19:50.2302186Z + python3 -mpip install boto3==1.35.33
2025-04-22T00:19:50.5935820Z Defaulting to user installation because normal site-packages is not writeable
2025-04-22T00:19:51.7835970Z Collecting boto3==1.35.33
2025-04-22T00:19:51.7867680Z   Using cached boto3-1.35.33-py3-none-any.whl (139 kB)
2025-04-22T00:19:51.7927741Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (1.35.99)
2025-04-22T00:19:51.7932358Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0)
2025-04-22T00:19:51.7941654Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.4)
2025-04-22T00:19:51.7995044Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.2)
2025-04-22T00:19:51.8005667Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10)
2025-04-22T00:19:51.8045819Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0)
2025-04-22T00:19:51.9658343Z Installing collected packages: boto3
2025-04-22T00:19:51.9658905Z   Attempting uninstall: boto3
2025-04-22T00:19:51.9660219Z     Found existing installation: boto3 1.35.42
2025-04-22T00:19:51.9795773Z     Uninstalling boto3-1.35.42:
2025-04-22T00:19:51.9824767Z       Successfully uninstalled boto3-1.35.42
2025-04-22T00:19:52.0589698Z Successfully installed boto3-1.35.33
2025-04-22T00:19:52.1337244Z ##[group]Run set -eux
2025-04-22T00:19:52.1337523Z set -eux
2025-04-22T00:19:52.1337746Z 
2025-04-22T00:19:52.1337981Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-04-22T00:19:52.1338330Z   echo "Missing github-token input"
2025-04-22T00:19:52.1338660Z   exit 1
2025-04-22T00:19:52.1338877Z fi
2025-04-22T00:19:52.1347537Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:52.1347906Z env:
2025-04-22T00:19:52.1348116Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:52.1348445Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:52.1348988Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:52.1349653Z   GITHUB_TOKEN: ***
2025-04-22T00:19:52.1349904Z ##[endgroup]
2025-04-22T00:19:52.1391827Z + [[ -z *** ]]
2025-04-22T00:19:52.1443975Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-04-22T00:19:52.1444597Z with:
2025-04-22T00:19:52.1444944Z   github-token: ***
2025-04-22T00:19:52.1445173Z env:
2025-04-22T00:19:52.1445390Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:52.1445709Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:52.1446246Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:52.1446739Z ##[endgroup]
2025-04-22T00:19:52.1471279Z ##[group]Run set -eux
2025-04-22T00:19:52.1471539Z set -eux
2025-04-22T00:19:52.1471762Z 
2025-04-22T00:19:52.1472215Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-04-22T00:19:52.1480868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:52.1481234Z env:
2025-04-22T00:19:52.1481453Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:52.1481778Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:52.1482326Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:52.1482939Z   GITHUB_TOKEN: ***
2025-04-22T00:19:52.1483168Z ##[endgroup]
2025-04-22T00:19:52.1509685Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 14580756816 i-0e08e74978c5f6915
2025-04-22T00:19:53.6236050Z setting job-id=40900324329
2025-04-22T00:19:53.6237123Z setting job-name=cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-22T00:19:53.6348643Z ##[group]Run set -eux
2025-04-22T00:19:53.6348971Z set -eux
2025-04-22T00:19:53.6349199Z 
2025-04-22T00:19:53.6349581Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-04-22T00:19:53.6350085Z   --schema-version "${SCHEMA_VERSION}" \
2025-04-22T00:19:53.6350432Z   --repo "${REPO}" \
2025-04-22T00:19:53.6350738Z   --head-branch "${HEAD_BRANCH}" \
2025-04-22T00:19:53.6351089Z   --head-sha "${HEAD_SHA}" \
2025-04-22T00:19:53.6351427Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-04-22T00:19:53.6351782Z   --run-attempt "${RUN_ATTEMPT}" \
2025-04-22T00:19:53.6352106Z   --job-id "${JOB_ID}" \
2025-04-22T00:19:53.6352412Z   --job-name "${JOB_NAME}"
2025-04-22T00:19:53.6361414Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:53.6361783Z env:
2025-04-22T00:19:53.6361993Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:53.6362322Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:53.6362858Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:53.6363341Z   SCHEMA_VERSION: v3
2025-04-22T00:19:53.6363600Z   REPO: pytorch/pytorch
2025-04-22T00:19:53.6363915Z   HEAD_BRANCH: refs/tags/ciflow/inductor/148893
2025-04-22T00:19:53.6364319Z   HEAD_SHA: b68c0effe97351b0c24b2f7aef671559be6aa977
2025-04-22T00:19:53.6364668Z   WORKFLOW_RUN_ID: 14580756816
2025-04-22T00:19:53.6364935Z   RUN_ATTEMPT: 1
2025-04-22T00:19:53.6365162Z   JOB_ID: 40900324329
2025-04-22T00:19:53.6365614Z   JOB_NAME: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-22T00:19:53.6366124Z ##[endgroup]
2025-04-22T00:19:53.6396112Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/tags/ciflow/inductor/148893 --head-sha b68c0effe97351b0c24b2f7aef671559be6aa977 --workflow-id 14580756816 --run-attempt 1 --job-id 40900324329 --job-name 'cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)'
2025-04-22T00:19:53.6720170Z ##[group]Run set -eux
2025-04-22T00:19:53.6720436Z set -eux
2025-04-22T00:19:53.6720660Z 
2025-04-22T00:19:53.6720897Z # TODO (huydhn): Implement this part
2025-04-22T00:19:53.6721781Z echo "runners=[]" >> "${GITHUB_OUTPUT}"
2025-04-22T00:19:53.6730947Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:53.6731303Z env:
2025-04-22T00:19:53.6731517Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:53.6731863Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:53.6732399Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:53.6732872Z ##[endgroup]
2025-04-22T00:19:53.6761420Z + echo 'runners=[]'
2025-04-22T00:19:53.6786994Z ##[group]Run set -eux
2025-04-22T00:19:53.6787259Z set -eux
2025-04-22T00:19:53.6787488Z 
2025-04-22T00:19:53.6787739Z # TODO (huydhn): Implement this part
2025-04-22T00:19:53.6788129Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-04-22T00:19:53.6796282Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:53.6796650Z env:
2025-04-22T00:19:53.6796869Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:53.6797218Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:53.6797777Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:53.6798256Z ##[endgroup]
2025-04-22T00:19:53.6823698Z + echo 'dependencies={}'
2025-04-22T00:19:53.6867686Z ##[group]Run set -eux
2025-04-22T00:19:53.6868099Z set -eux
2025-04-22T00:19:53.6868434Z 
2025-04-22T00:19:53.6868827Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-04-22T00:19:53.6869475Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-04-22T00:19:53.6870209Z   # We don't want the job to fail if the directory doesn't exist
2025-04-22T00:19:53.6870791Z   exit 0
2025-04-22T00:19:53.6871104Z fi
2025-04-22T00:19:53.6871411Z 
2025-04-22T00:19:53.6871762Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-04-22T00:19:53.6872499Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-04-22T00:19:53.6873444Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-04-22T00:19:53.6874145Z     --metadata "${BENCHMARK_METADATA}" \
2025-04-22T00:19:53.6874680Z     --runners "${RUNNER_INFO}" \
2025-04-22T00:19:53.6875205Z     --dependencies "${DEPENDENCIES}" \
2025-04-22T00:19:53.6875653Z     --dry-run
2025-04-22T00:19:53.6875899Z else
2025-04-22T00:19:53.6876260Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-04-22T00:19:53.6876806Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-04-22T00:19:53.6877228Z     --metadata "${BENCHMARK_METADATA}" \
2025-04-22T00:19:53.6877573Z     --runners "${RUNNER_INFO}" \
2025-04-22T00:19:53.6877905Z     --dependencies "${DEPENDENCIES}"
2025-04-22T00:19:53.6878226Z fi
2025-04-22T00:19:53.6886496Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:53.6886860Z env:
2025-04-22T00:19:53.6887081Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:53.6887416Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:53.6887960Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:53.6888473Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-04-22T00:19:53.6888790Z   DRY_RUN: false
2025-04-22T00:19:53.6890146Z   BENCHMARK_METADATA: {"timestamp": 1745281193, "schema_version": "v3", "name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/inductor/148893", "head_sha": "b68c0effe97351b0c24b2f7aef671559be6aa977", "workflow_id": 14580756816, "run_attempt": 1, "job_id": 40900324329}
2025-04-22T00:19:53.6891553Z   RUNNER_INFO: []
2025-04-22T00:19:53.6891783Z   DEPENDENCIES: {}
2025-04-22T00:19:53.6892018Z ##[endgroup]
2025-04-22T00:19:53.6918549Z + [[ ! -d test/test-reports ]]
2025-04-22T00:19:53.6918856Z + [[ false == \t\r\u\e ]]
2025-04-22T00:19:53.6921167Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1745281193, "schema_version": "v3", "name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/inductor/148893", "head_sha": "b68c0effe97351b0c24b2f7aef671559be6aa977", "workflow_id": 14580756816, "run_attempt": 1, "job_id": 40900324329}' --runners '[]' --dependencies '{}'
2025-04-22T00:19:53.8469194Z INFO:root:Upload test/test-reports/inference_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/14580756816/40900324329/inference_torchbench.json
2025-04-22T00:19:53.8869055Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role
2025-04-22T00:19:54.1234636Z INFO:root:Upload test/test-reports/inference_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/14580756816/40900324329/inference_torchbench_graph_breaks.json
2025-04-22T00:19:54.3117762Z INFO:root:Upload test/test-reports/training_torchbench.json to s3://ossci-benchmarks/v3/pytorch/pytorch/14580756816/40900324329/training_torchbench.json
2025-04-22T00:19:54.4459237Z INFO:root:Upload test/test-reports/training_torchbench_graph_breaks.json to s3://ossci-benchmarks/v3/pytorch/pytorch/14580756816/40900324329/training_torchbench_graph_breaks.json
2025-04-22T00:19:54.6917457Z ##[group]Run cat test/**/*_toprint.log || true
2025-04-22T00:19:54.6917854Z cat test/**/*_toprint.log || true
2025-04-22T00:19:54.6927224Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:54.6927585Z env:
2025-04-22T00:19:54.6927792Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:54.6928113Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:54.6928634Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:54.6929108Z ##[endgroup]
2025-04-22T00:19:54.7019820Z cat: 'test/**/*_toprint.log': No such file or directory
2025-04-22T00:19:54.7054121Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2025-04-22T00:19:54.7054483Z kill "$MONITOR_SCRIPT_PID"
2025-04-22T00:19:54.7062397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:54.7062769Z env:
2025-04-22T00:19:54.7062991Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:54.7063329Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:54.7063859Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:54.7064354Z   MONITOR_SCRIPT_PID: 107287
2025-04-22T00:19:54.7064622Z ##[endgroup]
2025-04-22T00:19:54.7210303Z Prepare all required actions
2025-04-22T00:19:54.7210694Z Getting action download info
2025-04-22T00:19:54.8461451Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-04-22T00:19:55.3142111Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-04-22T00:19:55.3142461Z with:
2025-04-22T00:19:55.3142860Z   file-suffix: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-22T00:19:55.3143371Z   s3-bucket: gha-artifacts
2025-04-22T00:19:55.3143617Z env:
2025-04-22T00:19:55.3143822Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3144152Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3144731Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3145205Z ##[endgroup]
2025-04-22T00:19:55.3188373Z ##[group]Run # Remove any previous test jsons if they exist
2025-04-22T00:19:55.3188855Z # Remove any previous test jsons if they exist
2025-04-22T00:19:55.3189218Z rm -f test-jsons-*.zip
2025-04-22T00:19:55.3189630Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json'
2025-04-22T00:19:55.3198798Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:55.3199170Z env:
2025-04-22T00:19:55.3199372Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3199698Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3200250Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3200901Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-22T00:19:55.3201367Z ##[endgroup]
2025-04-22T00:19:55.3321173Z   adding: test/test-reports/inference_torchbench.json (deflated 99%)
2025-04-22T00:19:55.3346264Z   adding: test/test-reports/inference_torchbench_graph_breaks.json (deflated 98%)
2025-04-22T00:19:55.3395676Z   adding: test/test-reports/training_torchbench.json (deflated 99%)
2025-04-22T00:19:55.3545364Z   adding: test/test-reports/training_torchbench_graph_breaks.json (deflated 98%)
2025-04-22T00:19:55.3580533Z ##[group]Run # Remove any previous test reports if they exist
2025-04-22T00:19:55.3581019Z # Remove any previous test reports if they exist
2025-04-22T00:19:55.3581412Z rm -f test-reports-*.zip
2025-04-22T00:19:55.3581889Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv'
2025-04-22T00:19:55.3591008Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:55.3591387Z env:
2025-04-22T00:19:55.3591613Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3591949Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3592509Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3593189Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-22T00:19:55.3593756Z ##[endgroup]
2025-04-22T00:19:55.3647372Z   adding: test/test-reports/inference_torchbench.csv (deflated 63%)
2025-04-22T00:19:55.3650978Z   adding: test/test-reports/inference_torchbench_graph_breaks.csv (deflated 92%)
2025-04-22T00:19:55.3652754Z   adding: test/test-reports/inference_torchbench_graph_break_deduped.csv (deflated 77%)
2025-04-22T00:19:55.3653661Z   adding: test/test-reports/training_torchbench.csv (deflated 61%)
2025-04-22T00:19:55.3666179Z   adding: test/test-reports/training_torchbench_graph_breaks.csv (deflated 97%)
2025-04-22T00:19:55.3668689Z   adding: test/test-reports/training_torchbench_graph_break_deduped.csv (deflated 79%)
2025-04-22T00:19:55.3700313Z ##[group]Run # Remove any previous usage logs if they exist
2025-04-22T00:19:55.3700766Z # Remove any previous usage logs if they exist
2025-04-22T00:19:55.3701127Z rm -f logs-*.zip
2025-04-22T00:19:55.3701469Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true
2025-04-22T00:19:55.3702182Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true
2025-04-22T00:19:55.3710404Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:55.3710774Z env:
2025-04-22T00:19:55.3710984Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3711317Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3711868Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3712543Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-22T00:19:55.3713021Z ##[endgroup]
2025-04-22T00:19:55.3832509Z   adding: usage_log.txt (deflated 96%)
2025-04-22T00:19:55.3845424Z 
2025-04-22T00:19:55.3845890Z zip error: Nothing to do! (logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip)
2025-04-22T00:19:55.3875674Z ##[group]Run # Remove any previous debugging artifacts if they exist
2025-04-22T00:19:55.3876190Z # Remove any previous debugging artifacts if they exist
2025-04-22T00:19:55.3876609Z rm -f debug-*.zip
2025-04-22T00:19:55.3876889Z if [ -d 'test/debug' ]; then
2025-04-22T00:19:55.3877240Z   zip -r "debug-${FILE_SUFFIX}.zip" test/debug
2025-04-22T00:19:55.3877580Z fi
2025-04-22T00:19:55.3885533Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:55.3885897Z env:
2025-04-22T00:19:55.3886113Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3886455Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3887007Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3887684Z   FILE_SUFFIX: test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329
2025-04-22T00:19:55.3888164Z ##[endgroup]
2025-04-22T00:19:55.3992919Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-04-22T00:19:55.3993248Z with:
2025-04-22T00:19:55.3993583Z   s3-bucket: gha-artifacts
2025-04-22T00:19:55.3993918Z   s3-prefix: pytorch/pytorch/14580756816/1/artifact

2025-04-22T00:19:55.3994264Z   retention-days: 14
2025-04-22T00:19:55.3994525Z   if-no-files-found: warn
2025-04-22T00:19:55.3994799Z   path: test-jsons-*.zip
2025-04-22T00:19:55.3995057Z   name: artifact
2025-04-22T00:19:55.3995282Z   region: us-east-1
2025-04-22T00:19:55.3995508Z env:
2025-04-22T00:19:55.3995724Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.3996051Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.3996797Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.3997292Z ##[endgroup]
2025-04-22T00:19:55.7622500Z NOTE: s3-prefix specified, ignoring name parameter
2025-04-22T00:19:55.7622960Z With the provided path, there will be 1 file uploaded
2025-04-22T00:19:55.7623404Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact
2025-04-22T00:19:55.7678044Z Starting upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:55.9476239Z Finished upload of test-jsons-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:55.9769969Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-04-22T00:19:55.9770301Z with:
2025-04-22T00:19:55.9770510Z   s3-bucket: gha-artifacts
2025-04-22T00:19:55.9770823Z   s3-prefix: pytorch/pytorch/14580756816/1/artifact

2025-04-22T00:19:55.9771173Z   retention-days: 14
2025-04-22T00:19:55.9771423Z   if-no-files-found: error
2025-04-22T00:19:55.9771706Z   path: test-reports-*.zip
2025-04-22T00:19:55.9771960Z   name: artifact
2025-04-22T00:19:55.9772182Z   region: us-east-1
2025-04-22T00:19:55.9772394Z env:
2025-04-22T00:19:55.9772599Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:55.9772923Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:55.9773461Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:55.9773936Z ##[endgroup]
2025-04-22T00:19:56.3193640Z NOTE: s3-prefix specified, ignoring name parameter
2025-04-22T00:19:56.3194347Z With the provided path, there will be 1 file uploaded
2025-04-22T00:19:56.3194836Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact
2025-04-22T00:19:56.3249260Z Starting upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:56.5083246Z Finished upload of test-reports-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:56.5378932Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-04-22T00:19:56.5379271Z with:
2025-04-22T00:19:56.5379487Z   s3-bucket: gha-artifacts
2025-04-22T00:19:56.5379808Z   s3-prefix: pytorch/pytorch/14580756816/1/artifact

2025-04-22T00:19:56.5380155Z   retention-days: 14
2025-04-22T00:19:56.5380410Z   if-no-files-found: ignore
2025-04-22T00:19:56.5380678Z   path: logs-*.zip
2025-04-22T00:19:56.5380909Z   name: artifact
2025-04-22T00:19:56.5381131Z   region: us-east-1
2025-04-22T00:19:56.5381357Z env:
2025-04-22T00:19:56.5381571Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:56.5381914Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:56.5382462Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:56.5382949Z ##[endgroup]
2025-04-22T00:19:56.8750658Z NOTE: s3-prefix specified, ignoring name parameter
2025-04-22T00:19:56.8751289Z With the provided path, there will be 1 file uploaded
2025-04-22T00:19:56.8751895Z Uploading to s3 prefix: pytorch/pytorch/14580756816/1/artifact
2025-04-22T00:19:56.8806853Z Starting upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:57.0069097Z Finished upload of logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:19:57.0365836Z ##[group]Run seemethere/upload-artifact-s3@v5
2025-04-22T00:19:57.0366170Z with:
2025-04-22T00:19:57.0366387Z   s3-bucket: gha-artifacts
2025-04-22T00:19:57.0366705Z   s3-prefix: pytorch/pytorch/14580756816/1/artifact

2025-04-22T00:19:57.0367071Z   retention-days: 14
2025-04-22T00:19:57.0367322Z   if-no-files-found: ignore
2025-04-22T00:19:57.0367589Z   path: debug-*.zip
2025-04-22T00:19:57.0367819Z   name: artifact
2025-04-22T00:19:57.0368040Z   region: us-east-1
2025-04-22T00:19:57.0368262Z env:
2025-04-22T00:19:57.0368478Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:57.0368807Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:57.0369360Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:57.0370049Z ##[endgroup]
2025-04-22T00:19:57.3672029Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded.
2025-04-22T00:19:57.3986571Z ##[group]Run # shellcheck disable=SC2156
2025-04-22T00:19:57.3986941Z # shellcheck disable=SC2156
2025-04-22T00:19:57.3987493Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-04-22T00:19:57.3997465Z shell: /usr/bin/bash -e {0}
2025-04-22T00:19:57.3997732Z env:
2025-04-22T00:19:57.3997947Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:57.3998274Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:57.3998814Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:57.3999291Z ##[endgroup]
2025-04-22T00:19:57.6947886Z Prepare all required actions
2025-04-22T00:19:57.6948254Z Getting action download info
2025-04-22T00:19:57.7957150Z ##[group]Run ./.github/actions/upload-utilization-stats
2025-04-22T00:19:57.7957518Z with:
2025-04-22T00:19:57.7957731Z   job_id: 40900324329
2025-04-22T00:19:57.7958205Z   job_name: cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-22T00:19:57.7958738Z   workflow_name: inductor
2025-04-22T00:19:57.7959014Z   workflow_run_id: 14580756816
2025-04-22T00:19:57.7959301Z   workflow_attempt: 1
2025-04-22T00:19:57.7959547Z env:
2025-04-22T00:19:57.7959758Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:57.7960091Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:57.7960646Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:57.7961136Z ##[endgroup]
2025-04-22T00:19:57.7985116Z ##[group]Run echo "workflow_id: 14580756816"
2025-04-22T00:19:57.7985486Z echo "workflow_id: 14580756816"
2025-04-22T00:19:57.7985821Z echo "workflow_attempt: 1"
2025-04-22T00:19:57.7986186Z echo "workflow_Name: inductor"
2025-04-22T00:19:57.7986499Z echo "job_id: 40900324329"
2025-04-22T00:19:57.7987044Z echo "job_name:  cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)"
2025-04-22T00:19:57.7996809Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:57.7997181Z env:
2025-04-22T00:19:57.7997398Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:57.7997734Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:57.7998282Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:57.7998764Z ##[endgroup]
2025-04-22T00:19:57.8026825Z workflow_id: 14580756816
2025-04-22T00:19:57.8027228Z workflow_attempt: 1
2025-04-22T00:19:57.8027565Z workflow_Name: inductor
2025-04-22T00:19:57.8027911Z job_id: 40900324329
2025-04-22T00:19:57.8028545Z job_name:  cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)
2025-04-22T00:19:57.8081108Z ##[group]Run nick-fields/retry@v3.0.0
2025-04-22T00:19:57.8081415Z with:
2025-04-22T00:19:57.8081618Z   shell: bash
2025-04-22T00:19:57.8081853Z   timeout_minutes: 5
2025-04-22T00:19:57.8082096Z   max_attempts: 5
2025-04-22T00:19:57.8082328Z   retry_wait_seconds: 30
2025-04-22T00:19:57.8082789Z   command: set -eu
python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3

2025-04-22T00:19:57.8083302Z   polling_interval_seconds: 1
2025-04-22T00:19:57.8083581Z   warning_on_retry: true
2025-04-22T00:19:57.8083845Z   continue_on_error: false
2025-04-22T00:19:57.8084102Z env:
2025-04-22T00:19:57.8084313Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:57.8084637Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:57.8085191Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:57.8085687Z ##[endgroup]
2025-04-22T00:19:58.1414778Z Defaulting to user installation because normal site-packages is not writeable
2025-04-22T00:19:58.1603355Z Requirement already satisfied: python-dateutil==2.8.2 in /home/ec2-user/.local/lib/python3.9/site-packages (2.8.2)
2025-04-22T00:19:59.2539334Z Collecting boto3==1.35.42
2025-04-22T00:19:59.2571806Z   Using cached boto3-1.35.42-py3-none-any.whl (139 kB)
2025-04-22T00:19:59.2595240Z Requirement already satisfied: pandas==2.1.3 in /home/ec2-user/.local/lib/python3.9/site-packages (2.1.3)
2025-04-22T00:19:59.2609922Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0)
2025-04-22T00:19:59.2655373Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4)
2025-04-22T00:19:59.2660222Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99)
2025-04-22T00:19:59.2664216Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0)
2025-04-22T00:19:59.3532419Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1)
2025-04-22T00:19:59.3541469Z Requirement already satisfied: numpy<2,>=1.22.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (1.26.4)
2025-04-22T00:19:59.3547704Z Requirement already satisfied: tzdata>=2022.1 in /home/ec2-user/.local/lib/python3.9/site-packages (from pandas==2.1.3) (2025.2)
2025-04-22T00:19:59.3606361Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10)
2025-04-22T00:19:59.4436508Z Installing collected packages: boto3
2025-04-22T00:19:59.4436837Z   Attempting uninstall: boto3
2025-04-22T00:19:59.4439227Z     Found existing installation: boto3 1.35.33
2025-04-22T00:19:59.4565985Z     Uninstalling boto3-1.35.33:
2025-04-22T00:19:59.4581504Z       Successfully uninstalled boto3-1.35.33
2025-04-22T00:19:59.5185887Z Successfully installed boto3-1.35.42
2025-04-22T00:19:59.8875182Z Command completed after 1 attempt(s).
2025-04-22T00:19:59.8952925Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \
2025-04-22T00:19:59.8953677Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \
2025-04-22T00:19:59.8954166Z   --workflow-run-id "14580756816" \
2025-04-22T00:19:59.8954511Z   --workflow-name "inductor" \
2025-04-22T00:19:59.8954846Z   --workflow-run-attempt "1" \
2025-04-22T00:19:59.8955152Z   --job-id "40900324329" \
2025-04-22T00:19:59.8955689Z   --job-name "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)"
2025-04-22T00:19:59.8964736Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:19:59.8965114Z env:
2025-04-22T00:19:59.8965327Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:19:59.8965656Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:19:59.8966209Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:19:59.8966692Z ##[endgroup]
2025-04-22T00:20:02.8805013Z repo: pytorch/pytorch
2025-04-22T00:20:02.8805527Z Downloading logs-test-inductor_torchbench-2-2-linux.g5.4xlarge.nvidia.gpu_40900324329.zip
2025-04-22T00:20:02.8806107Z Converted Log Model: UtilizationMetadata:
2025-04-22T00:20:02.8807499Z  UtilizationMetadata(level='metadata', workflow_id='14580756816', job_id='40900324329', workflow_name='inductor', job_name='cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)', usage_collect_interval=1.0, data_model_version=1.0, start_at=1745272217, gpu_count=1, cpu_count=16, gpu_type='pynvml', error=None)
2025-04-22T00:20:02.8808934Z [Db Segments] detected pytest cmd: 19, generated segments: 19
2025-04-22T00:20:02.8809345Z [db model] Peek db timeseries 
2025-04-22T00:20:02.8809615Z :{
2025-04-22T00:20:02.8809824Z     "created_at": 1745281202,
2025-04-22T00:20:02.8810387Z     "type": "utilization",
2025-04-22T00:20:02.8810640Z     "tags": [
2025-04-22T00:20:02.8810848Z         "record"
2025-04-22T00:20:02.8811059Z     ],
2025-04-22T00:20:02.8811258Z     "time_stamp": 1745272217,
2025-04-22T00:20:02.8811529Z     "repo": "pytorch/pytorch",
2025-04-22T00:20:02.8811805Z     "workflow_id": 14580756816,
2025-04-22T00:20:02.8812062Z     "run_attempt": 1,
2025-04-22T00:20:02.8812298Z     "job_id": 40900324329,
2025-04-22T00:20:02.8812558Z     "workflow_name": "inductor",
2025-04-22T00:20:02.8813056Z     "job_name": "cuda12.6-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu)",
2025-04-22T00:20:02.8813574Z     "json_data": "{}"
2025-04-22T00:20:02.8813802Z }
2025-04-22T00:20:02.8814461Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/14580756816/1/40900324329/metadata
2025-04-22T00:20:02.8815335Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.0/pytorch/pytorch/14580756816/1/40900324329/metadata 
2025-04-22T00:20:02.8816244Z Writing 1782 documents to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/14580756816/1/40900324329/time_series
2025-04-22T00:20:02.8817180Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.0/pytorch/pytorch/14580756816/1/40900324329/time_series 
2025-04-22T00:20:02.9992839Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2025-04-22T00:20:02.9993263Z with:
2025-04-22T00:20:02.9993525Z env:
2025-04-22T00:20:02.9993734Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:20:02.9994069Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:20:02.9994617Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:20:02.9995100Z ##[endgroup]
2025-04-22T00:20:03.0020384Z ##[group]Run set -eou pipefail
2025-04-22T00:20:03.0020697Z set -eou pipefail
2025-04-22T00:20:03.0020965Z 
2025-04-22T00:20:03.0021321Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2025-04-22T00:20:03.0022155Z for _ in $(seq 1440); do
2025-04-22T00:20:03.0022492Z     # Break if no ssh session exists anymore
2025-04-22T00:20:03.0022842Z     if [ "$(who)" = "" ]; then
2025-04-22T00:20:03.0023134Z       break
2025-04-22T00:20:03.0023402Z     fi
2025-04-22T00:20:03.0023635Z     echo "."
2025-04-22T00:20:03.0023880Z     sleep 5
2025-04-22T00:20:03.0024121Z done
2025-04-22T00:20:03.0033045Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:20:03.0033485Z env:
2025-04-22T00:20:03.0033700Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:20:03.0034031Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:20:03.0034568Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:20:03.0035044Z ##[endgroup]
2025-04-22T00:20:03.0061810Z Holding runner for 2 hours until all ssh sessions have logged out
2025-04-22T00:20:03.0155294Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-04-22T00:20:03.0155988Z # ignore expansion of "docker ps -q" since it could be empty
2025-04-22T00:20:03.0156407Z # shellcheck disable=SC2046
2025-04-22T00:20:03.0156754Z docker stop $(docker ps -q) || true
2025-04-22T00:20:03.0157087Z # Prune all of the docker images
2025-04-22T00:20:03.0157413Z docker system prune -af
2025-04-22T00:20:03.0166038Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:20:03.0166401Z env:
2025-04-22T00:20:03.0166615Z   GIT_DEFAULT_BRANCH: main
2025-04-22T00:20:03.0166945Z   GPU_FLAG: --gpus all -e NVIDIA_DRIVER_CAPABILITIES=all
2025-04-22T00:20:03.0167476Z   DOCKER_CONTAINER_ID: 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:20:03.0167964Z ##[endgroup]
2025-04-22T00:20:04.3270429Z 3e061bd09dd7
2025-04-22T00:20:08.3497428Z Deleted Containers:
2025-04-22T00:20:08.3498156Z 3e061bd09dd70e06b051b0a4d0dcfb7ee37acc24678b5591725f8195ab221746
2025-04-22T00:20:08.3498492Z 
2025-04-22T00:20:24.2019444Z Deleted Images:
2025-04-22T00:20:24.2020664Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks:f583c1656c56372ad0e1c57f3463a017fbf03138
2025-04-22T00:20:24.2022776Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-focal-cuda12.6-cudnn9-py3-gcc9-inductor-benchmarks@sha256:745e08885687216ce5adf3196dea7f49a9467042491210addb990693d7fef90a
2025-04-22T00:20:24.2024055Z deleted: sha256:7b29eacbf93f5bdd9e6b666f27edb34c0d2f289aa58ceaf3bdfcf4e06f505e55
2025-04-22T00:20:24.2024689Z deleted: sha256:0d03769fa66092351763fc95d7ff70e856e1c6dad39dffd06cd3cc6f705c2b90
2025-04-22T00:20:24.2025303Z deleted: sha256:03ad561430a1da2dad623afcca922b9bf999b9beec551534fb227d2905d98683
2025-04-22T00:20:24.2025975Z deleted: sha256:8999b0bbe1ce0aa447deea13eab8ab1f8c7e23d827f1cfb9ea3eda2d5862ff30
2025-04-22T00:20:24.2026613Z deleted: sha256:8c9a34c905c38106c707a3de05f76313f1dc1b5480e33030262e0c036fd86e2e
2025-04-22T00:20:24.2027217Z deleted: sha256:7aa296e90575147a3bdf977176396c671082fb5dc5c240a22ac72b9e1bc0e268
2025-04-22T00:20:24.2027826Z deleted: sha256:35429085a82ca073e718dc4ecc1c57868bb6c3ca77201e12ea81aa653bdde323
2025-04-22T00:20:24.2028972Z deleted: sha256:9a73de98215b59030f9b38fd04ed9f006b41baa1a1df96e1b130fd6771aafa70
2025-04-22T00:20:24.2029593Z deleted: sha256:aa989e7f91a166b7b6f2138d15d3a9eab2852c094c0f76f2ef4b85a015d5247a
2025-04-22T00:20:24.2030216Z deleted: sha256:aab94433009bfa1b6b672cbf53be6cfcc67e12bae3fccb39392f2ba356fed8e8
2025-04-22T00:20:24.2030825Z deleted: sha256:cf4b1db59c30d2e1191b22f938168f28c237ca50a4479f4bd96300300167a10d
2025-04-22T00:20:24.2031506Z deleted: sha256:d80a61f49181681ea66aabe7e2bd1a43e3652d30fbe47699b3c75bc320b3fd86
2025-04-22T00:20:24.2032135Z deleted: sha256:7ce0d2a5b6420f3c29e66c3a4c4e37437e961ea57a801549d154c17377b9cbe7
2025-04-22T00:20:24.2032750Z deleted: sha256:b035e1e0eefb5d757d8642ca66b5a56d8acd60b27793d8bb0fee024b61424f18
2025-04-22T00:20:24.2033442Z deleted: sha256:e9f028a07e4a18a1eb31d1377ffdc18189e530d827ec085900dab9634404cbfd
2025-04-22T00:20:24.2034057Z deleted: sha256:75a2b3d5fb5eabe18f21153ffe34412d0f6adb78a89cbb579271dc29a107109f
2025-04-22T00:20:24.2034666Z deleted: sha256:980637ca90042fdf0d35c1aa5459851186b2d884c583ac0c6018565e4123275d
2025-04-22T00:20:24.2035268Z deleted: sha256:353541bdf9a9dc69f68d46df3e6af379ce0d8fb710e2ae3ed989b4fd73a7f69e
2025-04-22T00:20:24.2035925Z deleted: sha256:9739cf32fe6a869643327bf4f2ddbdfd714451c07e0463e887cfc7827acf81b8
2025-04-22T00:20:24.2036569Z deleted: sha256:ea8da7f58fe158b166472f208d4787bd84a5f24715f3a3693b2ce09db3b56870
2025-04-22T00:20:24.2037165Z deleted: sha256:64d128ae465a7433018bb4f041b913f2c3d4fe4f7a4191fb15a976492613d25a
2025-04-22T00:20:24.2037759Z deleted: sha256:407bfba786761588aea6fe18b2486cd23dddd885d72c60a41949d7dfc6a942d5
2025-04-22T00:20:24.2038356Z deleted: sha256:9889d65ed67766c496c923d00c5e6fcf5d77647d1384b278c39e389fc2a99f71
2025-04-22T00:20:24.2038962Z deleted: sha256:885a49a89eaa7cd605832bad81aaef59d663c8b6a9941c4fd8a56414a5c15361
2025-04-22T00:20:24.2039567Z deleted: sha256:a754629243d15f78ad5b02c613ca5b138bae988a653e77ab7d7a1c2b1ff8430b
2025-04-22T00:20:24.2040176Z deleted: sha256:69986b2cde5dc2eedee2c34cc9a0f2862bd0bb76de325a159b3eb286e31ab267
2025-04-22T00:20:24.2040821Z deleted: sha256:1eb1b82d13079306ebed0331351fb0299aab14b537bd38ac3dfe9f86daba4a7c
2025-04-22T00:20:24.2041476Z deleted: sha256:3a53d5f678b95144bfb2f43df7c8f62c6eacdc36c4c9278d2672658ba981a6f1
2025-04-22T00:20:24.2042065Z deleted: sha256:816f3880d16e691a376b22135e30c5c92e77a2390692a2aec75c92a61a1028d6
2025-04-22T00:20:24.2042661Z deleted: sha256:6e9897a601a0378a8a2499be37549ea2df6a9dfbec84fad00d2e0796b0d8c897
2025-04-22T00:20:24.2043264Z deleted: sha256:4253008065add4d2e0a51dd0dc024420211cfa90f52ccf2fb3cd59a97941e820
2025-04-22T00:20:24.2043871Z deleted: sha256:84fc991bd2259ddde33c56bd6d83cd6856ce393c638a21f6c61183206847f599
2025-04-22T00:20:24.2044669Z deleted: sha256:52c94651685b1b2c788a091c20c8325005b4cb03693709ff02b9e659e790f122
2025-04-22T00:20:24.2045256Z deleted: sha256:afa943b0954b848b45f79913b746502e986aba75e5a7393fb4569f3450da4315
2025-04-22T00:20:24.2045948Z deleted: sha256:b3d00015deda08e79f7f4c3b4bf277c14ddf7cc07986f983b4e160ea069ffc47
2025-04-22T00:20:24.2046559Z deleted: sha256:e3b8a4f8a44b3187a6f78df4ee606a2a8ec8060bfca4163e47c3db94ff685c9e
2025-04-22T00:20:24.2047169Z deleted: sha256:5ad2fc574f42e35faf6c336d17334edc2e67b6601a49fbe3c12000a8091da6e8
2025-04-22T00:20:24.2047780Z deleted: sha256:397028d746ad951508252fafab3946aa1b8825cd1d8edb6c947cc54c9db7576a
2025-04-22T00:20:24.2048366Z deleted: sha256:454a57acb38698129b7675d5e6d4358906230673548a777f8a3dec9c4fab8500
2025-04-22T00:20:24.2048953Z deleted: sha256:95f1c8b10ced79e9ce5a542b118703418c66d52cc43cc8e7c9405916383bb5ac
2025-04-22T00:20:24.2049567Z deleted: sha256:06af89fbc9fb5a1ff9cc0ddaa04378aa25ac8a7d9e16b77b6aa6a48558d10cd6
2025-04-22T00:20:24.2050197Z deleted: sha256:d4d7f04a8b38bc30eb120874f5d53b51aefc2754f34b43cd5bcad3b58f14791d
2025-04-22T00:20:24.2050809Z deleted: sha256:a39fece22c881dde7689a0873a48724a18471faf44ad4de34c5987032a2f8aed
2025-04-22T00:20:24.2051401Z deleted: sha256:0b59474a1f61590a4946eb94287e7b9531ed471d49b34516ada330bf6e62ef3d
2025-04-22T00:20:24.2052246Z deleted: sha256:76af19197c611f8dd2aec8623ffa2ae9dfc96031e57c885acbf763d47cd4e10a
2025-04-22T00:20:24.2052872Z deleted: sha256:cec724490b7de23aa9af56de906b95d0722b33e990443300bc5fbaafd2d6ce7f
2025-04-22T00:20:24.2053480Z deleted: sha256:993012f9b10558b092cbaa6afb2f8de46c856adf9f39ccb3affb1ab88b4034f5
2025-04-22T00:20:24.2054087Z deleted: sha256:c03f3c99e80a992966c8a13067ca8d551914f2016063aabd58e10fc66a0b4aa7
2025-04-22T00:20:24.2054696Z deleted: sha256:a3826acb0870ba3e5dcb72651ebbae184d8964a70dfc99d88a6ada64df52070a
2025-04-22T00:20:24.2055315Z deleted: sha256:59eff532c4a87d4492cfb7a2dca5f091aba58f41f09dd44d78ed251566552fc0
2025-04-22T00:20:24.2055915Z deleted: sha256:7af9312575f7d280f4e404bfa4229d29b2f863308cd2847737dab0a80f912abd
2025-04-22T00:20:24.2056515Z deleted: sha256:255b3dc15181408481cc137ae98284eb8bbe550185f4c3600538c7f27cb97630
2025-04-22T00:20:24.2057106Z deleted: sha256:a46c03834bc35bcd52270359c7503317e2517a1bef96d6174b37510d49240a96
2025-04-22T00:20:24.2057703Z deleted: sha256:697ce89585f3321899a308e7d5fa44cfd01661762b83d4886fe6c7df23067deb
2025-04-22T00:20:24.2058312Z deleted: sha256:fe1a83ec01181b320ed6d1d9d96c7b05e0ec90ecae542be803fc5a55fd49563c
2025-04-22T00:20:24.2058921Z deleted: sha256:47a48329e8ee10fb1592b7f54749c23245dfdbdecb46c08c4b79dd9b141a13b9
2025-04-22T00:20:24.2059529Z deleted: sha256:b7b7528c3adac4746b228449b90274fd02e75b69ff3f626366683dea239108b4
2025-04-22T00:20:24.2060123Z deleted: sha256:b1563735209b77977fec8743d8915d3bb4acecdce565bf90f42e43b7a9d41eba
2025-04-22T00:20:24.2060734Z deleted: sha256:8f3dad83b005c9344155ed29ce5613f918718d2ceab31a2fc20adf404021a9ee
2025-04-22T00:20:24.2061438Z deleted: sha256:d19dff217a30a035796fe482ed413f6eda45d02ca46daa5cd9ce30ac4a5b86a9
2025-04-22T00:20:24.2062056Z deleted: sha256:ec94e4b914db9f6e5913eb4a203bf4c14cf9dffa794aaf0d01426e1d2d540662
2025-04-22T00:20:24.2062665Z deleted: sha256:94e49b5119d7680f10fa65841158de93cc070c7b19bb83c566b874158bec2fc3
2025-04-22T00:20:24.2063262Z deleted: sha256:6866ef40cf6f50d4d4c3823444758606f80314d683f650be1b6d69eff2d35329
2025-04-22T00:20:24.2063864Z deleted: sha256:f06c4dcc941cd4fa9c6b0f507c8be752e7be6b2191a3332128ab32fdb8c613eb
2025-04-22T00:20:24.2064482Z deleted: sha256:e7a9c0fbc4da83cebcbeab6a1ffc42508561041f97259ff2cf2e2e14a73d47bd
2025-04-22T00:20:24.2065093Z deleted: sha256:67f07ac2be3331b5421f04c08746bdb4ce5952c7094018813bdc134ddefd5ec4
2025-04-22T00:20:24.2065713Z deleted: sha256:9a1906e08abd2cbaad6411b296796a97964e0f89011633541b1fd2ac24e8d53a
2025-04-22T00:20:24.2066336Z deleted: sha256:594048c05b4663fff16eed21ea2fe1cba949ca1b0426b134bb275181ab727272
2025-04-22T00:20:24.2066932Z deleted: sha256:9ab83b5a3614325d1733a24e22eeae2e7ecd37cd0d0024660d9b43265e7c496b
2025-04-22T00:20:24.2067643Z deleted: sha256:de82c3b65910240582bad1cbc96209ab68e59c252450a7d42aad3a0673b154f4
2025-04-22T00:20:24.2068250Z deleted: sha256:46bc902662130b8d13f25b0294fac16ca6248034d5ccfaf3ee51cb56572946c5
2025-04-22T00:20:24.2068855Z deleted: sha256:43659ee56fac7f221f9388ec263ad2042847103213025f0a7a0ec7ca6a81b20f
2025-04-22T00:20:24.2069460Z deleted: sha256:5839c16cf80c38c4f69268b2c73c3d7ec37a33e18c9c5f49e6624239a182dbf8
2025-04-22T00:20:24.2070062Z deleted: sha256:e16d8e25c4393c96891af6d6a42889638fc30a7490ef9c5c5517798ea9860bda
2025-04-22T00:20:24.2070660Z deleted: sha256:868539926d1d83fe2b4fdb0231d8b57fb5b8b34144d733a4cf862d241e16c667
2025-04-22T00:20:24.2071340Z deleted: sha256:cc4224d01139aef3aeb29c568d184283602d52f47bb8e5e29de97bf6aa54d951
2025-04-22T00:20:24.2071953Z deleted: sha256:bd18b4fe898c9279c6e9ad952165522e24de55ab48d83289ca0ff6c8df97d85d
2025-04-22T00:20:24.2072564Z deleted: sha256:c7917cebb757d994f33b0f4bbad70fbb563939c2630d8f96113d437fa09e68cd
2025-04-22T00:20:24.2073171Z deleted: sha256:34fb1ecb50c48f883ea450e0364c90bdf16ce5e35e8a132012851ac3b040addf
2025-04-22T00:20:24.2073854Z deleted: sha256:92300fc3f1a943c0b6a295d87da98cb3d593b18b8ddc4f0b229e3ca909d03206
2025-04-22T00:20:24.2074656Z deleted: sha256:fffe76c64ef2dee2d80a8bb3ad13d65d596d04a45510b1956a976a69215dae92
2025-04-22T00:20:24.2075047Z 
2025-04-22T00:20:24.2075288Z Total reclaimed space: 51.93GB
2025-04-22T00:20:24.2184395Z Post job cleanup.
2025-04-22T00:20:24.2236277Z Post job cleanup.
2025-04-22T00:20:24.3289552Z [command]/usr/bin/git version
2025-04-22T00:20:24.3352364Z git version 2.47.1
2025-04-22T00:20:24.3391627Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/349aac35-e712-4461-8cdf-f817df303fc9/.gitconfig'
2025-04-22T00:20:24.3402931Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/349aac35-e712-4461-8cdf-f817df303fc9' before making global git config changes
2025-04-22T00:20:24.3403830Z Adding repository directory to the temporary git global config as a safe directory
2025-04-22T00:20:24.3408917Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2025-04-22T00:20:24.3454126Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-04-22T00:20:24.3491538Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-04-22T00:20:24.3868505Z Entering 'android/libs/fbjni'
2025-04-22T00:20:24.3939886Z Entering 'third_party/FP16'
2025-04-22T00:20:24.4011060Z Entering 'third_party/FXdiv'
2025-04-22T00:20:24.4081924Z Entering 'third_party/NNPACK'
2025-04-22T00:20:24.4151671Z Entering 'third_party/NVTX'
2025-04-22T00:20:24.4222650Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-22T00:20:24.4291854Z Entering 'third_party/XNNPACK'
2025-04-22T00:20:24.4375789Z Entering 'third_party/benchmark'
2025-04-22T00:20:24.4450746Z Entering 'third_party/composable_kernel'
2025-04-22T00:20:24.4528533Z Entering 'third_party/cpp-httplib'
2025-04-22T00:20:24.4599528Z Entering 'third_party/cpuinfo'
2025-04-22T00:20:24.4672094Z Entering 'third_party/cudnn_frontend'
2025-04-22T00:20:24.4743186Z Entering 'third_party/cutlass'
2025-04-22T00:20:24.4821097Z Entering 'third_party/eigen'
2025-04-22T00:20:24.4893929Z Entering 'third_party/fbgemm'
2025-04-22T00:20:24.4964107Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-22T00:20:24.5031021Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-22T00:20:24.5098608Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-22T00:20:24.5173641Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-22T00:20:24.5240966Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-22T00:20:24.5311543Z Entering 'third_party/flash-attention'
2025-04-22T00:20:24.5386710Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-22T00:20:24.5461989Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-22T00:20:24.5539798Z Entering 'third_party/flatbuffers'
2025-04-22T00:20:24.5611923Z Entering 'third_party/fmt'
2025-04-22T00:20:24.5682846Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-22T00:20:24.5753932Z Entering 'third_party/gloo'
2025-04-22T00:20:24.5823682Z Entering 'third_party/googletest'
2025-04-22T00:20:24.5893036Z Entering 'third_party/ideep'
2025-04-22T00:20:24.5960647Z Entering 'third_party/ideep/mkl-dnn'
2025-04-22T00:20:24.6040687Z Entering 'third_party/ittapi'
2025-04-22T00:20:24.6109823Z Entering 'third_party/kineto'
2025-04-22T00:20:24.6176049Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-22T00:20:24.6241773Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-22T00:20:24.6310277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-22T00:20:24.6379235Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-22T00:20:24.6447538Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-22T00:20:24.6513975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-22T00:20:24.6586815Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-22T00:20:24.6655570Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-22T00:20:24.6729526Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-22T00:20:24.6798996Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-22T00:20:24.6869459Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-22T00:20:24.6940197Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-22T00:20:24.7011610Z Entering 'third_party/kleidiai'
2025-04-22T00:20:24.7082631Z Entering 'third_party/mimalloc'
2025-04-22T00:20:24.7152526Z Entering 'third_party/nlohmann'
2025-04-22T00:20:24.7223673Z Entering 'third_party/onnx'
2025-04-22T00:20:24.7308270Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-22T00:20:24.7385855Z Entering 'third_party/opentelemetry-cpp'
2025-04-22T00:20:24.7455732Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-22T00:20:24.7527899Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-22T00:20:24.7599821Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-22T00:20:24.7666479Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-22T00:20:24.7742416Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-22T00:20:24.7808123Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-22T00:20:24.7875288Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-22T00:20:24.7944525Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-22T00:20:24.8013317Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-22T00:20:24.8083166Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-22T00:20:24.8173156Z Entering 'third_party/pocketfft'
2025-04-22T00:20:24.8243440Z Entering 'third_party/protobuf'
2025-04-22T00:20:24.8315428Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-22T00:20:24.8386539Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-22T00:20:24.8463928Z Entering 'third_party/psimd'
2025-04-22T00:20:24.8532992Z Entering 'third_party/pthreadpool'
2025-04-22T00:20:24.8603048Z Entering 'third_party/pybind11'
2025-04-22T00:20:24.8677385Z Entering 'third_party/python-peachpy'
2025-04-22T00:20:24.8751256Z Entering 'third_party/sleef'
2025-04-22T00:20:24.8820560Z Entering 'third_party/tensorpipe'
2025-04-22T00:20:24.8888341Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-22T00:20:24.8954059Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-22T00:20:24.9020731Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-22T00:20:24.9091483Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-22T00:20:24.9157635Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-22T00:20:24.9252411Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-04-22T00:20:24.9277840Z http.https://github.com/.extraheader
2025-04-22T00:20:24.9290119Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-04-22T00:20:24.9325951Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-04-22T00:20:24.9684068Z Entering 'android/libs/fbjni'
2025-04-22T00:20:24.9729956Z http.https://github.com/.extraheader
2025-04-22T00:20:24.9774741Z Entering 'third_party/FP16'
2025-04-22T00:20:24.9820288Z http.https://github.com/.extraheader
2025-04-22T00:20:24.9863947Z Entering 'third_party/FXdiv'
2025-04-22T00:20:24.9913200Z http.https://github.com/.extraheader
2025-04-22T00:20:24.9958858Z Entering 'third_party/NNPACK'
2025-04-22T00:20:25.0002635Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0048049Z Entering 'third_party/NVTX'
2025-04-22T00:20:25.0095667Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0139951Z Entering 'third_party/VulkanMemoryAllocator'
2025-04-22T00:20:25.0184142Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0228688Z Entering 'third_party/XNNPACK'
2025-04-22T00:20:25.0276422Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0336854Z Entering 'third_party/benchmark'
2025-04-22T00:20:25.0381162Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0424485Z Entering 'third_party/composable_kernel'
2025-04-22T00:20:25.0469066Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0518845Z Entering 'third_party/cpp-httplib'
2025-04-22T00:20:25.0562785Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0606202Z Entering 'third_party/cpuinfo'
2025-04-22T00:20:25.0650049Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0695520Z Entering 'third_party/cudnn_frontend'
2025-04-22T00:20:25.0744364Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0787950Z Entering 'third_party/cutlass'
2025-04-22T00:20:25.0832579Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0885626Z Entering 'third_party/eigen'
2025-04-22T00:20:25.0930539Z http.https://github.com/.extraheader
2025-04-22T00:20:25.0976600Z Entering 'third_party/fbgemm'
2025-04-22T00:20:25.1025361Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1069084Z Entering 'third_party/fbgemm/third_party/asmjit'
2025-04-22T00:20:25.1112773Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1157190Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2025-04-22T00:20:25.1200528Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1244371Z Entering 'third_party/fbgemm/third_party/cutlass'
2025-04-22T00:20:25.1286949Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1337274Z Entering 'third_party/fbgemm/third_party/googletest'
2025-04-22T00:20:25.1380468Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1424639Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2025-04-22T00:20:25.1468158Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1514689Z Entering 'third_party/flash-attention'
2025-04-22T00:20:25.1563918Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1606945Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-04-22T00:20:25.1649910Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1698659Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-04-22T00:20:25.1742357Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1796553Z Entering 'third_party/flatbuffers'
2025-04-22T00:20:25.1839966Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1887934Z Entering 'third_party/fmt'
2025-04-22T00:20:25.1934995Z http.https://github.com/.extraheader
2025-04-22T00:20:25.1981850Z Entering 'third_party/gemmlowp/gemmlowp'
2025-04-22T00:20:25.2026391Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2070742Z Entering 'third_party/gloo'
2025-04-22T00:20:25.2114260Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2159579Z Entering 'third_party/googletest'
2025-04-22T00:20:25.2203162Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2247015Z Entering 'third_party/ideep'
2025-04-22T00:20:25.2290945Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2332575Z Entering 'third_party/ideep/mkl-dnn'
2025-04-22T00:20:25.2374828Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2427066Z Entering 'third_party/ittapi'
2025-04-22T00:20:25.2470801Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2513896Z Entering 'third_party/kineto'
2025-04-22T00:20:25.2558777Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2600783Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-04-22T00:20:25.2645572Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2690764Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-04-22T00:20:25.2735074Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2779838Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-04-22T00:20:25.2823656Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2868486Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-04-22T00:20:25.2911843Z http.https://github.com/.extraheader
2025-04-22T00:20:25.2956002Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-04-22T00:20:25.3004030Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3046334Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-04-22T00:20:25.3090509Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3138824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-04-22T00:20:25.3181930Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3226022Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-04-22T00:20:25.3274288Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3318674Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-04-22T00:20:25.3362918Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3408524Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-04-22T00:20:25.3452429Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3502494Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-04-22T00:20:25.3546412Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3589889Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-04-22T00:20:25.3635220Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3682418Z Entering 'third_party/kleidiai'
2025-04-22T00:20:25.3726452Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3771683Z Entering 'third_party/mimalloc'
2025-04-22T00:20:25.3816017Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3860425Z Entering 'third_party/nlohmann'
2025-04-22T00:20:25.3903581Z http.https://github.com/.extraheader
2025-04-22T00:20:25.3948814Z Entering 'third_party/onnx'
2025-04-22T00:20:25.3992222Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4050269Z Entering 'third_party/onnx/third_party/pybind11'
2025-04-22T00:20:25.4093645Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4142863Z Entering 'third_party/opentelemetry-cpp'
2025-04-22T00:20:25.4187084Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4231514Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-04-22T00:20:25.4274457Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4317887Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-04-22T00:20:25.4360474Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4404001Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-04-22T00:20:25.4446905Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4489333Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-04-22T00:20:25.4531783Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4577394Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-04-22T00:20:25.4619111Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4663238Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-04-22T00:20:25.4705057Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4750524Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-04-22T00:20:25.4792611Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4834626Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-04-22T00:20:25.4878570Z http.https://github.com/.extraheader
2025-04-22T00:20:25.4923941Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-04-22T00:20:25.4967245Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5014974Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-04-22T00:20:25.5060149Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5129106Z Entering 'third_party/pocketfft'
2025-04-22T00:20:25.5173605Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5217703Z Entering 'third_party/protobuf'
2025-04-22T00:20:25.5266626Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5315260Z Entering 'third_party/protobuf/third_party/benchmark'
2025-04-22T00:20:25.5363734Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5407191Z Entering 'third_party/protobuf/third_party/googletest'
2025-04-22T00:20:25.5450872Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5499001Z Entering 'third_party/psimd'
2025-04-22T00:20:25.5546249Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5590078Z Entering 'third_party/pthreadpool'
2025-04-22T00:20:25.5635142Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5681132Z Entering 'third_party/pybind11'
2025-04-22T00:20:25.5728967Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5772534Z Entering 'third_party/python-peachpy'
2025-04-22T00:20:25.5816343Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5862741Z Entering 'third_party/sleef'
2025-04-22T00:20:25.5907437Z http.https://github.com/.extraheader
2025-04-22T00:20:25.5953923Z Entering 'third_party/tensorpipe'
2025-04-22T00:20:25.5997951Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6042363Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-04-22T00:20:25.6085178Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6128928Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-04-22T00:20:25.6171361Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6213687Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-04-22T00:20:25.6257179Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6302658Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-04-22T00:20:25.6345610Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6388036Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-04-22T00:20:25.6433279Z http.https://github.com/.extraheader
2025-04-22T00:20:25.6598763Z A job completed hook has been configured by the self-hosted runner administrator
2025-04-22T00:20:25.6628071Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh'
2025-04-22T00:20:25.6636012Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-04-22T00:20:25.6636390Z ##[endgroup]
2025-04-22T00:20:32.7037189Z Cleaning up orphan processes