2025-07-24T04:25:03.4803042Z Current runner version: '2.326.0'
2025-07-24T04:25:03.4810193Z Runner name: 'gpu9455'
2025-07-24T04:25:03.4811171Z Runner group name: 'linux.rocm.gpu.group'
2025-07-24T04:25:03.4812302Z Machine name: 'gpu9455'
2025-07-24T04:25:03.4816455Z ##[group]GITHUB_TOKEN Permissions
2025-07-24T04:25:03.4819173Z Contents: read
2025-07-24T04:25:03.4819939Z Metadata: read
2025-07-24T04:25:03.4820574Z ##[endgroup]
2025-07-24T04:25:03.4823223Z Secret source: Actions
2025-07-24T04:25:03.4824336Z Prepare workflow directory
2025-07-24T04:25:04.1740390Z Prepare all required actions
2025-07-24T04:25:04.1856834Z Getting action download info
2025-07-24T04:25:04.5355602Z Download action repository 'pytorch/pytorch@main' (SHA:e09d80c545d9f006f9fa28de6144531bb2af33ff)
2025-07-24T04:25:09.6636733Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-07-24T04:25:10.2380864Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-07-24T04:25:10.7157890Z Download action repository 'pytorch/test-infra@main' (SHA:e32c622fbd62d34f173b9f131f1eca78782b2be2)
2025-07-24T04:25:11.8368073Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-07-24T04:25:12.6256361Z Getting action download info
2025-07-24T04:25:12.7701066Z Download action repository 'actions/checkout@v4' (SHA:11bd71901bbe5b1630ceea73d27597364c9af683)
2025-07-24T04:25:13.3809844Z Getting action download info
2025-07-24T04:25:13.5257222Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-07-24T04:25:14.0728320Z Getting action download info
2025-07-24T04:25:14.2299199Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/tags/ciflow/trunk/149961 (27c8ef1839e0747b36769efb192befc9308ff93c)
2025-07-24T04:25:14.2306170Z ##[group] Inputs
2025-07-24T04:25:14.2306768Z   build-environment: linux-jammy-rocm-py3.10
2025-07-24T04:25:14.2308446Z   test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}
2025-07-24T04:25:14.2310609Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:14.2311799Z   sync-tag: 
2025-07-24T04:25:14.2313110Z   timeout-minutes: 300
2025-07-24T04:25:14.2314411Z   tests-to-include: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-07-24T04:25:14.2315692Z   dashboard-tag: 
2025-07-24T04:25:14.2316584Z   disable-monitor: true
2025-07-24T04:25:14.2317046Z   monitor-log-interval: 5
2025-07-24T04:25:14.2317550Z   monitor-data-collect-interval: 1
2025-07-24T04:25:14.2318041Z ##[endgroup]
2025-07-24T04:25:14.2318667Z Complete job name: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:25:14.4117924Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-07-24T04:25:14.4118502Z with:
2025-07-24T04:25:14.4118659Z   no-sudo: true
2025-07-24T04:25:14.4118853Z   submodules: recursive
2025-07-24T04:25:14.4119061Z   fetch-depth: 0
2025-07-24T04:25:14.4119351Z env:
2025-07-24T04:25:14.4119503Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:14.4119681Z ##[endgroup]
2025-07-24T04:25:14.4190812Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-07-24T04:25:14.4191532Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-07-24T04:25:14.4220310Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:14.4220887Z env:
2025-07-24T04:25:14.4221033Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:14.4221212Z ##[endgroup]
2025-07-24T04:25:14.4555327Z ##[group]Run # Use all available CPUs for fetching
2025-07-24T04:25:14.4556131Z # Use all available CPUs for fetching
2025-07-24T04:25:14.4556712Z cd "${GITHUB_WORKSPACE}"
2025-07-24T04:25:14.4557280Z git config --global fetch.parallel 0
2025-07-24T04:25:14.4557927Z git config --global submodule.fetchJobs 0
2025-07-24T04:25:14.4558482Z 
2025-07-24T04:25:14.4559060Z # Clean workspace. The default checkout action should also do this, but
2025-07-24T04:25:14.4559846Z # do it here as well just in case
2025-07-24T04:25:14.4560369Z if [[ -d .git ]]; then
2025-07-24T04:25:14.4560861Z   if [ -z "${NO_SUDO}" ]; then
2025-07-24T04:25:14.4561375Z     sudo git clean -ffdx
2025-07-24T04:25:14.4561812Z   else
2025-07-24T04:25:14.4562237Z     git clean -ffdx
2025-07-24T04:25:14.4562652Z   fi
2025-07-24T04:25:14.4562995Z fi
2025-07-24T04:25:14.4611788Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:14.4612380Z env:
2025-07-24T04:25:14.4612688Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:14.4613079Z   NO_SUDO: true
2025-07-24T04:25:14.4613506Z ##[endgroup]
2025-07-24T04:25:14.9639398Z Removing .additional_ci_files/
2025-07-24T04:25:14.9639780Z Removing build/
2025-07-24T04:25:14.9639999Z Removing dist/
2025-07-24T04:25:14.9640213Z Removing test/test-reports/
2025-07-24T04:25:14.9739284Z ##[group]Run actions/checkout@v4
2025-07-24T04:25:14.9739781Z with:
2025-07-24T04:25:14.9740202Z   ref: 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:25:14.9740769Z   fetch-depth: 0
2025-07-24T04:25:14.9741167Z   submodules: recursive
2025-07-24T04:25:14.9741594Z   show-progress: false
2025-07-24T04:25:14.9742042Z   repository: pytorch/pytorch
2025-07-24T04:25:14.9742778Z   token: ***
2025-07-24T04:25:14.9743157Z   ssh-strict: true
2025-07-24T04:25:14.9743554Z   ssh-user: git
2025-07-24T04:25:14.9744001Z   persist-credentials: true
2025-07-24T04:25:14.9744531Z   clean: true
2025-07-24T04:25:14.9745045Z   sparse-checkout-cone-mode: true
2025-07-24T04:25:14.9745611Z   fetch-tags: false
2025-07-24T04:25:14.9746058Z   lfs: false
2025-07-24T04:25:14.9746481Z   set-safe-directory: true
2025-07-24T04:25:14.9746976Z env:
2025-07-24T04:25:14.9747341Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:14.9747751Z ##[endgroup]
2025-07-24T04:25:15.1163865Z Syncing repository: pytorch/pytorch
2025-07-24T04:25:15.1166705Z ##[group]Getting Git version info
2025-07-24T04:25:15.1167674Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2025-07-24T04:25:15.1168910Z [command]/usr/bin/git version
2025-07-24T04:25:15.1169383Z git version 2.34.1
2025-07-24T04:25:15.1170786Z ##[endgroup]
2025-07-24T04:25:15.1178593Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/afb835bb-67ea-4378-8c76-02e8bfab1ea1/.gitconfig'
2025-07-24T04:25:15.1181426Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/afb835bb-67ea-4378-8c76-02e8bfab1ea1' before making global git config changes
2025-07-24T04:25:15.1182975Z Adding repository directory to the temporary git global config as a safe directory
2025-07-24T04:25:15.1184167Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-24T04:25:15.1193382Z [command]/usr/bin/git config --local --get remote.origin.url
2025-07-24T04:25:15.1240556Z https://github.com/pytorch/pytorch
2025-07-24T04:25:15.1262673Z ##[group]Removing previously created refs, to avoid conflicts
2025-07-24T04:25:15.1265747Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-07-24T04:25:15.1318167Z HEAD
2025-07-24T04:25:15.1400192Z ##[endgroup]
2025-07-24T04:25:15.1403808Z [command]/usr/bin/git submodule status
2025-07-24T04:25:15.2202391Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-07-24T04:25:15.2428549Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-07-24T04:25:15.2655538Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-07-24T04:25:15.2907913Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-07-24T04:25:15.3037086Z  2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07 third_party/NVTX (v3.1.0-263-g2942f16)
2025-07-24T04:25:15.3238821Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-07-24T04:25:15.4117056Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-07-24T04:25:15.4205291Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-07-24T04:25:15.4273348Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-07-24T04:25:15.4484679Z  df6023e305f389bbf7249b0c4414e649f3ad6598 third_party/composable_kernel (rocm-6.4.1-337-gdf6023e30)
2025-07-24T04:25:15.4780899Z  3af7f2c16147f3fbc6e4d717032daf505dc1652c third_party/cpp-httplib (v0.20.1)
2025-07-24T04:25:15.5096377Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-07-24T04:25:15.5194939Z  f937055efc6d414d11f4c6577e3977fe74f35fb6 third_party/cudnn_frontend (v0.5-52-gf937055)
2025-07-24T04:25:15.5433575Z  b995f933179c22d3fe0d871c3a53d11e4681950f third_party/cutlass (v4.0.0)
2025-07-24T04:25:15.5553009Z  157e88b750c452bef2ab4653fe9d1eeb151ce4c3 third_party/fbgemm (v1.2.0-186-g157e88b7)
2025-07-24T04:25:15.5768250Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-07-24T04:25:15.5836806Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-07-24T04:25:15.6519530Z  40626af88bd7df9a5fb80be7b25ac85b122d6c21 third_party/fmt (11.2.0)
2025-07-24T04:25:15.6806828Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-07-24T04:25:15.7132663Z  c7b7b022c124d9643957d9bd55f57ac59fce8fa2 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-33-gc7b7b02)
2025-07-24T04:25:15.7615833Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-07-24T04:25:15.7824307Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-07-24T04:25:15.7988712Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-07-24T04:25:15.8471404Z  5e7501833f1021ce6f618572d3baf657b6319658 third_party/kineto (heads/main)
2025-07-24T04:25:15.8544895Z  cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7 third_party/kleidiai (v1.8.0)
2025-07-24T04:25:15.8613538Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-07-24T04:25:15.8682358Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-07-24T04:25:15.9364367Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-07-24T04:25:15.9437527Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-07-24T04:25:15.9515805Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-07-24T04:25:16.0173179Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-07-24T04:25:16.0390498Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-07-24T04:25:16.0547436Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-07-24T04:25:16.0677407Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-07-24T04:25:16.0896699Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-07-24T04:25:16.1096535Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-07-24T04:25:16.1307272Z  52791a2fd214b2a9dc5759d36725909c1daa7f2e third_party/tensorpipe (remotes/origin/master)
2025-07-24T04:25:16.1339515Z ##[group]Cleaning the repository
2025-07-24T04:25:16.1348226Z [command]/usr/bin/git clean -ffdx
2025-07-24T04:25:16.1981224Z [command]/usr/bin/git reset --hard HEAD
2025-07-24T04:25:16.5033325Z HEAD is now at 5cdb3d896e1 [FSDP][Replicate] added replicate function that uses FSDP instead of DDP (#158207)
2025-07-24T04:25:16.5093877Z ##[endgroup]
2025-07-24T04:25:16.5096607Z ##[group]Disabling automatic garbage collection
2025-07-24T04:25:16.5107470Z [command]/usr/bin/git config --local gc.auto 0
2025-07-24T04:25:16.5183628Z ##[endgroup]
2025-07-24T04:25:16.5184423Z ##[group]Setting up auth
2025-07-24T04:25:16.5197337Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-07-24T04:25:16.5292943Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-07-24T04:25:16.5983081Z Entering 'android/libs/fbjni'
2025-07-24T04:25:16.6114522Z Entering 'third_party/FP16'
2025-07-24T04:25:16.6237964Z Entering 'third_party/FXdiv'
2025-07-24T04:25:16.6358583Z Entering 'third_party/NNPACK'
2025-07-24T04:25:16.6480824Z Entering 'third_party/NVTX'
2025-07-24T04:25:16.6604218Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:16.6731792Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:16.6883303Z Entering 'third_party/aiter'
2025-07-24T04:25:16.7008297Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:16.7147108Z Entering 'third_party/benchmark'
2025-07-24T04:25:16.7266795Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:16.7405662Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:16.7525922Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:16.7645559Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:16.7771747Z Entering 'third_party/cutlass'
2025-07-24T04:25:16.7909917Z Entering 'third_party/fbgemm'
2025-07-24T04:25:16.8034962Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:16.8149183Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:16.8274775Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:16.8388940Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:16.8520669Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:16.8635039Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:16.8748176Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:16.8871782Z Entering 'third_party/flash-attention'
2025-07-24T04:25:16.8990936Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:16.9123794Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:16.9268955Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:16.9398255Z Entering 'third_party/fmt'
2025-07-24T04:25:16.9521582Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:16.9645143Z Entering 'third_party/gloo'
2025-07-24T04:25:16.9776348Z Entering 'third_party/googletest'
2025-07-24T04:25:16.9898408Z Entering 'third_party/ideep'
2025-07-24T04:25:17.0013028Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:17.0150292Z Entering 'third_party/ittapi'
2025-07-24T04:25:17.0270497Z Entering 'third_party/kineto'
2025-07-24T04:25:17.0386853Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:17.0496185Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:17.0613056Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:17.0727558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:17.0842700Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:17.0949079Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:17.1069642Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:17.1180934Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:17.1291417Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:17.1412246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:17.1535271Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:17.1650819Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:17.1776311Z Entering 'third_party/kleidiai'
2025-07-24T04:25:17.1898182Z Entering 'third_party/mimalloc'
2025-07-24T04:25:17.2017439Z Entering 'third_party/nlohmann'
2025-07-24T04:25:17.2141424Z Entering 'third_party/onnx'
2025-07-24T04:25:17.2293108Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:17.2425823Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:17.2545915Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:17.2657330Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:17.2767148Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:17.2875318Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:17.2988900Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:17.3097564Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:17.3207311Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:17.3313146Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:17.3428230Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:17.3546888Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:17.3711603Z Entering 'third_party/pocketfft'
2025-07-24T04:25:17.3830157Z Entering 'third_party/protobuf'
2025-07-24T04:25:17.3952559Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:17.4062682Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:17.4181306Z Entering 'third_party/psimd'
2025-07-24T04:25:17.4299206Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:17.4418801Z Entering 'third_party/pybind11'
2025-07-24T04:25:17.4536388Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:17.4655694Z Entering 'third_party/sleef'
2025-07-24T04:25:17.4775295Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:17.4896561Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:17.5007260Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:17.5123638Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:17.5234701Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:17.5339843Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:17.5499742Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-07-24T04:25:17.5572116Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-07-24T04:25:17.6249851Z Entering 'android/libs/fbjni'
2025-07-24T04:25:17.6373419Z Entering 'third_party/FP16'
2025-07-24T04:25:17.6496500Z Entering 'third_party/FXdiv'
2025-07-24T04:25:17.6617410Z Entering 'third_party/NNPACK'
2025-07-24T04:25:17.6738495Z Entering 'third_party/NVTX'
2025-07-24T04:25:17.6859003Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:17.6979468Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:17.7129660Z Entering 'third_party/aiter'
2025-07-24T04:25:17.7246156Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:17.7381221Z Entering 'third_party/benchmark'
2025-07-24T04:25:17.7500682Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:17.7636727Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:17.7755964Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:17.7874803Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:17.7993184Z Entering 'third_party/cutlass'
2025-07-24T04:25:17.8135385Z Entering 'third_party/fbgemm'
2025-07-24T04:25:17.8254852Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:17.8366896Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:17.8494764Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:17.8611553Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:17.8742216Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:17.8853121Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:17.8960830Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:17.9081424Z Entering 'third_party/flash-attention'
2025-07-24T04:25:17.9197414Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:17.9322010Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:17.9456718Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:17.9580870Z Entering 'third_party/fmt'
2025-07-24T04:25:17.9699061Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:17.9818519Z Entering 'third_party/gloo'
2025-07-24T04:25:17.9938027Z Entering 'third_party/googletest'
2025-07-24T04:25:18.0056013Z Entering 'third_party/ideep'
2025-07-24T04:25:18.0169816Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:18.0299404Z Entering 'third_party/ittapi'
2025-07-24T04:25:18.0417658Z Entering 'third_party/kineto'
2025-07-24T04:25:18.0531775Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:18.0638035Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:18.0752468Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:18.0865291Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:18.0975427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:18.1080255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:18.1197794Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:18.1307939Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:18.1417960Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:18.1529601Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:18.1645726Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:18.1755347Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:18.1873310Z Entering 'third_party/kleidiai'
2025-07-24T04:25:18.1993863Z Entering 'third_party/mimalloc'
2025-07-24T04:25:18.2115082Z Entering 'third_party/nlohmann'
2025-07-24T04:25:18.2237040Z Entering 'third_party/onnx'
2025-07-24T04:25:18.2386895Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:18.2511194Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:18.2629619Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:18.2740253Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:18.2851186Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:18.2958952Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:18.3070345Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:18.3177276Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:18.3287009Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:18.3390660Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:18.3504859Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:18.3622808Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:18.3778588Z Entering 'third_party/pocketfft'
2025-07-24T04:25:18.3898189Z Entering 'third_party/protobuf'
2025-07-24T04:25:18.4019999Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:18.4129318Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:18.4247258Z Entering 'third_party/psimd'
2025-07-24T04:25:18.4367203Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:18.4485069Z Entering 'third_party/pybind11'
2025-07-24T04:25:18.4603650Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:18.4721293Z Entering 'third_party/sleef'
2025-07-24T04:25:18.4839728Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:18.4954997Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:18.5066193Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:18.5174592Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:18.5282657Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:18.5386784Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:18.5553342Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-07-24T04:25:18.5645075Z ##[endgroup]
2025-07-24T04:25:18.5645835Z ##[group]Fetching the repository
2025-07-24T04:25:18.5660190Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-07-24T04:25:19.5489402Z From https://github.com/pytorch/pytorch
2025-07-24T04:25:19.5489921Z  - [deleted]                 (none)     -> origin/gh/IvanKobzarev/117/base
2025-07-24T04:25:19.6314532Z  - [deleted]                 (none)     -> origin/gh/IvanKobzarev/117/head
2025-07-24T04:25:19.6315006Z  - [deleted]                 (none)     -> origin/gh/IvanKobzarev/117/orig
2025-07-24T04:25:19.6315625Z  - [deleted]                 (none)     -> origin/gh/Sidharth123-cpu/37/base
2025-07-24T04:25:19.6317942Z  - [deleted]                 (none)     -> origin/gh/Sidharth123-cpu/37/head
2025-07-24T04:25:19.6319665Z  - [deleted]                 (none)     -> origin/gh/Sidharth123-cpu/37/orig
2025-07-24T04:25:19.6321188Z  - [deleted]                 (none)     -> origin/gh/StrongerXi/126/base
2025-07-24T04:25:19.6321949Z  - [deleted]                 (none)     -> origin/gh/StrongerXi/126/head
2025-07-24T04:25:19.6324332Z  - [deleted]                 (none)     -> origin/gh/StrongerXi/126/orig
2025-07-24T04:25:19.6325314Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/313/base
2025-07-24T04:25:19.6327149Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/313/head
2025-07-24T04:25:19.6328545Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/313/orig
2025-07-24T04:25:19.6330030Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/314/base
2025-07-24T04:25:19.6332007Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/314/head
2025-07-24T04:25:19.6332975Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/314/orig
2025-07-24T04:25:19.6334557Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/315/base
2025-07-24T04:25:19.6336431Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/315/head
2025-07-24T04:25:19.6337441Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/315/orig
2025-07-24T04:25:19.6339012Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/316/base
2025-07-24T04:25:19.6341199Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/316/head
2025-07-24T04:25:19.6342072Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/316/orig
2025-07-24T04:25:19.6343746Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/317/base
2025-07-24T04:25:19.6345237Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/317/head
2025-07-24T04:25:19.6346659Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/317/orig
2025-07-24T04:25:19.6348212Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/319/base
2025-07-24T04:25:19.6349630Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/319/head
2025-07-24T04:25:19.6351133Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/319/orig
2025-07-24T04:25:19.6352624Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/321/base
2025-07-24T04:25:19.6354205Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/321/head
2025-07-24T04:25:19.6355611Z  - [deleted]                 (none)     -> origin/gh/XuehaiPan/321/orig
2025-07-24T04:25:19.6357136Z  - [deleted]                 (none)     -> origin/gh/andyanwang/24/base
2025-07-24T04:25:19.6358648Z  - [deleted]                 (none)     -> origin/gh/andyanwang/24/head
2025-07-24T04:25:19.6360361Z  - [deleted]                 (none)     -> origin/gh/andyanwang/24/orig
2025-07-24T04:25:19.6361655Z  - [deleted]                 (none)     -> origin/gh/andyanwang/25/base
2025-07-24T04:25:19.6363185Z  - [deleted]                 (none)     -> origin/gh/andyanwang/25/head
2025-07-24T04:25:19.6364858Z  - [deleted]                 (none)     -> origin/gh/andyanwang/25/orig
2025-07-24T04:25:19.6366297Z  - [deleted]                 (none)     -> origin/gh/andyanwang/26/base
2025-07-24T04:25:19.6367735Z  - [deleted]                 (none)     -> origin/gh/andyanwang/26/head
2025-07-24T04:25:19.6369246Z  - [deleted]                 (none)     -> origin/gh/andyanwang/26/orig
2025-07-24T04:25:19.6370735Z  - [deleted]                 (none)     -> origin/gh/andyanwang/27/base
2025-07-24T04:25:19.6372241Z  - [deleted]                 (none)     -> origin/gh/andyanwang/27/head
2025-07-24T04:25:19.6373798Z  - [deleted]                 (none)     -> origin/gh/andyanwang/27/orig
2025-07-24T04:25:19.6375278Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/468/base
2025-07-24T04:25:19.6376790Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/468/head
2025-07-24T04:25:19.6378295Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/468/orig
2025-07-24T04:25:19.6379826Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/476/base
2025-07-24T04:25:19.6381351Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/476/head
2025-07-24T04:25:19.6382817Z  - [deleted]                 (none)     -> origin/gh/bobrenjc93/476/orig
2025-07-24T04:25:19.6384360Z  - [deleted]                 (none)     -> origin/gh/ezyang/3081/base
2025-07-24T04:25:19.6385848Z  - [deleted]                 (none)     -> origin/gh/ezyang/3081/head
2025-07-24T04:25:19.6387374Z  - [deleted]                 (none)     -> origin/gh/ezyang/3081/orig
2025-07-24T04:25:19.6388894Z  - [deleted]                 (none)     -> origin/gh/henrylhtsang/99/base
2025-07-24T04:25:19.6390399Z  - [deleted]                 (none)     -> origin/gh/henrylhtsang/99/head
2025-07-24T04:25:19.6391908Z  - [deleted]                 (none)     -> origin/gh/henrylhtsang/99/orig
2025-07-24T04:25:19.6393430Z  - [deleted]                 (none)     -> origin/gh/isuruf/145/base
2025-07-24T04:25:19.6395203Z  - [deleted]                 (none)     -> origin/gh/isuruf/145/head
2025-07-24T04:25:19.6396656Z  - [deleted]                 (none)     -> origin/gh/isuruf/145/orig
2025-07-24T04:25:19.6397962Z  - [deleted]                 (none)     -> origin/gh/isuruf/147/base
2025-07-24T04:25:19.6399454Z  - [deleted]                 (none)     -> origin/gh/isuruf/147/head
2025-07-24T04:25:19.6400956Z  - [deleted]                 (none)     -> origin/gh/isuruf/147/orig
2025-07-24T04:25:19.6402460Z  - [deleted]                 (none)     -> origin/gh/kwen2501/177/base
2025-07-24T04:25:19.6404002Z  - [deleted]                 (none)     -> origin/gh/kwen2501/177/head
2025-07-24T04:25:19.6408166Z  - [deleted]                 (none)     -> origin/gh/kwen2501/177/orig
2025-07-24T04:25:19.6409640Z  - [deleted]                 (none)     -> origin/gh/malfet/415/base
2025-07-24T04:25:19.6411142Z  - [deleted]                 (none)     -> origin/gh/malfet/415/head
2025-07-24T04:25:19.6412708Z  - [deleted]                 (none)     -> origin/gh/malfet/415/orig
2025-07-24T04:25:19.6414156Z  - [deleted]                 (none)     -> origin/gh/malfet/416/base
2025-07-24T04:25:19.6415664Z  - [deleted]                 (none)     -> origin/gh/malfet/416/head
2025-07-24T04:25:19.6417132Z  - [deleted]                 (none)     -> origin/gh/malfet/416/orig
2025-07-24T04:25:19.6418657Z  - [deleted]                 (none)     -> origin/gh/manuelcandales/8/base
2025-07-24T04:25:19.6420156Z  - [deleted]                 (none)     -> origin/gh/manuelcandales/8/head
2025-07-24T04:25:19.6421657Z  - [deleted]                 (none)     -> origin/gh/manuelcandales/8/orig
2025-07-24T04:25:19.6423185Z  - [deleted]                 (none)     -> origin/gh/oulgen/38/base
2025-07-24T04:25:19.6424690Z  - [deleted]                 (none)     -> origin/gh/oulgen/38/head
2025-07-24T04:25:19.6426215Z  - [deleted]                 (none)     -> origin/gh/oulgen/38/orig
2025-07-24T04:25:19.6427726Z  - [deleted]                 (none)     -> origin/gh/rec/137/base
2025-07-24T04:25:19.6429212Z  - [deleted]                 (none)     -> origin/gh/rec/137/head
2025-07-24T04:25:19.6430725Z  - [deleted]                 (none)     -> origin/gh/rec/137/orig
2025-07-24T04:25:19.6432200Z  - [deleted]                 (none)     -> origin/gh/rec/146/base
2025-07-24T04:25:19.6433700Z  - [deleted]                 (none)     -> origin/gh/rec/146/head
2025-07-24T04:25:19.6435214Z  - [deleted]                 (none)     -> origin/gh/rec/146/orig
2025-07-24T04:25:19.6436766Z  - [deleted]                 (none)     -> origin/gh/swolchok/780/base
2025-07-24T04:25:19.6438244Z  - [deleted]                 (none)     -> origin/gh/swolchok/780/head
2025-07-24T04:25:19.6439759Z  - [deleted]                 (none)     -> origin/gh/swolchok/780/orig
2025-07-24T04:25:19.6441254Z  - [deleted]                 (none)     -> origin/gh/xmfan/243/base
2025-07-24T04:25:19.6442751Z  - [deleted]                 (none)     -> origin/gh/xmfan/243/head
2025-07-24T04:25:19.6444271Z  - [deleted]                 (none)     -> origin/gh/xmfan/243/orig
2025-07-24T04:25:19.6445939Z  - [deleted]                 (none)     -> origin/gh/zhxchen17/22/base
2025-07-24T04:25:19.6447409Z  - [deleted]                 (none)     -> origin/gh/zhxchen17/22/head
2025-07-24T04:25:19.6448881Z  - [deleted]                 (none)     -> origin/gh/zhxchen17/22/orig
2025-07-24T04:25:19.6450397Z  - [deleted]                 (none)     -> origin/malfet-patch-5
2025-07-24T04:25:19.6451930Z  - [deleted]                 (none)     -> origin/malfet-patch-6
2025-07-24T04:25:19.6453421Z  - [deleted]                 (none)     -> origin/mlazos/buf-fix
2025-07-24T04:25:19.6454966Z  - [deleted]                 (none)     -> ciflow/binaries_wheel/158408
2025-07-24T04:25:19.6456431Z  - [deleted]                 (none)     -> ciflow/inductor/143712
2025-07-24T04:25:19.6457936Z  - [deleted]                 (none)     -> ciflow/inductor/157580
2025-07-24T04:25:19.6459446Z  - [deleted]                 (none)     -> ciflow/inductor/158391
2025-07-24T04:25:19.6460927Z  - [deleted]                 (none)     -> ciflow/inductor/158426
2025-07-24T04:25:19.6462605Z  - [deleted]                 (none)     -> ciflow/inductor/158434
2025-07-24T04:25:19.6463932Z  - [deleted]                 (none)     -> ciflow/inductor/158442
2025-07-24T04:25:19.6465477Z  - [deleted]                 (none)     -> ciflow/inductor/158468
2025-07-24T04:25:19.6466999Z  - [deleted]                 (none)     -> ciflow/inductor/158476
2025-07-24T04:25:19.6468505Z  - [deleted]                 (none)     -> ciflow/inductor/158498
2025-07-24T04:25:19.6470014Z  - [deleted]                 (none)     -> ciflow/inductor/158499
2025-07-24T04:25:19.6471518Z  - [deleted]                 (none)     -> ciflow/inductor/158500
2025-07-24T04:25:19.6472999Z  - [deleted]                 (none)     -> ciflow/inductor/158859
2025-07-24T04:25:19.6474486Z  - [deleted]                 (none)     -> ciflow/inductor/158997
2025-07-24T04:25:19.6476039Z  - [deleted]                 (none)     -> ciflow/rocm/158408
2025-07-24T04:25:19.6477526Z  - [deleted]                 (none)     -> ciflow/triton_binaries/158408
2025-07-24T04:25:19.6479191Z  - [deleted]                 (none)     -> ciflow/trunk/143712
2025-07-24T04:25:19.6480588Z  - [deleted]                 (none)     -> ciflow/trunk/156559
2025-07-24T04:25:19.6482051Z  - [deleted]                 (none)     -> ciflow/trunk/156666
2025-07-24T04:25:19.6483534Z  - [deleted]                 (none)     -> ciflow/trunk/157580
2025-07-24T04:25:19.6485149Z  - [deleted]                 (none)     -> ciflow/trunk/158185
2025-07-24T04:25:19.6486704Z  - [deleted]                 (none)     -> ciflow/trunk/158341
2025-07-24T04:25:19.6488174Z  - [deleted]                 (none)     -> ciflow/trunk/158442
2025-07-24T04:25:19.6489645Z  - [deleted]                 (none)     -> ciflow/trunk/158964
2025-07-24T04:25:19.6491152Z  - [deleted]                 (none)     -> ciflow/trunk/158997
2025-07-24T04:25:19.6492737Z  - [deleted]                 (none)     -> trunk/06a67a8948dac9d02f22b3e2591a43b60981cdb4
2025-07-24T04:25:19.6494245Z  - [deleted]                 (none)     -> trunk/0a99b026d6bd0f67dc2c0a20fe3228ddc4144854
2025-07-24T04:25:19.6495768Z  - [deleted]                 (none)     -> trunk/0a9d450168ce58b2bb7f2cedc27a61012123564f
2025-07-24T04:25:19.6497223Z  - [deleted]                 (none)     -> trunk/0b19d463d963a0b2ee5558d2c0bb79b2cbff6e64
2025-07-24T04:25:19.6498747Z  - [deleted]                 (none)     -> trunk/14ecc0336185f2ca5591858bc74cd4aadf2d1161
2025-07-24T04:25:19.6500300Z  - [deleted]                 (none)     -> trunk/1cc62c2cb91e56ae50494f88f369cd6ec466a118
2025-07-24T04:25:19.6501796Z  - [deleted]                 (none)     -> trunk/1d584761622ff6e5519c5e3dbbb62a21b89ffe8a
2025-07-24T04:25:19.6503298Z  - [deleted]                 (none)     -> trunk/2179afd7149c117dace9e552419082094b10a386
2025-07-24T04:25:19.6504778Z  - [deleted]                 (none)     -> trunk/2ad5c25cfc603c3656e6699d6137419dbb009495
2025-07-24T04:25:19.6506330Z  - [deleted]                 (none)     -> trunk/2b0f9b1f6172a0d5817c7ac7406200897311da5f
2025-07-24T04:25:19.6507852Z  - [deleted]                 (none)     -> trunk/306dd19216b656467143483395ef582feb5d7d07
2025-07-24T04:25:19.6509352Z  - [deleted]                 (none)     -> trunk/382598ef872b2afb9a03f8d88277a6c2edeb507f
2025-07-24T04:25:19.6510832Z  - [deleted]                 (none)     -> trunk/382c6190c1329e96e71eef21a19737a3eda0040b
2025-07-24T04:25:19.6512327Z  - [deleted]                 (none)     -> trunk/38371f693b07a485705119407da2e5dc64cec4eb
2025-07-24T04:25:19.6513856Z  - [deleted]                 (none)     -> trunk/38410cf9b57079f3360c1e79601973a01cb2588c
2025-07-24T04:25:19.6515381Z  - [deleted]                 (none)     -> trunk/3863bbb55b38985c7d64c8a0be7beb2005a9cc07
2025-07-24T04:25:19.6516900Z  - [deleted]                 (none)     -> trunk/386aa7200324519ea9a8eff5eb1b3c0517756d24
2025-07-24T04:25:19.6518362Z  - [deleted]                 (none)     -> trunk/386bc9e2e990a6ac29bb90ba97b71c5b85e11080
2025-07-24T04:25:19.6519929Z  - [deleted]                 (none)     -> trunk/38757d94f1b3d65295b5ca2d7527ea0d582a5d3b
2025-07-24T04:25:19.6521665Z  - [deleted]                 (none)     -> trunk/38bfd462b8fb035de08a8c0d2b7b13eba78ee870
2025-07-24T04:25:19.6522938Z  - [deleted]                 (none)     -> trunk/38c4d05535f4e031a2ad5f39a4a415cc3a919cfb
2025-07-24T04:25:19.6524482Z  - [deleted]                 (none)     -> trunk/38e1e5d54ce42d42f8920ff46d9102800b94482d
2025-07-24T04:25:19.6527429Z  - [deleted]                 (none)     -> trunk/38e5e81e55fc5d85d6cf8a83c96c88578995e3fe
2025-07-24T04:25:19.6529022Z  - [deleted]                 (none)     -> trunk/391473cca0b0fa9667ad54c972e7b63bc589cd6d
2025-07-24T04:25:19.6530463Z  - [deleted]                 (none)     -> trunk/39270430c9e4dc7010a9754f522774d17ae9d578
2025-07-24T04:25:19.6532069Z  - [deleted]                 (none)     -> trunk/39456edbbad97a09e6507792e1df3ee1f2a16f98
2025-07-24T04:25:19.6533598Z  - [deleted]                 (none)     -> trunk/398fca9dcfcb9f90eae9258d728fab71f5b59db2
2025-07-24T04:25:19.6535020Z  - [deleted]                 (none)     -> trunk/39a8f66d5939e892bcb07ef97462af47d3201491
2025-07-24T04:25:19.6536504Z  - [deleted]                 (none)     -> trunk/39b71d11fc2dd9b4da6d23a34eb29aefbb1df672
2025-07-24T04:25:19.6538019Z  - [deleted]                 (none)     -> trunk/39c605e8b3c13e7f6b5b1bbbe3f4060e24b2a3e4
2025-07-24T04:25:19.6539514Z  - [deleted]                 (none)     -> trunk/3a43dba21ff5d01bb8b259af3839c90e447c6ec0
2025-07-24T04:25:19.6541051Z  - [deleted]                 (none)     -> trunk/3a5677a380c79810cc370dce6ef79b0871ddcf8c
2025-07-24T04:25:19.6542514Z  - [deleted]                 (none)     -> trunk/3a7ff829c597ef26a271d52ae2a3c8354b2f447e
2025-07-24T04:25:19.6544032Z  - [deleted]                 (none)     -> trunk/3b4b5f8d474a9c664f622012c6a61414ea7799a9
2025-07-24T04:25:19.6545544Z  - [deleted]                 (none)     -> trunk/3b6569b1ef4b9ff25f5b75fe0a216d6d084d573f
2025-07-24T04:25:19.6547088Z  - [deleted]                 (none)     -> trunk/3b7c5e6fa5c0be64ddde3a8edc5bcdc10390f1e3
2025-07-24T04:25:19.6548576Z  - [deleted]                 (none)     -> trunk/3bc6bdc8660c052d932f550d5734da6f801c2630
2025-07-24T04:25:19.6550166Z  - [deleted]                 (none)     -> trunk/3bdd5ae334b85a114a2b62fa17dcf204413eda32
2025-07-24T04:25:19.6551589Z  - [deleted]                 (none)     -> trunk/3beb915004f4e26b1e7c5e7692e6e8ca9b75de46
2025-07-24T04:25:19.6553102Z  - [deleted]                 (none)     -> trunk/3bec588bf5c4eda9a4d42ae4c25e6f87af7f078c
2025-07-24T04:25:19.6554614Z  - [deleted]                 (none)     -> trunk/3c2324c64ac6b7497d630788a66c82705bbb044e
2025-07-24T04:25:19.6556130Z  - [deleted]                 (none)     -> trunk/3c7079959c8e61d7acb4f704a0ecf74c61425c2e
2025-07-24T04:25:19.6557624Z  - [deleted]                 (none)     -> trunk/3c72b9fd8feed4588a040bc681ffe83cc7acd26d
2025-07-24T04:25:19.6559101Z  - [deleted]                 (none)     -> trunk/3cb11877aa30c04be7ffa9b4ca1722f1270a5828
2025-07-24T04:25:19.6560680Z  - [deleted]                 (none)     -> trunk/3cbae6dde8f2a9c3b4a3e4e079e97307c3aa52d8
2025-07-24T04:25:19.6562304Z  - [deleted]                 (none)     -> trunk/3d06ff82a84a118f0ed246864d4fc01ac4726328
2025-07-24T04:25:19.6563688Z  - [deleted]                 (none)     -> trunk/3d595fd5595f38bb5ed3d390dc50e1715e173ad6
2025-07-24T04:25:19.6565284Z  - [deleted]                 (none)     -> trunk/3d82a1dfb59fa5e248f7499a7ecdf784d4f61c0e
2025-07-24T04:25:19.6566725Z  - [deleted]                 (none)     -> trunk/3dabc351bb5581f69825eee6b24fbac9f9260241
2025-07-24T04:25:19.6568273Z  - [deleted]                 (none)     -> trunk/3dd872e6d53560933d8d7fc11357617746d37168
2025-07-24T04:25:19.6569761Z  - [deleted]                 (none)     -> trunk/3dda80e990121eaf156014fffe6e2a4602c8b195
2025-07-24T04:25:19.6571345Z  - [deleted]                 (none)     -> trunk/3df6360e8c956edd25453b21f6b7f56e0366dcb4
2025-07-24T04:25:19.6572791Z  - [deleted]                 (none)     -> trunk/3e131f7779af03b462f0598afb2569092b57c840
2025-07-24T04:25:19.6574485Z  - [deleted]                 (none)     -> trunk/3e38feb05fffdf5b181a1f4c7a6f43b00ef1c526
2025-07-24T04:25:19.6575812Z  - [deleted]                 (none)     -> trunk/3e56a9cdfb98a9b06568ee54e3157c800d98a17e
2025-07-24T04:25:19.6577313Z  - [deleted]                 (none)     -> trunk/3eb7084f7a3657f4ba6626aca63721a0020f1bd7
2025-07-24T04:25:19.6578884Z  - [deleted]                 (none)     -> trunk/3ed4384f5b4bb7ae7d12298632a258385a51446e
2025-07-24T04:25:19.6580360Z  - [deleted]                 (none)     -> trunk/3ee75b7eacef6758db602e87287ef9574609b327
2025-07-24T04:25:19.6581846Z  - [deleted]                 (none)     -> trunk/3ee8828c87ce6186607c3d3ab3852518fca49228
2025-07-24T04:25:19.6583366Z  - [deleted]                 (none)     -> trunk/3efb22e09111b92bedb01b2a8385c789fe69090a
2025-07-24T04:25:19.6584841Z  - [deleted]                 (none)     -> trunk/3effe0c293219b00a0eae7e139fe2d9aed84bc03
2025-07-24T04:25:19.6586388Z  - [deleted]                 (none)     -> trunk/3f29642ecf039129032cc61c986d7b62807163c0
2025-07-24T04:25:19.6587854Z  - [deleted]                 (none)     -> trunk/3f569f9af77d51d0328f16434e64252756681daa
2025-07-24T04:25:19.6589470Z  - [deleted]                 (none)     -> trunk/3f65e38b73cb8f0d0b4fb2e9a6671085d371ec40
2025-07-24T04:25:19.6590904Z  - [deleted]                 (none)     -> trunk/3f69e3b3a07efd5e9aa86d37773bf2a24dc0ea70
2025-07-24T04:25:19.6592404Z  - [deleted]                 (none)     -> trunk/3f83e3eeca0645f4b2cd16fa7d5a591e9cf810d4
2025-07-24T04:25:19.6593966Z  - [deleted]                 (none)     -> trunk/3f920f3d8f5bd15d2222758f21f9a5d36e4dad1f
2025-07-24T04:25:19.6595423Z  - [deleted]                 (none)     -> trunk/3fd84a8592a2a87d04f0e3f236a92605e2be12df
2025-07-24T04:25:19.6596946Z  - [deleted]                 (none)     -> trunk/400f439670ff8859c17a870eb83dc5cae5a9c2e4
2025-07-24T04:25:19.6598567Z  - [deleted]                 (none)     -> trunk/402ae09e41005ebff686e97e2f120a6b79e2afb7
2025-07-24T04:25:19.6600168Z  - [deleted]                 (none)     -> trunk/404008e3efdabeaf5b140a3aff77131461c33a0a
2025-07-24T04:25:19.6601660Z  - [deleted]                 (none)     -> trunk/4048a144abf82041f9b6653d9d4aa600a6a5409a
2025-07-24T04:25:19.6603091Z  - [deleted]                 (none)     -> trunk/408d9884b07cf7268961bae7138a6436916d4a43
2025-07-24T04:25:19.6604740Z  - [deleted]                 (none)     -> trunk/40a785103cf94a1dbc3e0e43d1ed6c41fb60bedb
2025-07-24T04:25:19.6607375Z  - [deleted]                 (none)     -> trunk/40d02eb481670325ba80dcb42b40c3ebb347f1c7
2025-07-24T04:25:19.6608858Z  - [deleted]                 (none)     -> trunk/40e39ae21f15f200828ab32826bed1a4b62532ba
2025-07-24T04:25:19.6610343Z  - [deleted]                 (none)     -> trunk/40fefe2871a5561b67ec268bab19db60e733f2d4
2025-07-24T04:25:19.6611910Z  - [deleted]                 (none)     -> trunk/414ad470450c654d97e73bef704a7b596b5b4cbc
2025-07-24T04:25:19.6613386Z  - [deleted]                 (none)     -> trunk/4162c0f70297818abd70f2fe7424dcd3cc9b2543
2025-07-24T04:25:19.6614879Z  - [deleted]                 (none)     -> trunk/41910d7a94d2f3ffe06f7c0c30971c71b9c3b09e
2025-07-24T04:25:19.6616385Z  - [deleted]                 (none)     -> trunk/41971335c98b0881e0784085096eceace575d563
2025-07-24T04:25:19.6617795Z  - [deleted]                 (none)     -> trunk/41e8b826d07bbf707bc2c64a78b4dc15e8f3c358
2025-07-24T04:25:19.6619682Z  - [deleted]                 (none)     -> trunk/41f6acef83d280a18909d58f8442e145c9e7ea6f
2025-07-24T04:25:19.6621198Z  - [deleted]                 (none)     -> trunk/42015db6a9602f40250f9afc18a3fbad4ca4ef39
2025-07-24T04:25:19.6622912Z  - [deleted]                 (none)     -> trunk/4237ee3c33b0edb01db96d8c1ecf8f5d2cf184bc
2025-07-24T04:25:19.6624383Z  - [deleted]                 (none)     -> trunk/4283d96bcdf9aaa4289985267186d74ba6534ee5
2025-07-24T04:25:19.6625850Z  - [deleted]                 (none)     -> trunk/42b48ee67229286127390000f103a11dfc8901f5
2025-07-24T04:25:19.6627374Z  - [deleted]                 (none)     -> trunk/42ff6a4a5c4e0d77bd18fcc5426622f1b8f20add
2025-07-24T04:25:19.6629051Z  - [deleted]                 (none)     -> trunk/430cc1c636380a7d50652df646274f7008a76747
2025-07-24T04:25:19.6630372Z  - [deleted]                 (none)     -> trunk/4311aea5e7568c9956a9f0d694f1980c387b6a37
2025-07-24T04:25:19.6631861Z  - [deleted]                 (none)     -> trunk/433a2471023f77a4e62e880ef59f22caf939c227
2025-07-24T04:25:19.6633379Z  - [deleted]                 (none)     -> trunk/43523bf1682c2926a84a1f65f00fabd3d34db4f2
2025-07-24T04:25:19.6634898Z  - [deleted]                 (none)     -> trunk/43a09189c68fe02bd9d8433c4a144ffc9bbf895c
2025-07-24T04:25:19.6636397Z  - [deleted]                 (none)     -> trunk/43f72163275ce1503b1de1480642c61d20e6158d
2025-07-24T04:25:19.6637936Z  - [deleted]                 (none)     -> trunk/442aca44d603ae6c2b7d2aa2190cc91f970c4202
2025-07-24T04:25:19.6639415Z  - [deleted]                 (none)     -> trunk/44303caabfa0bdc5688d693d491ba7c3fdb40f3a
2025-07-24T04:25:19.6641022Z  - [deleted]                 (none)     -> trunk/443b5b43c3d40b89bbb5db70f3f05a44b2a4ec66
2025-07-24T04:25:19.6642439Z  - [deleted]                 (none)     -> trunk/4486a6dbfd65ef490cfe73e0630929e85f61ee16
2025-07-24T04:25:19.6644005Z  - [deleted]                 (none)     -> trunk/4491326fb0c0e67eca1598ae33c41cdfced2cd33
2025-07-24T04:25:19.6645740Z  - [deleted]                 (none)     -> trunk/44a5f93462bd4aa4f36ae2573685ed7ba458c4c9
2025-07-24T04:25:19.6647090Z  - [deleted]                 (none)     -> trunk/44d0800d60e78fef8ab332e307c3134e3c276ba4
2025-07-24T04:25:19.6648596Z  - [deleted]                 (none)     -> trunk/44df7cf28dd37a4d900eb6b2f78dabf72b209d9b
2025-07-24T04:25:19.6650017Z  - [deleted]                 (none)     -> trunk/44f5b9312290866584088e2e42228c484f669faf
2025-07-24T04:25:19.6651535Z  - [deleted]                 (none)     -> trunk/4500a4aa50141ed30e093ef8491b30d1d1287348
2025-07-24T04:25:19.6653078Z  - [deleted]                 (none)     -> trunk/451b525bf0cb8840adbde1b6280c7b199f733ced
2025-07-24T04:25:19.6654622Z  - [deleted]                 (none)     -> trunk/45382b284d03015f3e6eb83a959f0e54a3ebd688
2025-07-24T04:25:19.6656081Z  - [deleted]                 (none)     -> trunk/453bc9fbdfb97fef925477cbbf4948e93fd22756
2025-07-24T04:25:19.6657601Z  - [deleted]                 (none)     -> trunk/45596ec58f5bc5489911ee932c2a55a7ff40d0a0
2025-07-24T04:25:19.6659114Z  - [deleted]                 (none)     -> trunk/455dfd258980294f0745bd90aee12a323e37224d
2025-07-24T04:25:19.6660607Z  - [deleted]                 (none)     -> trunk/456b7451c78096fd512c60e26cc618386a251bc7
2025-07-24T04:25:19.6662147Z  - [deleted]                 (none)     -> trunk/456f40cb09c6e5e0cd268d1601b89fc5ac1987ea
2025-07-24T04:25:19.6663580Z  - [deleted]                 (none)     -> trunk/4574b39aa45f0250cba04aa3cb053a686c4888b2
2025-07-24T04:25:19.6665114Z  - [deleted]                 (none)     -> trunk/457dd79927db6ae7f1c53f36768b7ac93b61e507
2025-07-24T04:25:19.6666657Z  - [deleted]                 (none)     -> trunk/4585c33e74079af8e9067bf39970b93c3f13629f
2025-07-24T04:25:19.6668132Z  - [deleted]                 (none)     -> trunk/458cc7213baf01ab742a2250c7a31f7796fcb1ef
2025-07-24T04:25:19.6669667Z  - [deleted]                 (none)     -> trunk/45c5a232373cac1b1524259f20758bfb56dde5f2
2025-07-24T04:25:19.6671212Z  - [deleted]                 (none)     -> trunk/4609699bfd440d1fe603c5ad9a942e45410094c1
2025-07-24T04:25:19.6672757Z  - [deleted]                 (none)     -> trunk/4628f1b7a9313df98346d582d9a789661a823e27
2025-07-24T04:25:19.6674212Z  - [deleted]                 (none)     -> trunk/463fe36532bc165b46c12cef4948f039434e0924
2025-07-24T04:25:19.6675695Z  - [deleted]                 (none)     -> trunk/4657a84bc55b6ce12f21706de2b90e1d43784f57
2025-07-24T04:25:19.6677196Z  - [deleted]                 (none)     -> trunk/46915b13614dbac90724d0f1802b8e0db037c9e4
2025-07-24T04:25:19.6678680Z  - [deleted]                 (none)     -> trunk/473208cb18d543e8f968918a6b3c9defa8a4ae10
2025-07-24T04:25:19.6680407Z  - [deleted]                 (none)     -> trunk/476874b37fff42a46d25dfac720ef4c71ec74fe0
2025-07-24T04:25:19.6681733Z  - [deleted]                 (none)     -> trunk/4781b0ee6057fdf2a82d2f9ed30299f219267a71
2025-07-24T04:25:19.6683320Z  - [deleted]                 (none)     -> trunk/4781d72faa6b72bf96fde9bedfca06a5285eebfb
2025-07-24T04:25:19.6684843Z  - [deleted]                 (none)     -> trunk/47c8810b5275179833d6b33ca3d70922f485272c
2025-07-24T04:25:19.6686341Z  - [deleted]                 (none)     -> trunk/47f10d0ad0dda281c886ff08ac2f938207027316
2025-07-24T04:25:19.6687868Z  - [deleted]                 (none)     -> trunk/4805a6ead6f1e7f32351056e2602be4e908f69b7
2025-07-24T04:25:19.6689311Z  - [deleted]                 (none)     -> trunk/48315181c75e43cab5957197d42e053d66b3fe1c
2025-07-24T04:25:19.6690883Z  - [deleted]                 (none)     -> trunk/4851863e3f98e1ffba1d5801f3533031ef4f905d
2025-07-24T04:25:19.6692348Z  - [deleted]                 (none)     -> trunk/48560eef80e97e855cbb8e2814acefe8f5cc6fbd
2025-07-24T04:25:19.6693998Z  - [deleted]                 (none)     -> trunk/4886ba64dceb24b8a0444e574abb907a18fdeee9
2025-07-24T04:25:19.6695342Z  - [deleted]                 (none)     -> trunk/48921721d8ed32a2c7f80605b434f9fcb2f948b4
2025-07-24T04:25:19.6696839Z  - [deleted]                 (none)     -> trunk/48de3da2539cecaee14af8e3841c133c9c0c0f1c
2025-07-24T04:25:19.6698430Z  - [deleted]                 (none)     -> trunk/48e7b62d3abda4a3eac2f538c9f35e56093c4ece
2025-07-24T04:25:19.6699892Z  - [deleted]                 (none)     -> trunk/4918502d2e685270cea9f1fd733a3414ac1ca5d7
2025-07-24T04:25:19.6701437Z  - [deleted]                 (none)     -> trunk/493bd625e252dea02e871346beaa49745b4b2663
2025-07-24T04:25:19.6702897Z  - [deleted]                 (none)     -> trunk/493f42a5417a8ac33d1070478d1869ea4f207b5b
2025-07-24T04:25:19.6704421Z  - [deleted]                 (none)     -> trunk/495c317005ad656dee752b6ad7bd5541b04e388e
2025-07-24T04:25:19.6705951Z  - [deleted]                 (none)     -> trunk/49888e6be0dcc00ba546746425893bb17c287248
2025-07-24T04:25:19.6707418Z  - [deleted]                 (none)     -> trunk/49ee1e7106db00778345efd7c70cc12d2ca6a91a
2025-07-24T04:25:19.6708986Z  - [deleted]                 (none)     -> trunk/4a26bb8a12ba5cb437ed5b7f035b5b533e07549e
2025-07-24T04:25:19.6710456Z  - [deleted]                 (none)     -> trunk/4a4cac0cefea3661cc69cfdafdba64832ee0841a
2025-07-24T04:25:19.6712054Z  - [deleted]                 (none)     -> trunk/4a80ddfbe70bf6b75acc3177e5d2095b285da841
2025-07-24T04:25:19.6713480Z  - [deleted]                 (none)     -> trunk/4a8f5e752beb5a6809ba866c83f32dd464a47bfd
2025-07-24T04:25:19.6715001Z  - [deleted]                 (none)     -> trunk/4a954fc1857ff41be604e8cae68908d41124b419
2025-07-24T04:25:19.6716505Z  - [deleted]                 (none)     -> trunk/4ab4d29cbee1f90daf14444d5ca3a53653c7784d
2025-07-24T04:25:19.6717981Z  - [deleted]                 (none)     -> trunk/4b11428cb5b3d97f3068a2dc4c55cee6ddd41979
2025-07-24T04:25:19.6719569Z  - [deleted]                 (none)     -> trunk/4b4c2a7b1dfd88313801878c5b4e3855fe5232df
2025-07-24T04:25:19.6721009Z  - [deleted]                 (none)     -> trunk/4b55871e06d6bad54eac45e45a9af615d758a39f
2025-07-24T04:25:19.6722648Z  - [deleted]                 (none)     -> trunk/4b6cbf528b8f7b95e02e3a0233945e0bb8b00f44
2025-07-24T04:25:19.6724029Z  - [deleted]                 (none)     -> trunk/4b9a6f7211123511e856ac8c8524bc332a741241
2025-07-24T04:25:19.6725703Z  - [deleted]                 (none)     -> trunk/4bb936d8b77c959efbff0bdb68f63e5a5faf60a9
2025-07-24T04:25:19.6727180Z  - [deleted]                 (none)     -> trunk/4bc3e4b497948af0a48d08b427339efc8960fedb
2025-07-24T04:25:19.6728655Z  - [deleted]                 (none)     -> trunk/4bd18e31e5a38d0e84ce915b1fa124058c6373fa
2025-07-24T04:25:19.6730407Z  - [deleted]                 (none)     -> trunk/4c0091fda65b714fa73671a15e379f814af153e0
2025-07-24T04:25:19.6731935Z  - [deleted]                 (none)     -> trunk/4c0aa37dda605e7cf3372c71940d7e3a93ecef8d
2025-07-24T04:25:19.6733339Z  - [deleted]                 (none)     -> trunk/4c3da611c2a80fa5073b164974d7edc0577b9ca7
2025-07-24T04:25:19.6734757Z  - [deleted]                 (none)     -> trunk/4c59edf0c5838087cbe09c67c6e0e776467a5f2d
2025-07-24T04:25:19.6736249Z  - [deleted]                 (none)     -> trunk/4c8eb65efb147cd263fc02f5588683f530363a0f
2025-07-24T04:25:19.6737809Z  - [deleted]                 (none)     -> trunk/4cbbc8b4583e9dac4af7e8d1d8535546de21178c
2025-07-24T04:25:19.6739313Z  - [deleted]                 (none)     -> trunk/4cc13c4af6dbd5aa36545ef577d0bee517cf8c57
2025-07-24T04:25:19.6740833Z  - [deleted]                 (none)     -> trunk/4cd6e96bf0d13f168772f6ad44267b1ac7632a97
2025-07-24T04:25:19.6742271Z  - [deleted]                 (none)     -> trunk/4cdbdcdbcf2b7f7a52a5c61cb35064f7b4aa2e44
2025-07-24T04:25:19.6743883Z  - [deleted]                 (none)     -> trunk/4ce6e6ec8890a3f6ee604c9efb3ff153825ce575
2025-07-24T04:25:19.6745328Z  - [deleted]                 (none)     -> trunk/4cfc0a320897b6ec75b85cfc07bc29009a824177
2025-07-24T04:25:21.2120118Z    a63f2722f9b..9c33dc30f9c  addbuildvllm                -> origin/addbuildvllm
2025-07-24T04:25:21.2198743Z  * [new branch]              gh/Sidharth123-cpu/44/base  -> origin/gh/Sidharth123-cpu/44/base
2025-07-24T04:25:21.2208310Z  * [new branch]              gh/Sidharth123-cpu/44/head  -> origin/gh/Sidharth123-cpu/44/head
2025-07-24T04:25:21.2211109Z  * [new branch]              gh/Sidharth123-cpu/44/orig  -> origin/gh/Sidharth123-cpu/44/orig
2025-07-24T04:25:21.2217467Z  * [new branch]              gh/Sidharth123-cpu/45/base  -> origin/gh/Sidharth123-cpu/45/base
2025-07-24T04:25:21.2220195Z  * [new branch]              gh/Sidharth123-cpu/45/head  -> origin/gh/Sidharth123-cpu/45/head
2025-07-24T04:25:21.2225197Z  * [new branch]              gh/Sidharth123-cpu/45/orig  -> origin/gh/Sidharth123-cpu/45/orig
2025-07-24T04:25:21.2273104Z    f4af658fa20..dad92517389  gh/benjaminglass1/79/base   -> origin/gh/benjaminglass1/79/base
2025-07-24T04:25:21.2277698Z    52da9230dfb..27c8ef1839e  gh/benjaminglass1/79/head   -> origin/gh/benjaminglass1/79/head
2025-07-24T04:25:21.2281586Z  + 46c75b7f6b7...c1380dc4c14 gh/benjaminglass1/79/orig   -> origin/gh/benjaminglass1/79/orig  (forced update)
2025-07-24T04:25:21.2287366Z    d039700eb01..8d9fd35a226  gh/benjaminglass1/89/base   -> origin/gh/benjaminglass1/89/base
2025-07-24T04:25:21.2292267Z    773e031adce..091eba09e5d  gh/benjaminglass1/89/head   -> origin/gh/benjaminglass1/89/head
2025-07-24T04:25:21.2297049Z  + 236813478bb...954dbb9e140 gh/benjaminglass1/89/orig   -> origin/gh/benjaminglass1/89/orig  (forced update)
2025-07-24T04:25:21.2302170Z    56e4568bae8..3b7f39e1133  gh/benjaminglass1/93/base   -> origin/gh/benjaminglass1/93/base
2025-07-24T04:25:21.2305944Z    019403bc448..a690e8642e3  gh/benjaminglass1/93/head   -> origin/gh/benjaminglass1/93/head
2025-07-24T04:25:21.2310195Z  + aad5ac72896...4926aacc053 gh/benjaminglass1/93/orig   -> origin/gh/benjaminglass1/93/orig  (forced update)
2025-07-24T04:25:21.2360335Z  + 48398e87c17...3a79398ade0 gh/ezyang/3109/orig         -> origin/gh/ezyang/3109/orig  (forced update)
2025-07-24T04:25:21.2366933Z    0f661b61514..4fdb7f91412  gh/ezyang/3110/base         -> origin/gh/ezyang/3110/base
2025-07-24T04:25:21.2371415Z    afcd4e5c261..7b5fe5a71db  gh/ezyang/3110/head         -> origin/gh/ezyang/3110/head
2025-07-24T04:25:21.2376322Z  + 9608417370c...8f109455683 gh/ezyang/3110/orig         -> origin/gh/ezyang/3110/orig  (forced update)
2025-07-24T04:25:21.2380496Z    28bdcabd2e9..9803dc65015  gh/ezyang/3111/base         -> origin/gh/ezyang/3111/base
2025-07-24T04:25:21.2385105Z    e88e7d69fc1..b31163436d1  gh/ezyang/3111/head         -> origin/gh/ezyang/3111/head
2025-07-24T04:25:21.2389901Z  + 730f1eb4d86...8a236f6e536 gh/ezyang/3111/orig         -> origin/gh/ezyang/3111/orig  (forced update)
2025-07-24T04:25:21.2394189Z    92c550b95aa..b90d98e2280  gh/ezyang/3112/base         -> origin/gh/ezyang/3112/base
2025-07-24T04:25:21.2399169Z    2a47a5eb9e2..27cab9dc744  gh/ezyang/3112/head         -> origin/gh/ezyang/3112/head
2025-07-24T04:25:21.2402797Z  + 1d19d14a929...3bd68b9eebe gh/ezyang/3112/orig         -> origin/gh/ezyang/3112/orig  (forced update)
2025-07-24T04:25:21.2407757Z  * [new branch]              gh/ezyang/3115/base         -> origin/gh/ezyang/3115/base
2025-07-24T04:25:21.2410804Z  * [new branch]              gh/ezyang/3115/head         -> origin/gh/ezyang/3115/head
2025-07-24T04:25:21.2414201Z  * [new branch]              gh/ezyang/3115/orig         -> origin/gh/ezyang/3115/orig
2025-07-24T04:25:21.2418663Z  * [new branch]              gh/ezyang/3116/base         -> origin/gh/ezyang/3116/base
2025-07-24T04:25:21.2421543Z  * [new branch]              gh/ezyang/3116/head         -> origin/gh/ezyang/3116/head
2025-07-24T04:25:21.2424170Z  * [new branch]              gh/ezyang/3116/orig         -> origin/gh/ezyang/3116/orig
2025-07-24T04:25:21.2445630Z  + b1bf4598b20...8c532d7320f gh/fffrog/120/orig          -> origin/gh/fffrog/120/orig  (forced update)
2025-07-24T04:25:21.2450090Z    0b6c4d78244..662f4b47362  gh/fffrog/121/head          -> origin/gh/fffrog/121/head
2025-07-24T04:25:21.2455399Z  + c3cd9cf9273...25f61c107b3 gh/fffrog/121/orig          -> origin/gh/fffrog/121/orig  (forced update)
2025-07-24T04:25:21.2459849Z    2cf131a53f9..e9b67f5334a  gh/fffrog/122/base          -> origin/gh/fffrog/122/base
2025-07-24T04:25:21.2464294Z    1de5a4948da..c86ecb1fa2f  gh/fffrog/122/head          -> origin/gh/fffrog/122/head
2025-07-24T04:25:21.2468888Z  + fb9bc1325a6...6a3bc42980f gh/fffrog/122/orig          -> origin/gh/fffrog/122/orig  (forced update)
2025-07-24T04:25:21.2473156Z    b8ddaaef998..d5ef89c7eb2  gh/fffrog/125/head          -> origin/gh/fffrog/125/head
2025-07-24T04:25:21.2477502Z  + 3ea69d0f38f...779b0782f83 gh/fffrog/125/orig          -> origin/gh/fffrog/125/orig  (forced update)
2025-07-24T04:25:21.2481967Z    f662a59a445..9e70a40efb2  gh/fffrog/126/base          -> origin/gh/fffrog/126/base
2025-07-24T04:25:21.2486607Z    eafd3f1210c..75013109407  gh/fffrog/126/head          -> origin/gh/fffrog/126/head
2025-07-24T04:25:21.2491223Z  + 938828806c6...06d26cab5be gh/fffrog/126/orig          -> origin/gh/fffrog/126/orig  (forced update)
2025-07-24T04:25:21.2527065Z    e724694cd0e..00658dfeb69  gh/jamesjwu/177/base        -> origin/gh/jamesjwu/177/base
2025-07-24T04:25:21.2532039Z    d5cf21d4fc4..1792d812af3  gh/jamesjwu/177/head        -> origin/gh/jamesjwu/177/head
2025-07-24T04:25:21.2537342Z  + d8ba665a47f...fc4dd1053d6 gh/jamesjwu/177/orig        -> origin/gh/jamesjwu/177/orig  (forced update)
2025-07-24T04:25:21.2574919Z    c447bac5ce3..565608a7081  gh/kwen2501/197/base        -> origin/gh/kwen2501/197/base
2025-07-24T04:25:21.2579743Z    87a5be4ecaf..376c425c279  gh/kwen2501/197/head        -> origin/gh/kwen2501/197/head
2025-07-24T04:25:21.2584672Z  + bce8057c719...fa047de8a2d gh/kwen2501/197/orig        -> origin/gh/kwen2501/197/orig  (forced update)
2025-07-24T04:25:21.2634795Z    a7f1bd9d2d6..2ebca39fb71  gh/ydwu4/253/base           -> origin/gh/ydwu4/253/base
2025-07-24T04:25:21.2636871Z    b03cc11da49..21e6c2dfd3a  gh/ydwu4/253/head           -> origin/gh/ydwu4/253/head
2025-07-24T04:25:21.2639050Z  + b4d70165556...4804f1cf6bb gh/ydwu4/253/orig           -> origin/gh/ydwu4/253/orig  (forced update)
2025-07-24T04:25:21.2644257Z    bf500d541ac..a39205550c3  gh/ydwu4/289/base           -> origin/gh/ydwu4/289/base
2025-07-24T04:25:21.2646332Z    7909e0b629a..123bbb64ae2  gh/ydwu4/289/head           -> origin/gh/ydwu4/289/head
2025-07-24T04:25:21.2648656Z  + 5d36abaefa1...bf9027566e9 gh/ydwu4/289/orig           -> origin/gh/ydwu4/289/orig  (forced update)
2025-07-24T04:25:21.2650888Z    f1e747d4c93..da7f76adc5d  gh/ydwu4/290/base           -> origin/gh/ydwu4/290/base
2025-07-24T04:25:21.2653299Z    c1c92882611..f882aaa76b6  gh/ydwu4/290/head           -> origin/gh/ydwu4/290/head
2025-07-24T04:25:21.2655329Z  + c0184825d59...2f2e93df74a gh/ydwu4/290/orig           -> origin/gh/ydwu4/290/orig  (forced update)
2025-07-24T04:25:21.2657891Z    0b73d3453a3..998ec733b6e  gh/ydwu4/291/base           -> origin/gh/ydwu4/291/base
2025-07-24T04:25:21.2659579Z    c6263dc0552..2a4c59db3d0  gh/ydwu4/291/head           -> origin/gh/ydwu4/291/head
2025-07-24T04:25:21.2661791Z  + 1f11b5f1050...f2c3bbfa5c7 gh/ydwu4/291/orig           -> origin/gh/ydwu4/291/orig  (forced update)
2025-07-24T04:25:21.2663824Z    8d6240e7388..0f3f25f3d4d  gh/ydwu4/292/base           -> origin/gh/ydwu4/292/base
2025-07-24T04:25:21.2665957Z    aa3267f18ca..fc737b49cb7  gh/ydwu4/292/head           -> origin/gh/ydwu4/292/head
2025-07-24T04:25:21.2668149Z  + c588754a36b...d503a839b13 gh/ydwu4/292/orig           -> origin/gh/ydwu4/292/orig  (forced update)
2025-07-24T04:25:21.2670217Z    b03cc11da49..6b3c253c8b9  gh/ydwu4/293/base           -> origin/gh/ydwu4/293/base
2025-07-24T04:25:21.2672430Z    1c4575a224f..bc7315e30ff  gh/ydwu4/293/head           -> origin/gh/ydwu4/293/head
2025-07-24T04:25:21.2674517Z  + e1a9ee9a21e...46218690756 gh/ydwu4/293/orig           -> origin/gh/ydwu4/293/orig  (forced update)
2025-07-24T04:25:21.2676466Z  * [new branch]              gh/ydwu4/295/base           -> origin/gh/ydwu4/295/base
2025-07-24T04:25:21.2677848Z  * [new branch]              gh/ydwu4/295/head           -> origin/gh/ydwu4/295/head
2025-07-24T04:25:21.2679300Z  * [new branch]              gh/ydwu4/295/orig           -> origin/gh/ydwu4/295/orig
2025-07-24T04:25:21.2681115Z  * [new branch]              gh/ydwu4/296/base           -> origin/gh/ydwu4/296/base
2025-07-24T04:25:21.2682432Z  * [new branch]              gh/ydwu4/296/head           -> origin/gh/ydwu4/296/head
2025-07-24T04:25:21.2683813Z  * [new branch]              gh/ydwu4/296/orig           -> origin/gh/ydwu4/296/orig
2025-07-24T04:25:21.2696241Z    0c0fcb53ff5..e09d80c545d  main                        -> origin/main
2025-07-24T04:25:21.2715924Z  t [tag update]              ciflow/binaries_wheel/155731 -> ciflow/binaries_wheel/155731
2025-07-24T04:25:21.2717576Z  t [tag update]              ciflow/inductor-rocm/158074 -> ciflow/inductor-rocm/158074
2025-07-24T04:25:21.2719361Z  t [tag update]              ciflow/inductor/149961      -> ciflow/inductor/149961
2025-07-24T04:25:21.2721006Z  t [tag update]              ciflow/inductor/154193      -> ciflow/inductor/154193
2025-07-24T04:25:21.2722464Z  t [tag update]              ciflow/inductor/155877      -> ciflow/inductor/155877
2025-07-24T04:25:21.2725015Z  t [tag update]              ciflow/inductor/158074      -> ciflow/inductor/158074
2025-07-24T04:25:21.2727965Z  t [tag update]              ciflow/inductor/158708      -> ciflow/inductor/158708
2025-07-24T04:25:21.2729370Z  t [tag update]              ciflow/inductor/158715      -> ciflow/inductor/158715
2025-07-24T04:25:21.2730932Z  t [tag update]              ciflow/inductor/158734      -> ciflow/inductor/158734
2025-07-24T04:25:21.2732957Z  t [tag update]              ciflow/inductor/158797      -> ciflow/inductor/158797
2025-07-24T04:25:21.2734834Z  t [tag update]              ciflow/inductor/158839      -> ciflow/inductor/158839
2025-07-24T04:25:21.2736327Z  t [tag update]              ciflow/inductor/158847      -> ciflow/inductor/158847
2025-07-24T04:25:21.2738651Z  t [tag update]              ciflow/inductor/158929      -> ciflow/inductor/158929
2025-07-24T04:25:21.2740245Z  * [new tag]                 ciflow/inductor/159005      -> ciflow/inductor/159005
2025-07-24T04:25:21.2741189Z  * [new tag]                 ciflow/inductor/159009      -> ciflow/inductor/159009
2025-07-24T04:25:21.2742209Z  * [new tag]                 ciflow/inductor/159010      -> ciflow/inductor/159010
2025-07-24T04:25:21.2743584Z  * [new tag]                 ciflow/inductor/159011      -> ciflow/inductor/159011
2025-07-24T04:25:21.2746528Z  t [tag update]              ciflow/rocm-mi300/158847    -> ciflow/rocm-mi300/158847
2025-07-24T04:25:21.2747813Z  t [tag update]              ciflow/rocm/155877          -> ciflow/rocm/155877
2025-07-24T04:25:21.2749634Z  t [tag update]              ciflow/rocm/158074          -> ciflow/rocm/158074
2025-07-24T04:25:21.2752097Z  t [tag update]              ciflow/trunk/149961         -> ciflow/trunk/149961
2025-07-24T04:25:21.2753026Z  t [tag update]              ciflow/trunk/154193         -> ciflow/trunk/154193
2025-07-24T04:25:21.2754328Z  t [tag update]              ciflow/trunk/155731         -> ciflow/trunk/155731
2025-07-24T04:25:21.2755524Z  * [new tag]                 ciflow/trunk/157892         -> ciflow/trunk/157892
2025-07-24T04:25:21.2757417Z  t [tag update]              ciflow/trunk/158441         -> ciflow/trunk/158441
2025-07-24T04:25:21.2759441Z  t [tag update]              ciflow/trunk/158670         -> ciflow/trunk/158670
2025-07-24T04:25:21.2760842Z  t [tag update]              ciflow/trunk/158708         -> ciflow/trunk/158708
2025-07-24T04:25:21.2762392Z  t [tag update]              ciflow/trunk/158847         -> ciflow/trunk/158847
2025-07-24T04:25:21.2763896Z  t [tag update]              ciflow/trunk/158929         -> ciflow/trunk/158929
2025-07-24T04:25:21.2771779Z  t [tag update]              ciflow/trunk/158938         -> ciflow/trunk/158938
2025-07-24T04:25:21.2772690Z  * [new tag]                 ciflow/trunk/159002         -> ciflow/trunk/159002
2025-07-24T04:25:21.2773443Z  * [new tag]                 ciflow/xpu/158337           -> ciflow/xpu/158337
2025-07-24T04:25:21.2774215Z  t [tag update]              ciflow/xpu/158533           -> ciflow/xpu/158533
2025-07-24T04:25:21.2775337Z  * [new tag]                 trunk/07df6ba7f5597488a93b3855d52d2ead55675125 -> trunk/07df6ba7f5597488a93b3855d52d2ead55675125
2025-07-24T04:25:21.2847245Z  * [new tag]                 trunk/d352c28dd176572120dc99f556b29c77b98e7a2b -> trunk/d352c28dd176572120dc99f556b29c77b98e7a2b
2025-07-24T04:25:21.2849438Z  * [new tag]                 trunk/e09d80c545d9f006f9fa28de6144531bb2af33ff -> trunk/e09d80c545d9f006f9fa28de6144531bb2af33ff
2025-07-24T04:25:21.4046086Z [command]/usr/bin/git rev-parse --verify --quiet 27c8ef1839e0747b36769efb192befc9308ff93c^{object}
2025-07-24T04:25:21.4183647Z 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:25:21.4196798Z ##[endgroup]
2025-07-24T04:25:21.4197565Z ##[group]Determining the checkout info
2025-07-24T04:25:21.4198345Z ##[endgroup]
2025-07-24T04:25:21.4208085Z [command]/usr/bin/git sparse-checkout disable
2025-07-24T04:25:21.4624201Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-07-24T04:25:21.4691437Z ##[group]Checking out the ref
2025-07-24T04:25:21.4699606Z [command]/usr/bin/git checkout --progress --force 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:25:21.7669774Z Previous HEAD position was 5cdb3d896e1 [FSDP][Replicate] added replicate function that uses FSDP instead of DDP (#158207)
2025-07-24T04:25:21.7688930Z HEAD is now at 27c8ef1839e Update
2025-07-24T04:25:21.7808274Z ##[endgroup]
2025-07-24T04:25:21.7809045Z ##[group]Setting up auth for fetching submodules
2025-07-24T04:25:21.7819578Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-07-24T04:25:21.7915173Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-07-24T04:25:21.7985733Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-07-24T04:25:21.8063423Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-07-24T04:25:21.8125962Z ##[endgroup]
2025-07-24T04:25:21.8126766Z ##[group]Fetching submodules
2025-07-24T04:25:21.8133764Z [command]/usr/bin/git submodule sync --recursive
2025-07-24T04:25:21.8834715Z Synchronizing submodule url for 'android/libs/fbjni'
2025-07-24T04:25:21.8947511Z Synchronizing submodule url for 'third_party/FP16'
2025-07-24T04:25:21.9057133Z Synchronizing submodule url for 'third_party/FXdiv'
2025-07-24T04:25:21.9161642Z Synchronizing submodule url for 'third_party/NNPACK'
2025-07-24T04:25:21.9267441Z Synchronizing submodule url for 'third_party/NVTX'
2025-07-24T04:25:21.9376354Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:21.9486292Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-07-24T04:25:21.9629166Z Synchronizing submodule url for 'third_party/aiter'
2025-07-24T04:25:21.9736215Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:21.9859933Z Synchronizing submodule url for 'third_party/benchmark'
2025-07-24T04:25:21.9967633Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-07-24T04:25:22.0093375Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-07-24T04:25:22.0199719Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-07-24T04:25:22.0308020Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-07-24T04:25:22.0419460Z Synchronizing submodule url for 'third_party/cutlass'
2025-07-24T04:25:22.0548297Z Synchronizing submodule url for 'third_party/fbgemm'
2025-07-24T04:25:22.0650292Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:22.0745964Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:22.0854565Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:22.0952901Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:22.1065983Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:22.1159510Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:22.1251541Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-07-24T04:25:22.1359800Z Synchronizing submodule url for 'third_party/flash-attention'
2025-07-24T04:25:22.1461948Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:22.1573178Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:22.1696637Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-07-24T04:25:22.1809995Z Synchronizing submodule url for 'third_party/fmt'
2025-07-24T04:25:22.1917957Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:22.2026243Z Synchronizing submodule url for 'third_party/gloo'
2025-07-24T04:25:22.2134559Z Synchronizing submodule url for 'third_party/googletest'
2025-07-24T04:25:22.2241617Z Synchronizing submodule url for 'third_party/ideep'
2025-07-24T04:25:22.2340364Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:22.2464578Z Synchronizing submodule url for 'third_party/ittapi'
2025-07-24T04:25:22.2569348Z Synchronizing submodule url for 'third_party/kineto'
2025-07-24T04:25:22.2669085Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:22.2757654Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:22.2855910Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:22.2952008Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:22.3046746Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:22.3144458Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:22.3249302Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:22.3348011Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:22.3444815Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:22.3553234Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:22.3661469Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:22.3755667Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:22.3874085Z Synchronizing submodule url for 'third_party/kleidiai'
2025-07-24T04:25:22.3986856Z Synchronizing submodule url for 'third_party/mimalloc'
2025-07-24T04:25:22.4095080Z Synchronizing submodule url for 'third_party/nlohmann'
2025-07-24T04:25:22.4204616Z Synchronizing submodule url for 'third_party/onnx'
2025-07-24T04:25:22.4343691Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:22.4455883Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-07-24T04:25:22.4559015Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:22.4652982Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:22.4748508Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:22.4841108Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:22.4938959Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:22.5032075Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:22.5125587Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:22.5215448Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:22.5313649Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:22.5414728Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:22.5559228Z Synchronizing submodule url for 'third_party/pocketfft'
2025-07-24T04:25:22.5666092Z Synchronizing submodule url for 'third_party/protobuf'
2025-07-24T04:25:22.5774011Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:22.5867250Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:22.5975431Z Synchronizing submodule url for 'third_party/psimd'
2025-07-24T04:25:22.6081956Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-07-24T04:25:22.6190444Z Synchronizing submodule url for 'third_party/pybind11'
2025-07-24T04:25:22.6298215Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-07-24T04:25:22.6405186Z Synchronizing submodule url for 'third_party/sleef'
2025-07-24T04:25:22.6520620Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-07-24T04:25:22.6619859Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:22.6713960Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:22.6808206Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:22.6903454Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:22.6989966Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:22.7133993Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-07-24T04:25:22.8432973Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-07-24T04:25:22.9063174Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-07-24T04:25:22.9683781Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-07-24T04:25:23.0324623Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-07-24T04:25:23.0989049Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07'
2025-07-24T04:25:23.1645712Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-07-24T04:25:23.2655086Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-07-24T04:25:23.3528271Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-07-24T04:25:23.4491815Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-07-24T04:25:23.5198008Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-07-24T04:25:23.7776274Z Submodule path 'third_party/composable_kernel': checked out '434d19f696da62c12b5372b32cbc9ba968588d7e'
2025-07-24T04:25:23.8473556Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c'
2025-07-24T04:25:23.9139250Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-07-24T04:25:23.9853609Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6'
2025-07-24T04:25:24.0716363Z Submodule path 'third_party/cutlass': checked out 'b995f933179c22d3fe0d871c3a53d11e4681950f'
2025-07-24T04:25:24.1602706Z Submodule path 'third_party/fbgemm': checked out '157e88b750c452bef2ab4653fe9d1eeb151ce4c3'
2025-07-24T04:25:24.2165650Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'e5d7c0bd5d9aec44d68830187138149e6a8c4e32'
2025-07-24T04:25:24.2885657Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out '4a61bdd4bd4ed730e078aebc7c0fcf046ff29406'
2025-07-24T04:25:24.3520525Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-07-24T04:25:24.4272183Z Submodule path 'third_party/fbgemm/external/cutlass': checked out '3ed8d2ec4ba35ef5d9d8353826209b6f868f63d3'
2025-07-24T04:25:24.4919376Z Submodule path 'third_party/fbgemm/external/googletest': checked out 'f8d7d77c06936315286eb55f8de22cd23c188571'
2025-07-24T04:25:24.5496822Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c'
2025-07-24T04:25:24.6195791Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-07-24T04:25:24.6929930Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-07-24T04:25:24.7858193Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-07-24T04:25:24.8658337Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-07-24T04:25:24.9450019Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-07-24T04:25:25.0140944Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-07-24T04:25:25.0776279Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-07-24T04:25:25.1440611Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2'
2025-07-24T04:25:25.2102687Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-07-24T04:25:25.2765933Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-07-24T04:25:25.3732652Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-07-24T04:25:25.4448819Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-07-24T04:25:25.5148707Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658'
2025-07-24T04:25:25.5806622Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-07-24T04:25:25.6452339Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-07-24T04:25:25.7065499Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-07-24T04:25:25.7684572Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-07-24T04:25:25.8303665Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-07-24T04:25:25.8875114Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-07-24T04:25:25.9484027Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-07-24T04:25:26.0106570Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-07-24T04:25:26.0813046Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-07-24T04:25:26.1412165Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-07-24T04:25:26.2019619Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-07-24T04:25:26.2620480Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-07-24T04:25:26.3311082Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7'
2025-07-24T04:25:26.4014844Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-07-24T04:25:26.4758375Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-07-24T04:25:26.5791964Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-07-24T04:25:26.6531411Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-07-24T04:25:26.7330530Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-07-24T04:25:26.7918969Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-07-24T04:25:26.8527208Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-07-24T04:25:26.9123221Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-07-24T04:25:26.9816186Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-07-24T04:25:27.0414278Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-07-24T04:25:27.0992999Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-07-24T04:25:27.1605739Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-07-24T04:25:27.2257160Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-07-24T04:25:27.2881803Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-07-24T04:25:27.3865175Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-07-24T04:25:27.4585785Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-07-24T04:25:27.5653481Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-07-24T04:25:27.6253320Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-07-24T04:25:27.6874014Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-07-24T04:25:27.7511866Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-07-24T04:25:27.8137674Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-07-24T04:25:27.8848768Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-07-24T04:25:27.9493765Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-07-24T04:25:28.0146332Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-07-24T04:25:28.0821112Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2025-07-24T04:25:28.1404170Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-07-24T04:25:28.1964782Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-07-24T04:25:28.2873853Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2025-07-24T04:25:28.3513907Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-07-24T04:25:28.4058659Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-07-24T04:25:28.4293485Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-07-24T04:25:28.4989219Z Entering 'android/libs/fbjni'
2025-07-24T04:25:28.5101211Z Entering 'third_party/FP16'
2025-07-24T04:25:28.5214258Z Entering 'third_party/FXdiv'
2025-07-24T04:25:28.5324947Z Entering 'third_party/NNPACK'
2025-07-24T04:25:28.5438043Z Entering 'third_party/NVTX'
2025-07-24T04:25:28.5552469Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:28.5670638Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:28.5815342Z Entering 'third_party/aiter'
2025-07-24T04:25:28.5925160Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:28.6052509Z Entering 'third_party/benchmark'
2025-07-24T04:25:28.6165519Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:28.6294088Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:28.6406575Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:28.6515564Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:28.6628865Z Entering 'third_party/cutlass'
2025-07-24T04:25:28.6759193Z Entering 'third_party/fbgemm'
2025-07-24T04:25:28.6872157Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:28.6974894Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:28.7092476Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:28.7196458Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:28.7317349Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:28.7419335Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:28.7525750Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:28.7640951Z Entering 'third_party/flash-attention'
2025-07-24T04:25:28.7750237Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:28.7868456Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:28.7995306Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:28.8111459Z Entering 'third_party/fmt'
2025-07-24T04:25:28.8222892Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:28.8332728Z Entering 'third_party/gloo'
2025-07-24T04:25:28.8440593Z Entering 'third_party/googletest'
2025-07-24T04:25:28.8548392Z Entering 'third_party/ideep'
2025-07-24T04:25:28.8654479Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:28.8777412Z Entering 'third_party/ittapi'
2025-07-24T04:25:28.8884198Z Entering 'third_party/kineto'
2025-07-24T04:25:28.8992536Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:28.9092173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:28.9200727Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:28.9304405Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:28.9407762Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:28.9505119Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:28.9615286Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:28.9718385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:28.9821595Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:28.9928032Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:29.0034897Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:29.0138252Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:29.0249195Z Entering 'third_party/kleidiai'
2025-07-24T04:25:29.0359838Z Entering 'third_party/mimalloc'
2025-07-24T04:25:29.0469407Z Entering 'third_party/nlohmann'
2025-07-24T04:25:29.0585330Z Entering 'third_party/onnx'
2025-07-24T04:25:29.0731997Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:29.0850984Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:29.0962963Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:29.1066996Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:29.1169418Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:29.1272043Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:29.1379246Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:29.1480923Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:29.1583684Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:29.1681478Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:29.1788177Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:29.1897939Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:29.2047827Z Entering 'third_party/pocketfft'
2025-07-24T04:25:29.2160145Z Entering 'third_party/protobuf'
2025-07-24T04:25:29.2275044Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:29.2379026Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:29.2493352Z Entering 'third_party/psimd'
2025-07-24T04:25:29.2604904Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:29.2719837Z Entering 'third_party/pybind11'
2025-07-24T04:25:29.2833417Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:29.2944816Z Entering 'third_party/sleef'
2025-07-24T04:25:29.3056162Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:29.3170146Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:29.3269332Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:29.3370429Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:29.3472842Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:29.3570070Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:29.3717955Z ##[endgroup]
2025-07-24T04:25:29.3718739Z ##[group]Persisting credentials for submodules
2025-07-24T04:25:29.3730221Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-07-24T04:25:29.4404972Z Entering 'android/libs/fbjni'
2025-07-24T04:25:29.4475217Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4475991Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4574833Z Entering 'third_party/FP16'
2025-07-24T04:25:29.4641961Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4642611Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4743803Z Entering 'third_party/FXdiv'
2025-07-24T04:25:29.4813520Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4814175Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4905936Z Entering 'third_party/NNPACK'
2025-07-24T04:25:29.4972778Z url.https://github.com/.insteadof
2025-07-24T04:25:29.4973428Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5071019Z Entering 'third_party/NVTX'
2025-07-24T04:25:29.5137753Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5138410Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5235610Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:29.5302927Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5303554Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5399386Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:29.5465903Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5466620Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5596084Z Entering 'third_party/aiter'
2025-07-24T04:25:29.5660928Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5661594Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5754973Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:29.5818914Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5819573Z url.https://github.com/.insteadof
2025-07-24T04:25:29.5934872Z Entering 'third_party/benchmark'
2025-07-24T04:25:29.6001149Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6001803Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6098703Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:29.6165242Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6170627Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6280865Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:29.6347973Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6348642Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6440492Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:29.6508111Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6508776Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6609236Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:29.6676404Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6677101Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6772351Z Entering 'third_party/cutlass'
2025-07-24T04:25:29.6838609Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6839284Z url.https://github.com/.insteadof
2025-07-24T04:25:29.6955178Z Entering 'third_party/fbgemm'
2025-07-24T04:25:29.7023428Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7024089Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7121428Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:29.7184662Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7185327Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7277107Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:29.7339101Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7339748Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7449229Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:29.7507181Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7507838Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7601667Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:29.7664047Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7664698Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7775534Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:29.7836960Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7837614Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7928636Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:29.7990347Z url.https://github.com/.insteadof
2025-07-24T04:25:29.7990980Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8081633Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:29.8144468Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8145109Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8248780Z Entering 'third_party/flash-attention'
2025-07-24T04:25:29.8313896Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8314547Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8407610Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:29.8469140Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8469782Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8574937Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:29.8635678Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8636325Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8753999Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:29.8819144Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8819806Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8918986Z Entering 'third_party/fmt'
2025-07-24T04:25:29.8984635Z url.https://github.com/.insteadof
2025-07-24T04:25:29.8985291Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9078796Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:29.9144307Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9144987Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9239490Z Entering 'third_party/gloo'
2025-07-24T04:25:29.9305480Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9306119Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9400433Z Entering 'third_party/googletest'
2025-07-24T04:25:29.9465096Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9465763Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9561106Z Entering 'third_party/ideep'
2025-07-24T04:25:29.9625013Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9625675Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9711782Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:29.9773561Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9774217Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9888399Z Entering 'third_party/ittapi'
2025-07-24T04:25:29.9954441Z url.https://github.com/.insteadof
2025-07-24T04:25:29.9955199Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0055925Z Entering 'third_party/kineto'
2025-07-24T04:25:30.0122958Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0124085Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0220201Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:30.0285062Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0285796Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0375254Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:30.0437461Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0438158Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0531122Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:30.0593874Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0594519Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0690003Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:30.0751190Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0751900Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0848223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:30.0909115Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0909755Z url.https://github.com/.insteadof
2025-07-24T04:25:30.0997859Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:30.1059863Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1060504Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1159370Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:30.1222472Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1223112Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1315213Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:30.1377724Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1378423Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1471906Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:30.1534212Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1535679Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1629564Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:30.1689911Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1690596Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1789939Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:30.1852008Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1852717Z url.https://github.com/.insteadof
2025-07-24T04:25:30.1944373Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:30.2006550Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2007395Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2105428Z Entering 'third_party/kleidiai'
2025-07-24T04:25:30.2170974Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2171614Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2266781Z Entering 'third_party/mimalloc'
2025-07-24T04:25:30.2331394Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2332060Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2431305Z Entering 'third_party/nlohmann'
2025-07-24T04:25:30.2496276Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2497003Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2594995Z Entering 'third_party/onnx'
2025-07-24T04:25:30.2661406Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2662133Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2791549Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:30.2856557Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2857192Z url.https://github.com/.insteadof
2025-07-24T04:25:30.2962068Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:30.3028589Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3029235Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3125194Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:30.3187961Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3188708Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3279741Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:30.3341902Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3342631Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3432385Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:30.3493971Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3494609Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3585731Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:30.3648460Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3649109Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3746333Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:30.3807803Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3808507Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3898307Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:30.3960339Z url.https://github.com/.insteadof
2025-07-24T04:25:30.3960979Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4052429Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:30.4113561Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4114212Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4199098Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:30.4261387Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4262076Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4357904Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:30.4419611Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4420309Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4517862Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:30.4578100Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4579314Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4718031Z Entering 'third_party/pocketfft'
2025-07-24T04:25:30.4783294Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4783947Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4878416Z Entering 'third_party/protobuf'
2025-07-24T04:25:30.4944567Z url.https://github.com/.insteadof
2025-07-24T04:25:30.4945209Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5041745Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:30.5105289Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5106030Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5199463Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:30.5262714Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5263358Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5365572Z Entering 'third_party/psimd'
2025-07-24T04:25:30.5431972Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5432620Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5523953Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:30.5590272Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5590915Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5684999Z Entering 'third_party/pybind11'
2025-07-24T04:25:30.5749941Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5750577Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5844360Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:30.5912714Z url.https://github.com/.insteadof
2025-07-24T04:25:30.5913368Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6009494Z Entering 'third_party/sleef'
2025-07-24T04:25:30.6074359Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6075005Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6170200Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:30.6234190Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6234832Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6329201Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:30.6389774Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6390401Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6480501Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:30.6542183Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6542838Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6630379Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:30.6692835Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6693482Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6784168Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:30.6848081Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6848822Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6933178Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:30.6994983Z url.https://github.com/.insteadof
2025-07-24T04:25:30.6995631Z url.https://github.com/.insteadof
2025-07-24T04:25:30.7192200Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-07-24T04:25:30.7881751Z Entering 'android/libs/fbjni'
2025-07-24T04:25:30.7990688Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-07-24T04:25:30.8049761Z Entering 'third_party/FP16'
2025-07-24T04:25:30.8153120Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-07-24T04:25:30.8210890Z Entering 'third_party/FXdiv'
2025-07-24T04:25:30.8312909Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-07-24T04:25:30.8371385Z Entering 'third_party/NNPACK'
2025-07-24T04:25:30.8473938Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-07-24T04:25:30.8533102Z Entering 'third_party/NVTX'
2025-07-24T04:25:30.8637647Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-07-24T04:25:30.8698044Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:30.8800391Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-07-24T04:25:30.8858496Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:30.8960085Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-07-24T04:25:30.9053439Z Entering 'third_party/aiter'
2025-07-24T04:25:30.9155870Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-07-24T04:25:30.9213644Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:30.9310504Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-07-24T04:25:30.9390058Z Entering 'third_party/benchmark'
2025-07-24T04:25:30.9492624Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-07-24T04:25:30.9552550Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:30.9654978Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-07-24T04:25:30.9733364Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:30.9834690Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-07-24T04:25:30.9893201Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:31.0001300Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-07-24T04:25:31.0066396Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:31.0168637Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-07-24T04:25:31.0226260Z Entering 'third_party/cutlass'
2025-07-24T04:25:31.0332779Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-07-24T04:25:31.0414491Z Entering 'third_party/fbgemm'
2025-07-24T04:25:31.0516993Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-07-24T04:25:31.0575360Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:31.0676882Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-07-24T04:25:31.0735781Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:31.0835589Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-07-24T04:25:31.0907963Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:31.1007641Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-07-24T04:25:31.1063629Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:31.1161503Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-07-24T04:25:31.1236344Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:31.1333429Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-07-24T04:25:31.1389051Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:31.1487191Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-07-24T04:25:31.1541454Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:31.1640386Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-07-24T04:25:31.1707653Z Entering 'third_party/flash-attention'
2025-07-24T04:25:31.1809720Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-07-24T04:25:31.1866153Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:31.1970828Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-07-24T04:25:31.2040015Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:31.2140109Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-07-24T04:25:31.2220259Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:31.2319019Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-07-24T04:25:31.2382884Z Entering 'third_party/fmt'
2025-07-24T04:25:31.2486391Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-07-24T04:25:31.2546402Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:31.2653752Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-07-24T04:25:31.2709122Z Entering 'third_party/gloo'
2025-07-24T04:25:31.2811670Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-07-24T04:25:31.2872244Z Entering 'third_party/googletest'
2025-07-24T04:25:31.2974689Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:31.3033084Z Entering 'third_party/ideep'
2025-07-24T04:25:31.3135892Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-07-24T04:25:31.3190627Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:31.3291941Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-07-24T04:25:31.3369126Z Entering 'third_party/ittapi'
2025-07-24T04:25:31.3473271Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-07-24T04:25:31.3533725Z Entering 'third_party/kineto'
2025-07-24T04:25:31.3636657Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-07-24T04:25:31.3691602Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:31.3793858Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-07-24T04:25:31.3845748Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:31.3950858Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-07-24T04:25:31.4009016Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:31.4114563Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-07-24T04:25:31.4169281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:31.4272487Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-07-24T04:25:31.4329176Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:31.4431182Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-07-24T04:25:31.4481929Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:31.4588355Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-07-24T04:25:31.4650017Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:31.4749793Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-07-24T04:25:31.4804556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:31.4904001Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:31.4957053Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:31.5061410Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-07-24T04:25:31.5118245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:31.5219336Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-07-24T04:25:31.5279781Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:31.5380688Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-07-24T04:25:31.5435228Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:31.5534059Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:31.5595443Z Entering 'third_party/kleidiai'
2025-07-24T04:25:31.5696972Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-07-24T04:25:31.5756968Z Entering 'third_party/mimalloc'
2025-07-24T04:25:31.5859289Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-07-24T04:25:31.5919329Z Entering 'third_party/nlohmann'
2025-07-24T04:25:31.6022478Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-07-24T04:25:31.6085552Z Entering 'third_party/onnx'
2025-07-24T04:25:31.6195485Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-07-24T04:25:31.6289518Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:31.6387164Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-07-24T04:25:31.6455725Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:31.6560234Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-07-24T04:25:31.6618457Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:31.6719537Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-07-24T04:25:31.6775151Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:31.6875701Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:31.6930970Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:31.7026483Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-07-24T04:25:31.7081266Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:31.7177480Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-07-24T04:25:31.7234802Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:31.7334862Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-07-24T04:25:31.7387530Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:31.7486686Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-07-24T04:25:31.7542184Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:31.7640865Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-07-24T04:25:31.7690609Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:31.7791896Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-07-24T04:25:31.7850335Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:31.7951946Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-07-24T04:25:31.8014124Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:31.8114815Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-07-24T04:25:31.8218572Z Entering 'third_party/pocketfft'
2025-07-24T04:25:31.8321739Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-07-24T04:25:31.8379905Z Entering 'third_party/protobuf'
2025-07-24T04:25:31.8482557Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-07-24T04:25:31.8545450Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:31.8649006Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-07-24T04:25:31.8704032Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:31.8809051Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:31.8874539Z Entering 'third_party/psimd'
2025-07-24T04:25:31.8978279Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-07-24T04:25:31.9036962Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:31.9138665Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-07-24T04:25:31.9196416Z Entering 'third_party/pybind11'
2025-07-24T04:25:31.9300347Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-07-24T04:25:31.9358709Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:31.9464923Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-07-24T04:25:31.9522923Z Entering 'third_party/sleef'
2025-07-24T04:25:31.9625393Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-07-24T04:25:31.9685277Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:31.9788040Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-07-24T04:25:31.9843137Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:31.9947548Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-07-24T04:25:32.0002119Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:32.0103711Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-07-24T04:25:32.0158861Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:32.0259537Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-07-24T04:25:32.0315244Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:32.0413367Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-07-24T04:25:32.0464155Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:32.0566650Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-07-24T04:25:32.1162158Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-07-24T04:25:32.1839303Z Entering 'android/libs/fbjni'
2025-07-24T04:25:32.1950591Z Entering 'third_party/FP16'
2025-07-24T04:25:32.2063143Z Entering 'third_party/FXdiv'
2025-07-24T04:25:32.2177343Z Entering 'third_party/NNPACK'
2025-07-24T04:25:32.2290811Z Entering 'third_party/NVTX'
2025-07-24T04:25:32.2401113Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:32.2510241Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:32.2653781Z Entering 'third_party/aiter'
2025-07-24T04:25:32.2762048Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:32.2889601Z Entering 'third_party/benchmark'
2025-07-24T04:25:32.3001577Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:32.3129713Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:32.3238173Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:32.3349025Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:32.3456358Z Entering 'third_party/cutlass'
2025-07-24T04:25:32.3587354Z Entering 'third_party/fbgemm'
2025-07-24T04:25:32.3694601Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:32.3796124Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:32.3910457Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:32.4012570Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:32.4138626Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:32.4242806Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:32.4346714Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:32.4458097Z Entering 'third_party/flash-attention'
2025-07-24T04:25:32.4568427Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:32.4689662Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:32.4819409Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:32.4934815Z Entering 'third_party/fmt'
2025-07-24T04:25:32.5048735Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:32.5158349Z Entering 'third_party/gloo'
2025-07-24T04:25:32.5268256Z Entering 'third_party/googletest'
2025-07-24T04:25:32.5379656Z Entering 'third_party/ideep'
2025-07-24T04:25:32.5490857Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:32.5614060Z Entering 'third_party/ittapi'
2025-07-24T04:25:32.5727076Z Entering 'third_party/kineto'
2025-07-24T04:25:32.5839385Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:32.5939064Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:32.6053556Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:32.6153100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:32.6265029Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:32.6373211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:32.6480378Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:32.6586267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:32.6696722Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:32.6804903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:32.6923749Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:32.7027356Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:32.7144431Z Entering 'third_party/kleidiai'
2025-07-24T04:25:32.7255836Z Entering 'third_party/mimalloc'
2025-07-24T04:25:32.7371542Z Entering 'third_party/nlohmann'
2025-07-24T04:25:32.7485382Z Entering 'third_party/onnx'
2025-07-24T04:25:32.7630189Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:32.7749192Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:32.7863956Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:32.7966219Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:32.8071517Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:32.8172584Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:32.8277414Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:32.8381825Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:32.8483564Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:32.8582732Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:32.8686402Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:32.8796506Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:32.8948911Z Entering 'third_party/pocketfft'
2025-07-24T04:25:32.9060405Z Entering 'third_party/protobuf'
2025-07-24T04:25:32.9174310Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:32.9277666Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:32.9388560Z Entering 'third_party/psimd'
2025-07-24T04:25:32.9499484Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:32.9608995Z Entering 'third_party/pybind11'
2025-07-24T04:25:32.9719385Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:32.9833391Z Entering 'third_party/sleef'
2025-07-24T04:25:32.9945465Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:33.0050864Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:33.0151774Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:33.0249666Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:33.0351504Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:33.0448507Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:33.0598624Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-07-24T04:25:33.1279016Z Entering 'android/libs/fbjni'
2025-07-24T04:25:33.1393783Z Entering 'third_party/FP16'
2025-07-24T04:25:33.1505321Z Entering 'third_party/FXdiv'
2025-07-24T04:25:33.1615310Z Entering 'third_party/NNPACK'
2025-07-24T04:25:33.1726533Z Entering 'third_party/NVTX'
2025-07-24T04:25:33.1838618Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:25:33.1949032Z Entering 'third_party/XNNPACK'
2025-07-24T04:25:33.2097940Z Entering 'third_party/aiter'
2025-07-24T04:25:33.2206593Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:25:33.2332134Z Entering 'third_party/benchmark'
2025-07-24T04:25:33.2439956Z Entering 'third_party/composable_kernel'
2025-07-24T04:25:33.2565231Z Entering 'third_party/cpp-httplib'
2025-07-24T04:25:33.2672787Z Entering 'third_party/cpuinfo'
2025-07-24T04:25:33.2780620Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:25:33.2888739Z Entering 'third_party/cutlass'
2025-07-24T04:25:33.3015346Z Entering 'third_party/fbgemm'
2025-07-24T04:25:33.3123965Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:25:33.3225148Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:25:33.3342538Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:25:33.3453080Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:25:33.3576649Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:25:33.3680482Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:25:33.3778076Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:25:33.3889728Z Entering 'third_party/flash-attention'
2025-07-24T04:25:33.3995299Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:25:33.4110508Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:25:33.4233011Z Entering 'third_party/flatbuffers'
2025-07-24T04:25:33.4347617Z Entering 'third_party/fmt'
2025-07-24T04:25:33.4455698Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:25:33.4561867Z Entering 'third_party/gloo'
2025-07-24T04:25:33.4669315Z Entering 'third_party/googletest'
2025-07-24T04:25:33.4776252Z Entering 'third_party/ideep'
2025-07-24T04:25:33.4877517Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:25:33.4998167Z Entering 'third_party/ittapi'
2025-07-24T04:25:33.5109871Z Entering 'third_party/kineto'
2025-07-24T04:25:33.5212598Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:25:33.5314267Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:25:33.5418998Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:25:33.5521430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:25:33.5622330Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:25:33.5721533Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:25:33.5828146Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:25:33.5928209Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:25:33.6027117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:25:33.6128846Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:25:33.6237207Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:25:33.6338463Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:25:33.6446070Z Entering 'third_party/kleidiai'
2025-07-24T04:25:33.6554428Z Entering 'third_party/mimalloc'
2025-07-24T04:25:33.6660250Z Entering 'third_party/nlohmann'
2025-07-24T04:25:33.6770115Z Entering 'third_party/onnx'
2025-07-24T04:25:33.6910770Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:25:33.7021700Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:25:33.7129344Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:25:33.7227458Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:25:33.7328972Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:25:33.7428592Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:25:33.7530647Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:25:33.7630979Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:25:33.7730973Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:25:33.7826806Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:25:33.7928886Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:25:33.8034091Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:25:33.8183463Z Entering 'third_party/pocketfft'
2025-07-24T04:25:33.8291759Z Entering 'third_party/protobuf'
2025-07-24T04:25:33.8402906Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:25:33.8502049Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:25:33.8606876Z Entering 'third_party/psimd'
2025-07-24T04:25:33.8714340Z Entering 'third_party/pthreadpool'
2025-07-24T04:25:33.8822684Z Entering 'third_party/pybind11'
2025-07-24T04:25:33.8931896Z Entering 'third_party/python-peachpy'
2025-07-24T04:25:33.9040381Z Entering 'third_party/sleef'
2025-07-24T04:25:33.9148858Z Entering 'third_party/tensorpipe'
2025-07-24T04:25:33.9252919Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:25:33.9350865Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:25:33.9448613Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:25:33.9546721Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:25:33.9640070Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:25:33.9782166Z ##[endgroup]
2025-07-24T04:25:34.0047078Z [command]/usr/bin/git log -1 --format=%H
2025-07-24T04:25:34.0160722Z 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:25:34.0561353Z Prepare all required actions
2025-07-24T04:25:34.0562194Z Getting action download info
2025-07-24T04:25:34.2136805Z ##[group]Run ./.github/actions/setup-rocm
2025-07-24T04:25:34.2137254Z env:
2025-07-24T04:25:34.2137553Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.2137922Z ##[endgroup]
2025-07-24T04:25:34.2175430Z ##[group]Run dpkg -l | grep -E "  rocm"
2025-07-24T04:25:34.2175947Z dpkg -l | grep -E "  rocm"
2025-07-24T04:25:34.2223905Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.2224486Z env:
2025-07-24T04:25:34.2224822Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.2225195Z ##[endgroup]
2025-07-24T04:25:34.2661616Z ii  rocm                                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) software stack meta package
2025-07-24T04:25:34.2662621Z ii  rocm-cmake                             0.14.0.60303-74~22.04                   amd64        rocm-cmake built using CMake
2025-07-24T04:25:34.2663608Z ii  rocm-core                              6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2664597Z ii  rocm-dbgapi                            0.77.0.60303-74~22.04                   amd64        Library to provide AMD GPU debugger API
2025-07-24T04:25:34.2665660Z ii  rocm-debug-agent                       2.0.3.60303-74~22.04                    amd64        Radeon Open Compute Debug Agent (ROCdebug-agent)
2025-07-24T04:25:34.2667207Z ii  rocm-developer-tools                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2668268Z ii  rocm-device-libs                       1.0.0.60303-74~22.04                    amd64        Radeon Open Compute - device libraries
2025-07-24T04:25:34.2669165Z ii  rocm-gdb                               15.2.60303-74~22.04                     amd64        ROCgdb
2025-07-24T04:25:34.2670080Z ii  rocm-hip-libraries                     6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2671131Z ii  rocm-hip-runtime                       6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2672497Z ii  rocm-hip-runtime-dev                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2673537Z ii  rocm-hip-sdk                           6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2674607Z ii  rocm-language-runtime                  6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2675577Z ii  rocm-llvm                              18.0.0.25012.60303-74~22.04             amd64        ROCm core compiler
2025-07-24T04:25:34.2676533Z ii  rocm-ml-libraries                      6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2677534Z ii  rocm-ml-sdk                            6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2678424Z ii  rocm-opencl                            2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-07-24T04:25:34.2679252Z ii  rocm-opencl-dev                        2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-07-24T04:25:34.2680233Z ii  rocm-opencl-runtime                    6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2681282Z ii  rocm-opencl-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2682766Z ii  rocm-openmp-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) OpenMP Software development Kit.
2025-07-24T04:25:34.2683827Z ii  rocm-smi-lib                           7.4.0.60303-74~22.04                    amd64        AMD System Management libraries
2025-07-24T04:25:34.2684882Z ii  rocm-utils                             6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-07-24T04:25:34.2685852Z ii  rocminfo                               1.0.0.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime rocminfo tool
2025-07-24T04:25:34.2715858Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-07-24T04:25:34.2716667Z # ignore expansion of "docker ps -q" since it could be empty
2025-07-24T04:25:34.2717265Z # shellcheck disable=SC2046
2025-07-24T04:25:34.2717749Z docker stop $(docker ps -q) || true
2025-07-24T04:25:34.2718240Z # Prune all stopped containers.
2025-07-24T04:25:34.2718723Z docker container prune -f
2025-07-24T04:25:34.2761108Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.2761653Z env:
2025-07-24T04:25:34.2761951Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.2762318Z ##[endgroup]
2025-07-24T04:25:34.3500766Z "docker stop" requires at least 1 argument.
2025-07-24T04:25:34.3501410Z See 'docker stop --help'.
2025-07-24T04:25:34.3501747Z 
2025-07-24T04:25:34.3502033Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-07-24T04:25:34.3503200Z 
2025-07-24T04:25:34.3503395Z Stop one or more running containers
2025-07-24T04:25:34.3855330Z Total reclaimed space: 0B
2025-07-24T04:25:34.3947013Z ##[group]Run cat /etc/os-release || true
2025-07-24T04:25:34.3947648Z cat /etc/os-release || true
2025-07-24T04:25:34.3948245Z cat /etc/apt/sources.list.d/rocm.list || true
2025-07-24T04:25:34.3948884Z cat /opt/rocm/.info/version || true
2025-07-24T04:25:34.3949398Z whoami
2025-07-24T04:25:34.4006649Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.4007412Z env:
2025-07-24T04:25:34.4007821Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.4008312Z ##[endgroup]
2025-07-24T04:25:34.4120800Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2025-07-24T04:25:34.4121364Z NAME="Ubuntu"
2025-07-24T04:25:34.4121750Z VERSION_ID="22.04"
2025-07-24T04:25:34.4122160Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2025-07-24T04:25:34.4122697Z VERSION_CODENAME=jammy
2025-07-24T04:25:34.4123111Z ID=ubuntu
2025-07-24T04:25:34.4123440Z ID_LIKE=debian
2025-07-24T04:25:34.4123856Z HOME_URL="https://www.ubuntu.com/"
2025-07-24T04:25:34.4124542Z SUPPORT_URL="https://help.ubuntu.com/"
2025-07-24T04:25:34.4125185Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-07-24T04:25:34.4126113Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-07-24T04:25:34.4127077Z UBUNTU_CODENAME=jammy
2025-07-24T04:25:34.4149828Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main
2025-07-24T04:25:34.4176661Z 6.3.3-74
2025-07-24T04:25:34.4220925Z pytorchci
2025-07-24T04:25:34.4270006Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-07-24T04:25:34.4270605Z dpkg -l | grep -E "  amdgpu"
2025-07-24T04:25:34.4324323Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.4325295Z env:
2025-07-24T04:25:34.4325649Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.4326049Z ##[endgroup]
2025-07-24T04:25:34.4729864Z ii  amdgpu-core                            1:6.3.60303-2119913.22.04               all          Core meta package for unified amdgpu driver.
2025-07-24T04:25:34.4730905Z ii  amdgpu-dkms                            1:6.10.5.60303-2119913.22.04            all          amdgpu driver in DKMS format.
2025-07-24T04:25:34.4731964Z ii  amdgpu-dkms-firmware                   1:6.10.5.60303-2119913.22.04            all          firmware blobs used by amdgpu driver in DKMS format
2025-07-24T04:25:34.4733516Z ii  amdgpu-install                         6.3.60303-2119913.22.04                 all          AMDGPU driver repository and installer
2025-07-24T04:25:34.4779891Z ##[group]Run rocm-smi
2025-07-24T04:25:34.4780324Z rocm-smi
2025-07-24T04:25:34.4832738Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.4833361Z env:
2025-07-24T04:25:34.4833716Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.4834124Z ##[endgroup]
2025-07-24T04:25:34.6775628Z 
2025-07-24T04:25:34.6775813Z 
2025-07-24T04:25:34.6776442Z ========================================= ROCm System Management Interface =========================================
2025-07-24T04:25:34.6777398Z =================================================== Concise Info ===================================================
2025-07-24T04:25:34.6778387Z Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2025-07-24T04:25:34.6779916Z               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
2025-07-24T04:25:34.6780717Z ====================================================================================================================
2025-07-24T04:25:34.6781972Z 0       4     0x740c,   57586  42.0°C  96.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-24T04:25:34.6783114Z 1       5     0x740c,   45873  43.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-24T04:25:34.6784825Z 2       2     0x740c,   51627  39.0°C  98.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-24T04:25:34.6785906Z 3       3     0x740c,   64489  39.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-24T04:25:34.6786985Z 4       8     0x740c,   30939  44.0°C  87.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-24T04:25:34.6788056Z 5       9     0x740c,   8466   49.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-24T04:25:34.6789089Z 6       6     0x740c,   41154  40.0°C  89.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-07-24T04:25:34.6790163Z 7       7     0x740c,   63755  43.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-07-24T04:25:34.6790903Z ====================================================================================================================
2025-07-24T04:25:34.6791615Z =============================================== End of ROCm SMI Log ================================================
2025-07-24T04:25:34.6946879Z ##[group]Run rocminfo
2025-07-24T04:25:34.6947298Z rocminfo
2025-07-24T04:25:34.6996314Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.6996939Z env:
2025-07-24T04:25:34.6997295Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.6997700Z ##[endgroup]
2025-07-24T04:25:34.8487593Z ROCk module version 6.10.5 is loaded
2025-07-24T04:25:34.8488336Z =====================    
2025-07-24T04:25:34.8488824Z HSA System Attributes    
2025-07-24T04:25:34.8489273Z =====================    
2025-07-24T04:25:34.8489741Z Runtime Version:         1.14
2025-07-24T04:25:34.8490198Z Runtime Ext Version:     1.6
2025-07-24T04:25:34.8490677Z System Timestamp Freq.:  1000.000000MHz
2025-07-24T04:25:34.8491509Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-07-24T04:25:34.8492414Z Machine Model:           LARGE                              
2025-07-24T04:25:34.8493138Z System Endianness:       LITTLE                             
2025-07-24T04:25:34.8493751Z Mwaitx:                  DISABLED
2025-07-24T04:25:34.8494227Z DMAbuf Support:          YES
2025-07-24T04:25:34.8494508Z 
2025-07-24T04:25:34.8494655Z ==========               
2025-07-24T04:25:34.8495041Z HSA Agents               
2025-07-24T04:25:34.8495446Z ==========               
2025-07-24T04:25:34.8496492Z *******                  
2025-07-24T04:25:34.8496897Z Agent 1                  
2025-07-24T04:25:34.8497262Z *******                  
2025-07-24T04:25:34.8497747Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-24T04:25:34.8498388Z   Uuid:                    CPU-XX                             
2025-07-24T04:25:34.8499026Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:25:34.8499673Z   Vendor Name:             CPU                                
2025-07-24T04:25:34.8500299Z   Feature:                 None specified                     
2025-07-24T04:25:34.8500929Z   Profile:                 FULL_PROFILE                       
2025-07-24T04:25:34.8501550Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8502202Z   Max Queue Number:        0(0x0)                             
2025-07-24T04:25:34.8502822Z   Queue Min Size:          0(0x0)                             
2025-07-24T04:25:34.8503456Z   Queue Max Size:          0(0x0)                             
2025-07-24T04:25:34.8504063Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8504647Z   Node:                    0                                  
2025-07-24T04:25:34.8505232Z   Device Type:             CPU                                
2025-07-24T04:25:34.8532057Z   Cache Info:              
2025-07-24T04:25:34.8532710Z     L1:                      32768(0x8000) KB                   
2025-07-24T04:25:34.8533326Z   Chip ID:                 0(0x0)                             
2025-07-24T04:25:34.8534448Z   ASIC Revision:           0(0x0)                             
2025-07-24T04:25:34.8535112Z   Cacheline Size:          64(0x40)                           
2025-07-24T04:25:34.8535782Z   Max Clock Freq. (MHz):   2000                               
2025-07-24T04:25:34.8536364Z   BDFID:                   0                                  
2025-07-24T04:25:34.8536970Z   Internal Node ID:        0                                  
2025-07-24T04:25:34.8537612Z   Compute Unit:            64                                 
2025-07-24T04:25:34.8538229Z   SIMDs per CU:            0                                  
2025-07-24T04:25:34.8538847Z   Shader Engines:          0                                  
2025-07-24T04:25:34.8539482Z   Shader Arrs. per Eng.:   0                                  
2025-07-24T04:25:34.8540170Z   WatchPts on Addr. Ranges:1                                  
2025-07-24T04:25:34.8540764Z   Memory Properties:       
2025-07-24T04:25:34.8541220Z   Features:                None
2025-07-24T04:25:34.8541673Z   Pool Info:               
2025-07-24T04:25:34.8542095Z     Pool 1                   
2025-07-24T04:25:34.8542621Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8543270Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:25:34.8543902Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8544567Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8545250Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8545938Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8546601Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8547161Z     Pool 2                   
2025-07-24T04:25:34.8547674Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8548309Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:25:34.8548928Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8549577Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8550246Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8550923Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8551888Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8552472Z     Pool 3                   
2025-07-24T04:25:34.8552980Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-24T04:25:34.8553589Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:25:34.8554187Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8554813Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8555482Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8556136Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8556788Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8557331Z     Pool 4                   
2025-07-24T04:25:34.8558071Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8558666Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:25:34.8559257Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8559885Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8560533Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8561193Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8561826Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8562371Z   ISA Info:                
2025-07-24T04:25:34.8563067Z *******                  
2025-07-24T04:25:34.8563444Z Agent 2                  
2025-07-24T04:25:34.8563819Z *******                  
2025-07-24T04:25:34.8564267Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-24T04:25:34.8565196Z   Uuid:                    CPU-XX                             
2025-07-24T04:25:34.8565824Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:25:34.8566486Z   Vendor Name:             CPU                                
2025-07-24T04:25:34.8567098Z   Feature:                 None specified                     
2025-07-24T04:25:34.8567690Z   Profile:                 FULL_PROFILE                       
2025-07-24T04:25:34.8568324Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8568958Z   Max Queue Number:        0(0x0)                             
2025-07-24T04:25:34.8569580Z   Queue Min Size:          0(0x0)                             
2025-07-24T04:25:34.8570190Z   Queue Max Size:          0(0x0)                             
2025-07-24T04:25:34.8570795Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8571375Z   Node:                    1                                  
2025-07-24T04:25:34.8571949Z   Device Type:             CPU                                
2025-07-24T04:25:34.8572486Z   Cache Info:              
2025-07-24T04:25:34.8572927Z     L1:                      32768(0x8000) KB                   
2025-07-24T04:25:34.8573493Z   Chip ID:                 0(0x0)                             
2025-07-24T04:25:34.8574072Z   ASIC Revision:           0(0x0)                             
2025-07-24T04:25:34.8574698Z   Cacheline Size:          64(0x40)                           
2025-07-24T04:25:34.8575322Z   Max Clock Freq. (MHz):   2000                               
2025-07-24T04:25:34.8575908Z   BDFID:                   0                                  
2025-07-24T04:25:34.8576506Z   Internal Node ID:        1                                  
2025-07-24T04:25:34.8577119Z   Compute Unit:            64                                 
2025-07-24T04:25:34.8577713Z   SIMDs per CU:            0                                  
2025-07-24T04:25:34.8578315Z   Shader Engines:          0                                  
2025-07-24T04:25:34.8578946Z   Shader Arrs. per Eng.:   0                                  
2025-07-24T04:25:34.8579593Z   WatchPts on Addr. Ranges:1                                  
2025-07-24T04:25:34.8580534Z   Memory Properties:       
2025-07-24T04:25:34.8580981Z   Features:                None
2025-07-24T04:25:34.8581406Z   Pool Info:               
2025-07-24T04:25:34.8581801Z     Pool 1                   
2025-07-24T04:25:34.8582303Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8582945Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:25:34.8583554Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8584219Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8584906Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8585627Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8586304Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8586880Z     Pool 2                   
2025-07-24T04:25:34.8587409Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8588034Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:25:34.8588659Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8589291Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8589974Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8590656Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8591705Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8592286Z     Pool 3                   
2025-07-24T04:25:34.8592802Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-24T04:25:34.8593444Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:25:34.8594060Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8594733Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8595432Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8596114Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8596795Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8597360Z     Pool 4                   
2025-07-24T04:25:34.8597889Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8598495Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:25:34.8599122Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8599785Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8600455Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:25:34.8601151Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8601814Z       Accessible by all:       TRUE                               
2025-07-24T04:25:34.8602405Z   ISA Info:                
2025-07-24T04:25:34.8602809Z *******                  
2025-07-24T04:25:34.8603228Z Agent 3                  
2025-07-24T04:25:34.8603617Z *******                  
2025-07-24T04:25:34.8604073Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8604801Z   Uuid:                    GPU-18ffdfe035d4f207               
2025-07-24T04:25:34.8605452Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8606135Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8606753Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8607385Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8608016Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8608670Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8609629Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8610260Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8610889Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8611462Z   Node:                    2                                  
2025-07-24T04:25:34.8612053Z   Device Type:             GPU                                
2025-07-24T04:25:34.8612602Z   Cache Info:              
2025-07-24T04:25:34.8613069Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8613618Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8614181Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8614800Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8615431Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8616091Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8616687Z   BDFID:                   12800                              
2025-07-24T04:25:34.8617300Z   Internal Node ID:        2                                  
2025-07-24T04:25:34.8617927Z   Compute Unit:            104                                
2025-07-24T04:25:34.8618549Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8619177Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8619814Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8620823Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8621505Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8622107Z   Memory Properties:       
2025-07-24T04:25:34.8622571Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8623178Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8623854Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8624496Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8625106Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8625598Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8626121Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8626618Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8627166Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8627828Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8628456Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8629034Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8629483Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8630010Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8630533Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8631157Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8641028Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8641818Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8642510Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8643094Z   Pool Info:               
2025-07-24T04:25:34.8643555Z     Pool 1                   
2025-07-24T04:25:34.8644075Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8645110Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8645782Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8646416Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8647544Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8648250Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8648921Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8649470Z     Pool 2                   
2025-07-24T04:25:34.8649994Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8650624Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8651244Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8651895Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8652555Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8653238Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8653901Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8654459Z     Pool 3                   
2025-07-24T04:25:34.8654964Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8655549Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8656139Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8656764Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8657404Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8657982Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8658856Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8659344Z     Pool 4                   
2025-07-24T04:25:34.8659754Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8660260Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8660774Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8661339Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8661914Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8662497Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8663078Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8663583Z   ISA Info:                
2025-07-24T04:25:34.8663940Z     ISA 1                    
2025-07-24T04:25:34.8664395Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8665026Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8665604Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8666247Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8667057Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8667620Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8668188Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8668736Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8669193Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8669655Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8670087Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8670590Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8671100Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8671520Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8671975Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8672421Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8673214Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8673718Z *******                  
2025-07-24T04:25:34.8674059Z Agent 4                  
2025-07-24T04:25:34.8674389Z *******                  
2025-07-24T04:25:34.8674780Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8675297Z   Uuid:                    GPU-1fc9e435699e2f6c               
2025-07-24T04:25:34.8675867Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8676431Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8676971Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8677516Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8678144Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8678707Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8679246Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8679798Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8680335Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8680828Z   Node:                    3                                  
2025-07-24T04:25:34.8681345Z   Device Type:             GPU                                
2025-07-24T04:25:34.8681814Z   Cache Info:              
2025-07-24T04:25:34.8682215Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8682925Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8683413Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8683938Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8684591Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8685156Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8685671Z   BDFID:                   13568                              
2025-07-24T04:25:34.8686197Z   Internal Node ID:        3                                  
2025-07-24T04:25:34.8686731Z   Compute Unit:            104                                
2025-07-24T04:25:34.8687268Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8687813Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8688370Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8688968Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8689552Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8690069Z   Memory Properties:       
2025-07-24T04:25:34.8690470Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8690994Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8691556Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8692140Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8692661Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8693091Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8693549Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8693978Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8694483Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8695051Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8695613Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8696109Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8696504Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8696969Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8697413Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8698236Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8698861Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8699459Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8700020Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8700503Z   Pool Info:               
2025-07-24T04:25:34.8700860Z     Pool 1                   
2025-07-24T04:25:34.8701313Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8701885Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8702429Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8703013Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8703611Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8704215Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8704806Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8705288Z     Pool 2                   
2025-07-24T04:25:34.8705735Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8706269Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8706807Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8707365Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8708245Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8708852Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8709419Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8709907Z     Pool 3                   
2025-07-24T04:25:34.8710328Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8710859Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8711369Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8711923Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8712514Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8713109Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8713695Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8714194Z     Pool 4                   
2025-07-24T04:25:34.8714621Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8715121Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8715663Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8716241Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8716831Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8717451Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8718018Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8718523Z   ISA Info:                
2025-07-24T04:25:34.8718879Z     ISA 1                    
2025-07-24T04:25:34.8719359Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8720007Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8720598Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8721194Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8721806Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8722366Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8723220Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8723789Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8724259Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8724844Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8725295Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8725790Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8726356Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8726770Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8727238Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8727684Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8728208Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8728699Z *******                  
2025-07-24T04:25:34.8729041Z Agent 5                  
2025-07-24T04:25:34.8729382Z *******                  
2025-07-24T04:25:34.8729761Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8730291Z   Uuid:                    GPU-8fcbcaf6dd10fade               
2025-07-24T04:25:34.8730851Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8731430Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8731983Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8732826Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8733387Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8733944Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8734499Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8735033Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8735583Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8736088Z   Node:                    4                                  
2025-07-24T04:25:34.8736588Z   Device Type:             GPU                                
2025-07-24T04:25:34.8737091Z   Cache Info:              
2025-07-24T04:25:34.8737476Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8737956Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8738441Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8738975Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8739524Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8740091Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8740618Z   BDFID:                   4352                               
2025-07-24T04:25:34.8741155Z   Internal Node ID:        4                                  
2025-07-24T04:25:34.8741709Z   Compute Unit:            104                                
2025-07-24T04:25:34.8742239Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8742807Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8743367Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8743953Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8744565Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8745079Z   Memory Properties:       
2025-07-24T04:25:34.8745495Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8746008Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8746582Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8747149Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8747986Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8748422Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8748883Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8749334Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8749822Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8750391Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8750956Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8751460Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8751848Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8752309Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8752767Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8753291Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8753899Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8754480Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8755051Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8755527Z   Pool Info:               
2025-07-24T04:25:34.8755879Z     Pool 1                   
2025-07-24T04:25:34.8756336Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8757115Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8757667Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8758228Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8758822Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8759412Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8760019Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8760526Z     Pool 2                   
2025-07-24T04:25:34.8760968Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8761511Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8762040Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8762612Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8763198Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8763796Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8764377Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8765114Z     Pool 3                   
2025-07-24T04:25:34.8765551Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8766076Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8766606Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8767157Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8767742Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8768348Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8768920Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8769433Z     Pool 4                   
2025-07-24T04:25:34.8769843Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8770354Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8770869Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8771432Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8772322Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8772920Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8773495Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8773986Z   ISA Info:                
2025-07-24T04:25:34.8774337Z     ISA 1                    
2025-07-24T04:25:34.8774788Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8775400Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8775993Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8776600Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8777235Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8777806Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8778388Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8778926Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8779389Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8779851Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8780304Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8780809Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8781294Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8781991Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8782445Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8782901Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8783416Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8783907Z *******                  
2025-07-24T04:25:34.8784250Z Agent 6                  
2025-07-24T04:25:34.8784581Z *******                  
2025-07-24T04:25:34.8784971Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8785480Z   Uuid:                    GPU-61b30b7b05420286               
2025-07-24T04:25:34.8786039Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8786610Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8787171Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8787722Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8788277Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8788848Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8789393Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8789940Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8790478Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8790993Z   Node:                    5                                  
2025-07-24T04:25:34.8791499Z   Device Type:             GPU                                
2025-07-24T04:25:34.8791996Z   Cache Info:              
2025-07-24T04:25:34.8792402Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8792876Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8793399Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8793949Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8794528Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8795085Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8795621Z   BDFID:                   5120                               
2025-07-24T04:25:34.8796161Z   Internal Node ID:        5                                  
2025-07-24T04:25:34.8796947Z   Compute Unit:            104                                
2025-07-24T04:25:34.8797497Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8798040Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8798615Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8799193Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8799793Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8800334Z   Memory Properties:       
2025-07-24T04:25:34.8800746Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8801274Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8801841Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8802425Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8802944Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8803410Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8803868Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8804323Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8804929Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8805492Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8806061Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8806839Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8807251Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8807705Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8808168Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8808704Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8809328Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8809929Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8810493Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8810993Z   Pool Info:               
2025-07-24T04:25:34.8811347Z     Pool 1                   
2025-07-24T04:25:34.8811819Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8812394Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8812962Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8813549Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8814147Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8814765Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8815349Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8815869Z     Pool 2                   
2025-07-24T04:25:34.8816313Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8816862Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8817411Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8817975Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8818574Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8819178Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8819769Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8820263Z     Pool 3                   
2025-07-24T04:25:34.8820709Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8821258Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8822109Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8822696Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8823285Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8823899Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8824471Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8824982Z     Pool 4                   
2025-07-24T04:25:34.8825420Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8825938Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8826472Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8827032Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8827640Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8828241Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8828831Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8829334Z   ISA Info:                
2025-07-24T04:25:34.8829685Z     ISA 1                    
2025-07-24T04:25:34.8830161Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8830772Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8831375Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8832216Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8832833Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8833397Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8833944Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8834487Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8834950Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8835418Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8835866Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8836399Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8836911Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8837336Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8837802Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8838250Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8838766Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8839240Z *******                  
2025-07-24T04:25:34.8839579Z Agent 7                  
2025-07-24T04:25:34.8839895Z *******                  
2025-07-24T04:25:34.8840289Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8840802Z   Uuid:                    GPU-5f75bf9184112d67               
2025-07-24T04:25:34.8841361Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8841942Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8842480Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8843029Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8843578Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8844144Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8844803Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8845345Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8845888Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8846690Z   Node:                    6                                  
2025-07-24T04:25:34.8847218Z   Device Type:             GPU                                
2025-07-24T04:25:34.8847687Z   Cache Info:              
2025-07-24T04:25:34.8848083Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8848547Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8849031Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8849565Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8850138Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8850729Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8851246Z   BDFID:                   44544                              
2025-07-24T04:25:34.8851781Z   Internal Node ID:        6                                  
2025-07-24T04:25:34.8852321Z   Compute Unit:            104                                
2025-07-24T04:25:34.8852869Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8853421Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8853980Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8854563Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8855146Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8855682Z   Memory Properties:       
2025-07-24T04:25:34.8856388Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8856906Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8857481Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8858040Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8858560Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8858989Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8859461Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8859904Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8860400Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8860954Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8861516Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8862023Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8862423Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8862892Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8863344Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8863880Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8864485Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8865092Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8865673Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8866159Z   Pool Info:               
2025-07-24T04:25:34.8866525Z     Pool 1                   
2025-07-24T04:25:34.8866976Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8867533Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8868073Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8868664Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8869266Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8869874Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8870468Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8870969Z     Pool 2                   
2025-07-24T04:25:34.8871678Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8872233Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8872774Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8873343Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8873931Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8874549Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8875139Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8875642Z     Pool 3                   
2025-07-24T04:25:34.8876083Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8876624Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8877168Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8877737Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8878339Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8878935Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8879524Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8880019Z     Pool 4                   
2025-07-24T04:25:34.8880451Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8881221Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8881739Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8882316Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8882898Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8883491Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8884068Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8884789Z   ISA Info:                
2025-07-24T04:25:34.8885129Z     ISA 1                    
2025-07-24T04:25:34.8885598Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8886218Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8886806Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8887399Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8888013Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8888580Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8889135Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8889679Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8890144Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8890613Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8891072Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8891573Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8892084Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8892502Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8892972Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8893444Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8893961Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8894458Z *******                  
2025-07-24T04:25:34.8894791Z Agent 8                  
2025-07-24T04:25:34.8895132Z *******                  
2025-07-24T04:25:34.8895506Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8896321Z   Uuid:                    GPU-5fa6c987997a0603               
2025-07-24T04:25:34.8896899Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8897491Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8898058Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8898608Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8899175Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8899757Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8900327Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8900869Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8901425Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8901946Z   Node:                    7                                  
2025-07-24T04:25:34.8902479Z   Device Type:             GPU                                
2025-07-24T04:25:34.8902984Z   Cache Info:              
2025-07-24T04:25:34.8903376Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8903875Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8904352Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8904893Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8905461Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8906356Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8906886Z   BDFID:                   45824                              
2025-07-24T04:25:34.8907408Z   Internal Node ID:        7                                  
2025-07-24T04:25:34.8907979Z   Compute Unit:            104                                
2025-07-24T04:25:34.8908513Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8909074Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8909658Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8910241Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8910847Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8911359Z   Memory Properties:       
2025-07-24T04:25:34.8911769Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8912302Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8912890Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8913455Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8913988Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8914426Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8914883Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8915339Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8915833Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8916408Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8916969Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8917479Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8917885Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8918350Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8918822Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8919351Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8919969Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8920561Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8921390Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8921899Z   Pool Info:               
2025-07-24T04:25:34.8922248Z     Pool 1                   
2025-07-24T04:25:34.8922714Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8923262Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8923818Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8924405Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8925128Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8925751Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8926340Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8926857Z     Pool 2                   
2025-07-24T04:25:34.8927313Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8927876Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8928421Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8929013Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8929615Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8930231Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8930839Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8931640Z     Pool 3                   
2025-07-24T04:25:34.8932102Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8932639Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8933195Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8933753Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8934361Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8934966Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8935550Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8936052Z     Pool 4                   
2025-07-24T04:25:34.8936464Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8936997Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8937547Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8938132Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8938727Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8939311Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8939891Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8940393Z   ISA Info:                
2025-07-24T04:25:34.8940752Z     ISA 1                    
2025-07-24T04:25:34.8941203Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8941827Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8942421Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8943009Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8943632Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8944183Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8944750Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8945291Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8945760Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8946556Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8947012Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8947525Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8948031Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8948449Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8948906Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8949367Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8949899Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8950390Z *******                  
2025-07-24T04:25:34.8950739Z Agent 9                  
2025-07-24T04:25:34.8951075Z *******                  
2025-07-24T04:25:34.8951465Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8951974Z   Uuid:                    GPU-805d7be51a59017b               
2025-07-24T04:25:34.8952559Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8953221Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8953775Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8954325Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8954874Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8955449Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8956263Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8956810Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8957351Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8957861Z   Node:                    8                                  
2025-07-24T04:25:34.8958397Z   Device Type:             GPU                                
2025-07-24T04:25:34.8958890Z   Cache Info:              
2025-07-24T04:25:34.8959281Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8959746Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.8960231Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.8960749Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.8961318Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.8961900Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.8962421Z   BDFID:                   36352                              
2025-07-24T04:25:34.8962958Z   Internal Node ID:        8                                  
2025-07-24T04:25:34.8963500Z   Compute Unit:            104                                
2025-07-24T04:25:34.8964039Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.8964847Z   Shader Engines:          8                                  
2025-07-24T04:25:34.8965448Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.8966042Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.8966644Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.8967222Z   Memory Properties:       
2025-07-24T04:25:34.8967689Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.8968308Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.8968987Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.8969676Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8970284Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8970798Z     x                        1024(0x400)                        
2025-07-24T04:25:34.8971352Z     y                        1024(0x400)                        
2025-07-24T04:25:34.8972253Z     z                        1024(0x400)                        
2025-07-24T04:25:34.8972891Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.8973393Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.8973726Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8973984Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.8974186Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8974409Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8974644Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8974903Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.8975188Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.8975485Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.8975756Z   IOMMU Support::          None                               
2025-07-24T04:25:34.8976007Z   Pool Info:               
2025-07-24T04:25:34.8976174Z     Pool 1                   
2025-07-24T04:25:34.8976400Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.8976671Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8976933Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8977211Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8977645Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8977943Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8984674Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8984943Z     Pool 2                   
2025-07-24T04:25:34.8985176Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.8985451Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8985710Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8985977Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8986258Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8986534Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8986817Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8987068Z     Pool 3                   
2025-07-24T04:25:34.8987283Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.8987544Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.8987796Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.8988056Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.8988330Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.8988614Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.8988887Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8989112Z     Pool 4                   
2025-07-24T04:25:34.8989308Z       Segment:                 GROUP                              
2025-07-24T04:25:34.8989538Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.8989786Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.8990049Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.8990324Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.8990597Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.8990861Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.8991092Z   ISA Info:                
2025-07-24T04:25:34.8991418Z     ISA 1                    
2025-07-24T04:25:34.8991641Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.8991928Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.8992201Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.8992466Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8992741Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.8993006Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.8993256Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.8993499Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.8993709Z         x                        1024(0x400)                        
2025-07-24T04:25:34.8993920Z         y                        1024(0x400)                        
2025-07-24T04:25:34.8994124Z         z                        1024(0x400)                        
2025-07-24T04:25:34.8994352Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.8994583Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.8994767Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8994977Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8995183Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.8995572Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.8995793Z *******                  
2025-07-24T04:25:34.8995947Z Agent 10                 
2025-07-24T04:25:34.8996101Z *******                  
2025-07-24T04:25:34.8996285Z   Name:                    gfx90a                             
2025-07-24T04:25:34.8996528Z   Uuid:                    GPU-7fd720b66acc078e               
2025-07-24T04:25:34.8996786Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:25:34.8997052Z   Vendor Name:             AMD                                
2025-07-24T04:25:34.8997301Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:25:34.8997551Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:25:34.8997803Z   Float Round Mode:        NEAR                               
2025-07-24T04:25:34.8998060Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:25:34.8998315Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:25:34.8998566Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:25:34.8998808Z   Queue Type:              MULTI                              
2025-07-24T04:25:34.8999035Z   Node:                    9                                  
2025-07-24T04:25:34.8999265Z   Device Type:             GPU                                
2025-07-24T04:25:34.8999485Z   Cache Info:              
2025-07-24T04:25:34.8999673Z     L1:                      16(0x10) KB                        
2025-07-24T04:25:34.8999892Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:25:34.9000111Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:25:34.9000352Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:25:34.9000603Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:25:34.9000859Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:25:34.9001095Z   BDFID:                   37632                              
2025-07-24T04:25:34.9001337Z   Internal Node ID:        9                                  
2025-07-24T04:25:34.9001588Z   Compute Unit:            104                                
2025-07-24T04:25:34.9001830Z   SIMDs per CU:            4                                  
2025-07-24T04:25:34.9002077Z   Shader Engines:          8                                  
2025-07-24T04:25:34.9002457Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:25:34.9002727Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:25:34.9002997Z   Coherent Host Access:    FALSE                              
2025-07-24T04:25:34.9003232Z   Memory Properties:       
2025-07-24T04:25:34.9003413Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:25:34.9003651Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:25:34.9003912Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:25:34.9004176Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.9004497Z   Workgroup Max Size per Dimension:
2025-07-24T04:25:34.9004696Z     x                        1024(0x400)                        
2025-07-24T04:25:34.9004901Z     y                        1024(0x400)                        
2025-07-24T04:25:34.9005097Z     z                        1024(0x400)                        
2025-07-24T04:25:34.9005331Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:25:34.9005587Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:25:34.9005837Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.9006064Z   Grid Max Size per Dimension:
2025-07-24T04:25:34.9006241Z     x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9006449Z     y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9006654Z     z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9007049Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:25:34.9007336Z   Packet Processor uCode:: 92                                 
2025-07-24T04:25:34.9007611Z   SDMA engine uCode::      9                                  
2025-07-24T04:25:34.9007870Z   IOMMU Support::          None                               
2025-07-24T04:25:34.9008092Z   Pool Info:               
2025-07-24T04:25:34.9008259Z     Pool 1                   
2025-07-24T04:25:34.9008466Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:25:34.9008729Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.9008985Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.9009246Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.9009520Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.9009802Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.9010071Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.9010294Z     Pool 2                   
2025-07-24T04:25:34.9010500Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:25:34.9010745Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.9010997Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.9011253Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.9011517Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.9011784Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.9012044Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.9012268Z     Pool 3                   
2025-07-24T04:25:34.9012462Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:25:34.9012709Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:25:34.9012951Z       Allocatable:             TRUE                               
2025-07-24T04:25:34.9013205Z       Alloc Granule:           4KB                                
2025-07-24T04:25:34.9013473Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:25:34.9013885Z       Alloc Alignment:         4KB                                
2025-07-24T04:25:34.9014152Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.9014374Z     Pool 4                   
2025-07-24T04:25:34.9014566Z       Segment:                 GROUP                              
2025-07-24T04:25:34.9014801Z       Size:                    64(0x40) KB                        
2025-07-24T04:25:34.9015035Z       Allocatable:             FALSE                              
2025-07-24T04:25:34.9015290Z       Alloc Granule:           0KB                                
2025-07-24T04:25:34.9015561Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:25:34.9015830Z       Alloc Alignment:         0KB                                
2025-07-24T04:25:34.9016091Z       Accessible by all:       FALSE                              
2025-07-24T04:25:34.9016318Z   ISA Info:                
2025-07-24T04:25:34.9016475Z     ISA 1                    
2025-07-24T04:25:34.9016685Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:25:34.9016963Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:25:34.9017226Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:25:34.9017491Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.9017762Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:25:34.9018014Z       Fast f16:                TRUE                               
2025-07-24T04:25:34.9018403Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:25:34.9018641Z       Workgroup Max Size per Dimension:
2025-07-24T04:25:34.9018851Z         x                        1024(0x400)                        
2025-07-24T04:25:34.9019063Z         y                        1024(0x400)                        
2025-07-24T04:25:34.9019266Z         z                        1024(0x400)                        
2025-07-24T04:25:34.9019498Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:25:34.9019729Z       Grid Max Size per Dimension:
2025-07-24T04:25:34.9019919Z         x                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9020128Z         y                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9020334Z         z                        4294967295(0xffffffff)             
2025-07-24T04:25:34.9020568Z       FBarrier Max Size:       32                                 
2025-07-24T04:25:34.9020797Z *** Done ***             
2025-07-24T04:25:34.9037125Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-07-24T04:25:34.9037437Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-07-24T04:25:34.9037944Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-07-24T04:25:34.9038422Z if [[ $ngpu -eq 0 ]]; then
2025-07-24T04:25:34.9038673Z     echo "Error: Failed to detect any GPUs on the runner"
2025-07-24T04:25:34.9038918Z     echo "$msg"
2025-07-24T04:25:34.9039077Z     exit 1
2025-07-24T04:25:34.9039223Z fi
2025-07-24T04:25:34.9039366Z if [[ $ngpu -eq 1 ]]; then
2025-07-24T04:25:34.9039668Z     echo "Error: only 1 GPU detected, at least 2 GPUs are needed for distributed jobs"
2025-07-24T04:25:34.9039973Z     echo "$msg"
2025-07-24T04:25:34.9040130Z     exit 1
2025-07-24T04:25:34.9040278Z fi
2025-07-24T04:25:34.9061272Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:34.9061516Z env:
2025-07-24T04:25:34.9061652Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:34.9061817Z ##[endgroup]
2025-07-24T04:25:35.0716839Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-07-24T04:25:35.0717522Z with:
2025-07-24T04:25:35.0717882Z   diskspace-cutoff: 70
2025-07-24T04:25:35.0718259Z env:
2025-07-24T04:25:35.0719054Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.0719486Z ##[endgroup]
2025-07-24T04:25:35.0792765Z ##[group]Run set -ex
2025-07-24T04:25:35.0793270Z set -ex
2025-07-24T04:25:35.0793653Z diskspace_cutoff=70
2025-07-24T04:25:35.0794266Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-07-24T04:25:35.0794951Z if [ ! -d "$docker_root_dir" ]; then
2025-07-24T04:25:35.0795864Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-07-24T04:25:35.0796724Z     exit 0
2025-07-24T04:25:35.0797101Z fi
2025-07-24T04:25:35.0797808Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-24T04:25:35.0799325Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-07-24T04:25:35.0800644Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-07-24T04:25:35.0801277Z     docker system prune -af
2025-07-24T04:25:35.0802134Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-24T04:25:35.0803090Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-07-24T04:25:35.0804073Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-07-24T04:25:35.0805366Z         echo "$msg"
2025-07-24T04:25:35.0806278Z         exit 1
2025-07-24T04:25:35.0806684Z     else
2025-07-24T04:25:35.0807153Z         difference=$((diskspace - diskspace_new))
2025-07-24T04:25:35.0807827Z         echo "Diskspace saved: $difference percent"
2025-07-24T04:25:35.0808384Z     fi
2025-07-24T04:25:35.0808721Z fi
2025-07-24T04:25:35.0858388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:35.0858937Z env:
2025-07-24T04:25:35.0859265Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.0859630Z ##[endgroup]
2025-07-24T04:25:35.0942845Z + diskspace_cutoff=70
2025-07-24T04:25:35.0951512Z ++ docker info -f '{{.DockerRootDir}}'
2025-07-24T04:25:35.1917180Z + docker_root_dir=/media/4TB/docker-rootless
2025-07-24T04:25:35.1917902Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-07-24T04:25:35.1932789Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-07-24T04:25:35.1938041Z ++ sed -n 2p
2025-07-24T04:25:35.1940470Z ++ sed s/%//
2025-07-24T04:25:35.1944294Z ++ sed 's/ //'
2025-07-24T04:25:35.1986140Z + diskspace=38
2025-07-24T04:25:35.1987383Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-07-24T04:25:35.1988514Z + [[ 38 -ge 70 ]]
2025-07-24T04:25:35.2038604Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-07-24T04:25:35.2039319Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-07-24T04:25:35.2039904Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-07-24T04:25:35.2040409Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-07-24T04:25:35.2041064Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-07-24T04:25:35.2041687Z 
2025-07-24T04:25:35.2042119Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-07-24T04:25:35.2042743Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-07-24T04:25:35.2043353Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-07-24T04:25:35.2044205Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-07-24T04:25:35.2045005Z 
2025-07-24T04:25:35.2045338Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-07-24T04:25:35.2045826Z rm -rf "${RUNNER_DOCS_DIR}"
2025-07-24T04:25:35.2046273Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-07-24T04:25:35.2046959Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-07-24T04:25:35.2091125Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:35.2091683Z env:
2025-07-24T04:25:35.2091992Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.2092350Z ##[endgroup]
2025-07-24T04:25:35.2414793Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-24T04:25:35.2415736Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-24T04:25:35.2416565Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-07-24T04:25:35.2469936Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:35.2470557Z env:
2025-07-24T04:25:35.2470899Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.2471598Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:35.2472648Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:35.2473618Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:35.2474305Z ##[endgroup]
2025-07-24T04:25:35.2701565Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-07-24T04:25:35.2702876Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-07-24T04:25:35.2703765Z # Add render group for container creation.
2025-07-24T04:25:35.2704502Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-07-24T04:25:35.2705844Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-07-24T04:25:35.2706753Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-07-24T04:25:35.2707515Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-07-24T04:25:35.2708142Z else
2025-07-24T04:25:35.2708566Z   DEVICE_FLAG="--device /dev/dri"
2025-07-24T04:25:35.2709073Z fi
2025-07-24T04:25:35.2709913Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-07-24T04:25:35.2711171Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-07-24T04:25:35.2712350Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-07-24T04:25:35.2713580Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-07-24T04:25:35.2715688Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-07-24T04:25:35.2770972Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:35.2771606Z env:
2025-07-24T04:25:35.2771953Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.2772684Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:35.2773730Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:35.2774711Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:35.2775377Z ##[endgroup]
2025-07-24T04:25:35.3049131Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-07-24T04:25:35.3050035Z with:
2025-07-24T04:25:35.3050686Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-07-24T04:25:35.3051589Z   aws-region: us-east-1
2025-07-24T04:25:35.3052127Z   role-duration-seconds: 18000
2025-07-24T04:25:35.3052683Z   audience: sts.amazonaws.com
2025-07-24T04:25:35.3053123Z env:
2025-07-24T04:25:35.3053448Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.3054156Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:35.3055207Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:35.3056195Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:35.3057939Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:35.3059361Z ##[endgroup]
2025-07-24T04:25:35.6409146Z Assuming role with OIDC
2025-07-24T04:25:35.8163015Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-07-24T04:25:35.8891664Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076
2025-07-24T04:25:35.8892540Z with:
2025-07-24T04:25:35.8892953Z   mask-password: true
2025-07-24T04:25:35.8893401Z   registry-type: private
2025-07-24T04:25:35.8893859Z   skip-logout: false
2025-07-24T04:25:35.8894257Z env:
2025-07-24T04:25:35.8894623Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:35.8895353Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:35.8896417Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:35.8897417Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:35.8899104Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:35.8900611Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:35.8901653Z   AWS_REGION: us-east-1
2025-07-24T04:25:35.8902688Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:35.8903373Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:35.8914373Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:35.8914824Z ##[endgroup]
2025-07-24T04:25:36.2436541Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.6646130Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-07-24T04:25:36.6646891Z with:
2025-07-24T04:25:36.6647977Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.6649222Z   use-custom-docker-registry: true
2025-07-24T04:25:36.6649702Z   docker-build-dir: .ci/docker
2025-07-24T04:25:36.6650140Z   docker-build-script: ./build.sh
2025-07-24T04:25:36.6650578Z   working-directory: .
2025-07-24T04:25:36.6651091Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.6651723Z   force-push: false
2025-07-24T04:25:36.6652065Z env:
2025-07-24T04:25:36.6652388Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:36.6653028Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:36.6653960Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:36.6654900Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:36.6656402Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:36.6657712Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:36.6658135Z   AWS_REGION: us-east-1
2025-07-24T04:25:36.6658660Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:36.6659246Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:36.6668648Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:36.6669040Z ##[endgroup]
2025-07-24T04:25:36.6702210Z ##[group]Run set -ex
2025-07-24T04:25:36.6702668Z set -ex
2025-07-24T04:25:36.6703006Z 
2025-07-24T04:25:36.6703608Z # If the docker build directory or the build script doesn't exist, the action will
2025-07-24T04:25:36.6704607Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-07-24T04:25:36.6705443Z # job could then download the pre-built image as usual
2025-07-24T04:25:36.6706461Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-07-24T04:25:36.6707410Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6707908Z else
2025-07-24T04:25:36.6708296Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6708958Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6709571Z 
2025-07-24T04:25:36.6710412Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-07-24T04:25:36.6711387Z   exit 0
2025-07-24T04:25:36.6711745Z fi
2025-07-24T04:25:36.6712075Z 
2025-07-24T04:25:36.6712606Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-07-24T04:25:36.6713521Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-07-24T04:25:36.6714338Z   # use it as it is, but first let's extract the tag
2025-07-24T04:25:36.6715069Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-07-24T04:25:36.6715854Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6716597Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6717189Z else
2025-07-24T04:25:36.6717602Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-07-24T04:25:36.6718577Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-07-24T04:25:36.6719190Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-07-24T04:25:36.6719698Z   fi
2025-07-24T04:25:36.6720715Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-07-24T04:25:36.6721657Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6722633Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6723660Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.6724291Z fi
2025-07-24T04:25:36.6772799Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:36.6773376Z env:
2025-07-24T04:25:36.6773712Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:36.6774362Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:36.6775342Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:36.6776249Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:36.6777752Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:36.6779112Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:36.6779554Z   AWS_REGION: us-east-1
2025-07-24T04:25:36.6780101Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:36.6780704Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:36.6790178Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:36.6790588Z   REPO_NAME: pytorch
2025-07-24T04:25:36.6791716Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.6792933Z   DOCKER_BUILD_DIR: .ci/docker
2025-07-24T04:25:36.6793385Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-07-24T04:25:36.6793961Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.6794580Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-07-24T04:25:36.6795034Z   CUSTOM_TAG_PREFIX: 
2025-07-24T04:25:36.6795420Z ##[endgroup]
2025-07-24T04:25:36.6882683Z + [[ -d .ci/docker ]]
2025-07-24T04:25:36.6883273Z + [[ -f .ci/docker/./build.sh ]]
2025-07-24T04:25:36.6883787Z + [[ true == \t\r\u\e ]]
2025-07-24T04:25:36.6884241Z + echo skip=false
2025-07-24T04:25:36.6886145Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-07-24T04:25:36.6903211Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.6905571Z ++ awk -F '[:,]' '{print $2}'
2025-07-24T04:25:36.6966360Z + DOCKER_TAG=pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.6967721Z + echo docker-tag=pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.6969541Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.7033014Z ##[group]Run set +e
2025-07-24T04:25:36.7033571Z set +e
2025-07-24T04:25:36.7033980Z set -x
2025-07-24T04:25:36.7034380Z 
2025-07-24T04:25:36.7034768Z login() {
2025-07-24T04:25:36.7035654Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-07-24T04:25:36.7036576Z }
2025-07-24T04:25:36.7036952Z 
2025-07-24T04:25:36.7037323Z retry () {
2025-07-24T04:25:36.7038259Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-07-24T04:25:36.7038808Z }
2025-07-24T04:25:36.7039170Z 
2025-07-24T04:25:36.7039574Z retry login "${DOCKER_REGISTRY}"
2025-07-24T04:25:36.7040106Z 
2025-07-24T04:25:36.7040497Z START_TIME=$(date +%s)
2025-07-24T04:25:36.7041430Z # Wait up to 120 minutes
2025-07-24T04:25:36.7042100Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-07-24T04:25:36.7043002Z   # Check if image already exists, if it does then skip building it
2025-07-24T04:25:36.7043875Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-07-24T04:25:36.7044890Z     exit 0
2025-07-24T04:25:36.7045327Z   fi
2025-07-24T04:25:36.7045724Z 
2025-07-24T04:25:36.7046512Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-07-24T04:25:36.7047886Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-07-24T04:25:36.7049110Z   # latter, it will wait for the Docker images to become available before continuing
2025-07-24T04:25:36.7050028Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-07-24T04:25:36.7050748Z     # It's a Docker build job, let's build the image
2025-07-24T04:25:36.7051375Z     break
2025-07-24T04:25:36.7051798Z   else
2025-07-24T04:25:36.7052403Z     # It's a regular build job, wait for the image to become available
2025-07-24T04:25:36.7053149Z     sleep 300
2025-07-24T04:25:36.7053658Z   fi
2025-07-24T04:25:36.7054108Z done
2025-07-24T04:25:36.7054558Z 
2025-07-24T04:25:36.7055307Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-07-24T04:25:36.7056511Z # be empty.  The default action would be to continue rebuild the image
2025-07-24T04:25:36.7057602Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-07-24T04:25:36.7058520Z   # if we're on the base branch then use the parent commit
2025-07-24T04:25:36.7059255Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-07-24T04:25:36.7059829Z else
2025-07-24T04:25:36.7060421Z   # otherwise we're on a PR, so use the most recent base commit
2025-07-24T04:25:36.7061264Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-07-24T04:25:36.7061912Z fi
2025-07-24T04:25:36.7062289Z 
2025-07-24T04:25:36.7062696Z if [[ -z "${MERGE_BASE}" ]]; then
2025-07-24T04:25:36.7063329Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.7063995Z 
2025-07-24T04:25:36.7064961Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-07-24T04:25:36.7066115Z   exit 0
2025-07-24T04:25:36.7066568Z fi
2025-07-24T04:25:36.7066943Z 
2025-07-24T04:25:36.7067506Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-07-24T04:25:36.7068720Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-07-24T04:25:36.7069743Z   exit 1
2025-07-24T04:25:36.7070129Z fi
2025-07-24T04:25:36.7070492Z 
2025-07-24T04:25:36.7071125Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-07-24T04:25:36.7072285Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-07-24T04:25:36.7073319Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-07-24T04:25:36.7074521Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-07-24T04:25:36.7075870Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-07-24T04:25:36.7077101Z fi
2025-07-24T04:25:36.7077474Z 
2025-07-24T04:25:36.7077947Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-07-24T04:25:36.7132766Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:36.7133516Z env:
2025-07-24T04:25:36.7133980Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:36.7135328Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:36.7136735Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:36.7138003Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:36.7139828Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:36.7141426Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:36.7141948Z   AWS_REGION: us-east-1
2025-07-24T04:25:36.7142582Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:36.7143329Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:36.7155183Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:36.7155682Z   DOCKER_BUILD_DIR: .ci/docker
2025-07-24T04:25:36.7156292Z   BASE_REVISION: 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:25:36.7157820Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.7159567Z   DOCKER_TAG: pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:36.7160653Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.7161351Z   DOCKER_PUSH: 
2025-07-24T04:25:36.7161754Z ##[endgroup]
2025-07-24T04:25:36.7250398Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.7251092Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:36.7258779Z + aws ecr get-login-password --region us-east-1
2025-07-24T04:25:36.7262428Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:37.8303880Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-07-24T04:25:37.8305055Z Configure a credential helper to remove this warning. See
2025-07-24T04:25:37.8306136Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-07-24T04:25:37.8306859Z 
2025-07-24T04:25:37.8307874Z Login Succeeded
2025-07-24T04:25:37.8367869Z ++ date +%s
2025-07-24T04:25:37.8395663Z + START_TIME=1753331137
2025-07-24T04:25:37.8405600Z ++ date +%s
2025-07-24T04:25:37.8434135Z + [[ 1753323937 -lt 1753331137 ]]
2025-07-24T04:25:37.8435739Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:38.5297147Z {
2025-07-24T04:25:38.5297569Z 	"schemaVersion": 2,
2025-07-24T04:25:38.5298399Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-07-24T04:25:38.5299542Z 	"config": {
2025-07-24T04:25:38.5300300Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-07-24T04:25:38.5301066Z 		"size": 28426,
2025-07-24T04:25:38.5301769Z 		"digest": "sha256:441c23f68601012989b37ca64859d010f9dc3cbc8ea65694e4ace6ed235631cc"
2025-07-24T04:25:38.5302566Z 	},
2025-07-24T04:25:38.5302933Z 	"layers": [
2025-07-24T04:25:38.5303278Z 		{
2025-07-24T04:25:38.5303849Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5304553Z 			"size": 30446997,
2025-07-24T04:25:38.5305246Z 			"digest": "sha256:66587c81b81a58d07e40c48d900a1517516bbf58c4378c687d89d645824f5e5f"
2025-07-24T04:25:38.5306014Z 		},
2025-07-24T04:25:38.5306337Z 		{
2025-07-24T04:25:38.5306884Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5307567Z 			"size": 1554,
2025-07-24T04:25:38.5308231Z 			"digest": "sha256:c467531473a7140e2846e8648a245301c77a465d0561b48e8dd31b30e191e523"
2025-07-24T04:25:38.5309541Z 		},
2025-07-24T04:25:38.5309859Z 		{
2025-07-24T04:25:38.5310398Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5311068Z 			"size": 313286081,
2025-07-24T04:25:38.5311783Z 			"digest": "sha256:6edafb9a3aa208d2ac734c6205f560f7b7dcdec1a7b832bc7894e1278787f2b1"
2025-07-24T04:25:38.5312897Z 		},
2025-07-24T04:25:38.5313229Z 		{
2025-07-24T04:25:38.5313768Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5314449Z 			"size": 704,
2025-07-24T04:25:38.5315138Z 			"digest": "sha256:67bbdb9be01f9e05df0be4996ff180f6ee99763b4e8bed028f22e8c5c687f1f3"
2025-07-24T04:25:38.5316028Z 		},
2025-07-24T04:25:38.5316400Z 		{
2025-07-24T04:25:38.5317006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5317693Z 			"size": 1214,
2025-07-24T04:25:38.5318369Z 			"digest": "sha256:d30866a5fbe67a4d3ed6411fd4d9040a5715bdb1f83db530ab44c7275bf0cd12"
2025-07-24T04:25:38.5319148Z 		},
2025-07-24T04:25:38.5319460Z 		{
2025-07-24T04:25:38.5319982Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5320652Z 			"size": 485,
2025-07-24T04:25:38.5321335Z 			"digest": "sha256:ef2b71d5d5187f9bc8eb30b15d328bc65465f4b4b4c67061cfd8027aba8f72dd"
2025-07-24T04:25:38.5322113Z 		},
2025-07-24T04:25:38.5322441Z 		{
2025-07-24T04:25:38.5322967Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5323652Z 			"size": 110343867,
2025-07-24T04:25:38.5324352Z 			"digest": "sha256:3a79f1be9328db38db5d9682fef0e9659b2d24e666d5ef2f39f2a233084e548c"
2025-07-24T04:25:38.5325338Z 		},
2025-07-24T04:25:38.5325653Z 		{
2025-07-24T04:25:38.5326174Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5326842Z 			"size": 4164,
2025-07-24T04:25:38.5327494Z 			"digest": "sha256:ab67e3b4c18a18c51c22703a2493006d45815828baf0b1df64b12da7ff0bd2e7"
2025-07-24T04:25:38.5328309Z 		},
2025-07-24T04:25:38.5328673Z 		{
2025-07-24T04:25:38.5329286Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5330080Z 			"size": 1710,
2025-07-24T04:25:38.5330760Z 			"digest": "sha256:035eb636c57bd7202decf2a456e8537bddc1be808adbd80f6dd75e12c13e1be3"
2025-07-24T04:25:38.5331520Z 		},
2025-07-24T04:25:38.5331836Z 		{
2025-07-24T04:25:38.5332364Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5333028Z 			"size": 724,
2025-07-24T04:25:38.5333690Z 			"digest": "sha256:d4ec37220e3a61f253b7bff3b60dce616d51caf1327ea11d322d85cd281dbf07"
2025-07-24T04:25:38.5334446Z 		},
2025-07-24T04:25:38.5334756Z 		{
2025-07-24T04:25:38.5335277Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5335953Z 			"size": 3305033147,
2025-07-24T04:25:38.5336628Z 			"digest": "sha256:9d9b3bb5166b5733af7605b65690330a088bd8636c36433e8e173a6659b16421"
2025-07-24T04:25:38.5337363Z 		},
2025-07-24T04:25:38.5337670Z 		{
2025-07-24T04:25:38.5338197Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5338859Z 			"size": 381,
2025-07-24T04:25:38.5339497Z 			"digest": "sha256:554a1868b180d4e2032cc390ea00a9f7c180d8885392be12550bc3e1c5e1a23c"
2025-07-24T04:25:38.5340234Z 		},
2025-07-24T04:25:38.5340538Z 		{
2025-07-24T04:25:38.5341066Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5341769Z 			"size": 233567,
2025-07-24T04:25:38.5342446Z 			"digest": "sha256:090ade5d5ee9d68cd421dbc7cb6e401709f6c178a61d3f87019b041c6774b2db"
2025-07-24T04:25:38.5343209Z 		},
2025-07-24T04:25:38.5343514Z 		{
2025-07-24T04:25:38.5344030Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5344698Z 			"size": 792,
2025-07-24T04:25:38.5345332Z 			"digest": "sha256:a998e898d9939b4c1d30567427b67cc709d6852ee65ef64fa2d9c318db592126"
2025-07-24T04:25:38.5346198Z 		},
2025-07-24T04:25:38.5346568Z 		{
2025-07-24T04:25:38.5347159Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5348182Z 			"size": 106,
2025-07-24T04:25:38.5348848Z 			"digest": "sha256:cfaabb19fb471ea7d79bd4c76b041e6a316873783814724e35c08c71bf89cd2c"
2025-07-24T04:25:38.5349609Z 		},
2025-07-24T04:25:38.5349921Z 		{
2025-07-24T04:25:38.5350468Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5351602Z 			"size": 1495,
2025-07-24T04:25:38.5352403Z 			"digest": "sha256:53b92662290aaff40a4779c14e8bf37ca7e400c814eadabbdeea922a157f26f0"
2025-07-24T04:25:38.5353201Z 		},
2025-07-24T04:25:38.5353511Z 		{
2025-07-24T04:25:38.5354034Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5354715Z 			"size": 455235035,
2025-07-24T04:25:38.5355396Z 			"digest": "sha256:95b6332c3b55e9d6fe681de21373e3993ddc9b286515039dc2f7ed9914277167"
2025-07-24T04:25:38.5356143Z 		},
2025-07-24T04:25:38.5356455Z 		{
2025-07-24T04:25:38.5356968Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5357653Z 			"size": 163,
2025-07-24T04:25:38.5358320Z 			"digest": "sha256:b67085f0f4fdd797574ea5df363a0cbde9ca856b230bbdd7aae29b90ad5e79cb"
2025-07-24T04:25:38.5359094Z 		},
2025-07-24T04:25:38.5359404Z 		{
2025-07-24T04:25:38.5359922Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5360593Z 			"size": 2438,
2025-07-24T04:25:38.5361261Z 			"digest": "sha256:5e2e975cce85260c1165f6d3d8280f9d678fed13e21fc8b6efba9f1a2bca97bc"
2025-07-24T04:25:38.5362017Z 		},
2025-07-24T04:25:38.5362318Z 		{
2025-07-24T04:25:38.5362834Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5363512Z 			"size": 7164124491,
2025-07-24T04:25:38.5364193Z 			"digest": "sha256:90308017eb7efa2434edc95a07bc92d8710942262bf3d76dda289feca59fb5f5"
2025-07-24T04:25:38.5365101Z 		},
2025-07-24T04:25:38.5365411Z 		{
2025-07-24T04:25:38.5366014Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5366804Z 			"size": 105,
2025-07-24T04:25:38.5367525Z 			"digest": "sha256:7f8c898d8b8417ffa2b0a975a6e9a7ccaee952d8e8acb7e90906c53f9f41a7a6"
2025-07-24T04:25:38.5368288Z 		},
2025-07-24T04:25:38.5368583Z 		{
2025-07-24T04:25:38.5369110Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5369902Z 			"size": 612,
2025-07-24T04:25:38.5370688Z 			"digest": "sha256:5b1565991f5c0f1defeb2448eaac6383107d330f11fa60e3c9f8488752ee12e8"
2025-07-24T04:25:38.5371539Z 		},
2025-07-24T04:25:38.5371848Z 		{
2025-07-24T04:25:38.5372365Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5373042Z 			"size": 677677324,
2025-07-24T04:25:38.5373719Z 			"digest": "sha256:cfa6c02ce350f691ca01e9bbc91554b3448765199582f80d0dc44b8b4d226f46"
2025-07-24T04:25:38.5374467Z 		},
2025-07-24T04:25:38.5374774Z 		{
2025-07-24T04:25:38.5375291Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5375958Z 			"size": 111,
2025-07-24T04:25:38.5376611Z 			"digest": "sha256:4690cd0b661d36beb72956d1933777b5e2eb63ac53f3549cbe117619cbed11ab"
2025-07-24T04:25:38.5377366Z 		},
2025-07-24T04:25:38.5377672Z 		{
2025-07-24T04:25:38.5378192Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5379270Z 			"size": 1557,
2025-07-24T04:25:38.5379984Z 			"digest": "sha256:8dab92a6cabfe1c1cc4cf921541571080c1db7eeaa5d6fb5e1dc421b7627954e"
2025-07-24T04:25:38.5380785Z 		},
2025-07-24T04:25:38.5381097Z 		{
2025-07-24T04:25:38.5381641Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5382312Z 			"size": 107,
2025-07-24T04:25:38.5382974Z 			"digest": "sha256:741b9f0a45dbdd117063541fe5928c8656c711df99ce2f6c93fb837df6c54000"
2025-07-24T04:25:38.5383742Z 		},
2025-07-24T04:25:38.5384051Z 		{
2025-07-24T04:25:38.5384575Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5385243Z 			"size": 166,
2025-07-24T04:25:38.5385968Z 			"digest": "sha256:78a6125437f3d51017ccb8735f1c3fce4ce00aa14f7490ba2d1f3a0254101595"
2025-07-24T04:25:38.5387244Z 		},
2025-07-24T04:25:38.5387556Z 		{
2025-07-24T04:25:38.5388076Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5388741Z 			"size": 2801110,
2025-07-24T04:25:38.5389410Z 			"digest": "sha256:3be44072ab4e7975f53fa3962eb807664f771ebafad2dae86f07404fb56be803"
2025-07-24T04:25:38.5390462Z 		},
2025-07-24T04:25:38.5390787Z 		{
2025-07-24T04:25:38.5391303Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5391963Z 			"size": 107,
2025-07-24T04:25:38.5392618Z 			"digest": "sha256:04044610df175f1bf00fb5ea1215ba9784924b5a20293aafe94a155cfc75d56d"
2025-07-24T04:25:38.5393372Z 		},
2025-07-24T04:25:38.5393676Z 		{
2025-07-24T04:25:38.5394190Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5394852Z 			"size": 802,
2025-07-24T04:25:38.5395499Z 			"digest": "sha256:152aa5a5158e010217f23fd8fbb0d9b0826c39bb13e6d8652c6965cd56b5fc7d"
2025-07-24T04:25:38.5396265Z 		},
2025-07-24T04:25:38.5396570Z 		{
2025-07-24T04:25:38.5397079Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5397752Z 			"size": 26113617,
2025-07-24T04:25:38.5398420Z 			"digest": "sha256:63f73895622ecab7d061d93953419fbdf7c48814c5008105fc0b432177c033a0"
2025-07-24T04:25:38.5399167Z 		},
2025-07-24T04:25:38.5399488Z 		{
2025-07-24T04:25:38.5400012Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5400674Z 			"size": 104,
2025-07-24T04:25:38.5401338Z 			"digest": "sha256:5af6f2f093ea5798a44a5decddf46cb875cbf4b54e5e938d3ad6d5a0d3b61967"
2025-07-24T04:25:38.5402110Z 		},
2025-07-24T04:25:38.5402406Z 		{
2025-07-24T04:25:38.5402919Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5403589Z 			"size": 425,
2025-07-24T04:25:38.5404257Z 			"digest": "sha256:fda40cf9ab8372afeefe16b77d0233d4b63b678cace2059714f7c35d772e1637"
2025-07-24T04:25:38.5405355Z 		},
2025-07-24T04:25:38.5405692Z 		{
2025-07-24T04:25:38.5406211Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5406879Z 			"size": 19308731,
2025-07-24T04:25:38.5428693Z 			"digest": "sha256:5aae7747cc25bf67f6bfe370770b12dccc584ea9afb3f8b825b03b0cb15918b1"
2025-07-24T04:25:38.5429553Z 		},
2025-07-24T04:25:38.5429892Z 		{
2025-07-24T04:25:38.5430523Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5431251Z 			"size": 425,
2025-07-24T04:25:38.5431964Z 			"digest": "sha256:5b62e7dff82033606f1b96c42c49540ebbe46772aeb12467df1e1bb07eb2e175"
2025-07-24T04:25:38.5432751Z 		},
2025-07-24T04:25:38.5433077Z 		{
2025-07-24T04:25:38.5433634Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5434330Z 			"size": 724,
2025-07-24T04:25:38.5435019Z 			"digest": "sha256:d4ec37220e3a61f253b7bff3b60dce616d51caf1327ea11d322d85cd281dbf07"
2025-07-24T04:25:38.5435790Z 		},
2025-07-24T04:25:38.5436107Z 		{
2025-07-24T04:25:38.5436660Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5437344Z 			"size": 141,
2025-07-24T04:25:38.5438024Z 			"digest": "sha256:02460b2a4f6a2d3bc23ff65db1afd3750ee9ae49a35ce9ddb8e74f3c46b5705f"
2025-07-24T04:25:38.5438821Z 		},
2025-07-24T04:25:38.5439130Z 		{
2025-07-24T04:25:38.5439666Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5440331Z 			"size": 136,
2025-07-24T04:25:38.5440995Z 			"digest": "sha256:344aa86a61d68f5f8573f99122b4efe5c555bdceaeb3a1f1b5502715dbaee4c7"
2025-07-24T04:25:38.5441767Z 		},
2025-07-24T04:25:38.5442071Z 		{
2025-07-24T04:25:38.5442597Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5443278Z 			"size": 5098992948,
2025-07-24T04:25:38.5443972Z 			"digest": "sha256:04b592ce3217c257cc0b889e15f3c3731d22baca68e5760ec146ef749d0ff1f1"
2025-07-24T04:25:38.5444950Z 		},
2025-07-24T04:25:38.5445258Z 		{
2025-07-24T04:25:38.5445767Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5447121Z + exit 0
2025-07-24T04:25:38.5447469Z 			"size": 195,
2025-07-24T04:25:38.5448151Z 			"digest": "sha256:e3aedebec93030bc06d19aacab68750c6e085bb1c27dc358d16858848935cf71"
2025-07-24T04:25:38.5449078Z 		},
2025-07-24T04:25:38.5449450Z 		{
2025-07-24T04:25:38.5450495Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5451188Z 			"size": 346,
2025-07-24T04:25:38.5451842Z 			"digest": "sha256:2b5211eebec64099919bea1197c9c43983976b5037fd1ff6236509f49dd4a979"
2025-07-24T04:25:38.5452588Z 		},
2025-07-24T04:25:38.5452899Z 		{
2025-07-24T04:25:38.5453414Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5454078Z 			"size": 88303,
2025-07-24T04:25:38.5454748Z 			"digest": "sha256:88aac01ec1db72a960d19e691f39b315730a97c2330390a8e6d0cbd2f80d3ca0"
2025-07-24T04:25:38.5455508Z 		},
2025-07-24T04:25:38.5455814Z 		{
2025-07-24T04:25:38.5456327Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5457002Z 			"size": 106,
2025-07-24T04:25:38.5457653Z 			"digest": "sha256:f154dd81b32fa92d4faa8040506993076e2912a919c2afe6dded50740d86d5be"
2025-07-24T04:25:38.5458406Z 		},
2025-07-24T04:25:38.5458713Z 		{
2025-07-24T04:25:38.5459221Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5459898Z 			"size": 1666,
2025-07-24T04:25:38.5460564Z 			"digest": "sha256:089f2260c49023e792a2bee3867520ef852c8a55f37ce04ea81b07b7dfc2dae7"
2025-07-24T04:25:38.5461314Z 		},
2025-07-24T04:25:38.5461623Z 		{
2025-07-24T04:25:38.5462141Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5462823Z 			"size": 724,
2025-07-24T04:25:38.5463488Z 			"digest": "sha256:d4ec37220e3a61f253b7bff3b60dce616d51caf1327ea11d322d85cd281dbf07"
2025-07-24T04:25:38.5464258Z 		},
2025-07-24T04:25:38.5464574Z 		{
2025-07-24T04:25:38.5465081Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5465792Z 			"size": 137,
2025-07-24T04:25:38.5466576Z 			"digest": "sha256:ca8b54819584ab34ad98cd66be3fceb44c157dc0b7f140070c4584816919bbe5"
2025-07-24T04:25:38.5467461Z 		},
2025-07-24T04:25:38.5467774Z 		{
2025-07-24T04:25:38.5468306Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5468985Z 			"size": 120,
2025-07-24T04:25:38.5469663Z 			"digest": "sha256:76870a4c3f8fd6d57b2ec17e72b7a85a29d885057ef18bf331afb97a748d6d40"
2025-07-24T04:25:38.5470431Z 		},
2025-07-24T04:25:38.5470741Z 		{
2025-07-24T04:25:38.5471270Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5471945Z 			"size": 5356440017,
2025-07-24T04:25:38.5472644Z 			"digest": "sha256:f4a9f30fcdabff1a1167cb66c7560f21b3668899a96b35b9fc4fde046e4ef265"
2025-07-24T04:25:38.5473418Z 		},
2025-07-24T04:25:38.5473730Z 		{
2025-07-24T04:25:38.5474276Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5474970Z 			"size": 175,
2025-07-24T04:25:38.5475657Z 			"digest": "sha256:110b5a00f043de5141a35fc11bdd3d6541ec027ba726178f89e3bd5b44f18d8b"
2025-07-24T04:25:38.5476429Z 		},
2025-07-24T04:25:38.5476751Z 		{
2025-07-24T04:25:38.5477290Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5477977Z 			"size": 1897,
2025-07-24T04:25:38.5478672Z 			"digest": "sha256:83772e66e52fac49c784a3a4dee3cad95c1653ee093350b48399492da564b335"
2025-07-24T04:25:38.5479436Z 		},
2025-07-24T04:25:38.5479751Z 		{
2025-07-24T04:25:38.5480281Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5480955Z 			"size": 197493594,
2025-07-24T04:25:38.5481647Z 			"digest": "sha256:eee6d9689c2d5b11deada1d789902ffa6c32c42bd82c1c99df6264532d0af8d8"
2025-07-24T04:25:38.5482424Z 		},
2025-07-24T04:25:38.5482761Z 		{
2025-07-24T04:25:38.5483306Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5483989Z 			"size": 303,
2025-07-24T04:25:38.5484770Z 			"digest": "sha256:40f61293f85973eb3d6df45c40671ffd92439d1bc29f39c887e4342e5bb05461"
2025-07-24T04:25:38.5485863Z 		},
2025-07-24T04:25:38.5486179Z 		{
2025-07-24T04:25:38.5486708Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5487375Z 			"size": 32,
2025-07-24T04:25:38.5488338Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-07-24T04:25:38.5489255Z 		},
2025-07-24T04:25:38.5489628Z 		{
2025-07-24T04:25:38.5490247Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5490955Z 			"size": 108,
2025-07-24T04:25:38.5491620Z 			"digest": "sha256:dfc71386efac7d21b798410f8f2ead070171f7f687d570ebf35eed00ecaf48c8"
2025-07-24T04:25:38.5492386Z 		},
2025-07-24T04:25:38.5492698Z 		{
2025-07-24T04:25:38.5493219Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-07-24T04:25:38.5493880Z 			"size": 54145699,
2025-07-24T04:25:38.5494583Z 			"digest": "sha256:f4813cdb60f2c95fd11144ffcc01f5908a02cabb3606fc1f84f9b6bdf44177e0"
2025-07-24T04:25:38.5495376Z 		}
2025-07-24T04:25:38.5495682Z 	]
2025-07-24T04:25:38.5495995Z }
2025-07-24T04:25:38.5543672Z ##[group]Run set -eux
2025-07-24T04:25:38.5544079Z set -eux
2025-07-24T04:25:38.5545399Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-07-24T04:25:38.5592216Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:38.5592785Z env:
2025-07-24T04:25:38.5593113Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:38.5593758Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:38.5594705Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:38.5595581Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:38.5597052Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:38.5598379Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:38.5598803Z   AWS_REGION: us-east-1
2025-07-24T04:25:38.5599314Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:38.5599895Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:38.5609619Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:38.5610080Z ##[endgroup]
2025-07-24T04:25:38.5705847Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-07-24T04:25:38.5709417Z + jq --raw-output .SecretString
2025-07-24T04:25:38.5712479Z + jq -r .docker_hub_readonly_token
2025-07-24T04:25:38.5716969Z + docker login --username pytorchbot --password-stdin
2025-07-24T04:25:39.3104488Z 
2025-07-24T04:25:39.3107246Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2025-07-24T04:25:39.4034116Z Error: Cannot perform an interactive login from a non TTY device
2025-07-24T04:25:39.4105190Z ##[error]Process completed with exit code 1.
2025-07-24T04:25:39.4270972Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-07-24T04:25:39.4271639Z with:
2025-07-24T04:25:39.4272674Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:39.4273962Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:39.4274530Z env:
2025-07-24T04:25:39.4274851Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:39.4275493Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:39.4276427Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:39.4277706Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:39.4279298Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:39.4280637Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:39.4281057Z   AWS_REGION: us-east-1
2025-07-24T04:25:39.4281540Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:39.4282126Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:39.4292249Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:39.4292653Z ##[endgroup]
2025-07-24T04:25:39.4318453Z ##[group]Run set -x
2025-07-24T04:25:39.4318882Z set -x
2025-07-24T04:25:39.4319220Z set +e
2025-07-24T04:25:39.4319539Z 
2025-07-24T04:25:39.4319848Z login() {
2025-07-24T04:25:39.4320574Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-07-24T04:25:39.4321386Z }
2025-07-24T04:25:39.4321693Z 
2025-07-24T04:25:39.4321998Z retry () {
2025-07-24T04:25:39.4322399Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-07-24T04:25:39.4322855Z }
2025-07-24T04:25:39.4323156Z 
2025-07-24T04:25:39.4323501Z retry login "${DOCKER_REGISTRY}"
2025-07-24T04:25:39.4323953Z 
2025-07-24T04:25:39.4324820Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-07-24T04:25:39.4325817Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-07-24T04:25:39.4326373Z 
2025-07-24T04:25:39.4326689Z set -e
2025-07-24T04:25:39.4327196Z # ignore output since only exit code is used for conditional
2025-07-24T04:25:39.4327924Z # only pull docker image if it's not available locally
2025-07-24T04:25:39.4328736Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-07-24T04:25:39.4329494Z   retry docker pull "${DOCKER_IMAGE}"
2025-07-24T04:25:39.4329956Z fi
2025-07-24T04:25:39.4372296Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:39.4372861Z env:
2025-07-24T04:25:39.4373197Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:39.4373838Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:39.4374777Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:39.4375662Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:39.4377173Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:39.4378504Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:39.4378938Z   AWS_REGION: us-east-1
2025-07-24T04:25:39.4379414Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:39.4380007Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:39.4389481Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:39.4390996Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:39.4392329Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:39.4392913Z ##[endgroup]
2025-07-24T04:25:39.4462211Z + set +e
2025-07-24T04:25:39.4462906Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:39.4463821Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:39.4472220Z + aws ecr get-login-password --region us-east-1
2025-07-24T04:25:39.4475219Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:25:40.5162360Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-07-24T04:25:40.5164159Z Configure a credential helper to remove this warning. See
2025-07-24T04:25:40.5165322Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-07-24T04:25:40.5165991Z 
2025-07-24T04:25:40.5166834Z Login Succeeded
2025-07-24T04:25:40.5241099Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:40.5243179Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-07-24T04:25:41.2219678Z + IMAGE_SIZE=21755.0470495224
2025-07-24T04:25:41.2220378Z + echo 'Compressed size of image in MB: 21755.0470495224'
2025-07-24T04:25:41.2221123Z Compressed size of image in MB: 21755.0470495224
2025-07-24T04:25:41.2225931Z + set -e
2025-07-24T04:25:41.2227621Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:25:41.2690503Z Prepare all required actions
2025-07-24T04:25:41.2746942Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-07-24T04:25:41.2747518Z with:
2025-07-24T04:25:41.2748260Z   github-token: ***
2025-07-24T04:25:41.2748682Z env:
2025-07-24T04:25:41.2749048Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:41.2749780Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:41.2750867Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:41.2751920Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:41.2753638Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:41.2755161Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:41.2755639Z   AWS_REGION: us-east-1
2025-07-24T04:25:41.2756224Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:41.2756887Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:41.2767722Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:41.2768173Z ##[endgroup]
2025-07-24T04:25:41.2796866Z ##[group]Run set -eux
2025-07-24T04:25:41.2797373Z set -eux
2025-07-24T04:25:41.2798257Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-07-24T04:25:41.2851647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:25:41.2852219Z env:
2025-07-24T04:25:41.2852556Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:41.2853193Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:41.2854137Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:41.2855047Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:41.2856577Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:41.2857939Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:41.2858371Z   AWS_REGION: us-east-1
2025-07-24T04:25:41.2858860Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:41.2859531Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:41.2869002Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:41.2869577Z   GITHUB_TOKEN: ***
2025-07-24T04:25:41.2869938Z ##[endgroup]
2025-07-24T04:25:41.2954648Z + python3 .github/scripts/get_workflow_job_id.py 16487210679 gpu9455
2025-07-24T04:25:42.2600126Z Setting output job-id=46615439125
2025-07-24T04:25:42.2601133Z Setting output job-name=linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:25:42.2874185Z Prepare all required actions
2025-07-24T04:25:42.2874974Z Getting action download info
2025-07-24T04:25:42.4600751Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-07-24T04:25:43.3232610Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-07-24T04:25:43.9398292Z ##[group]Run ./.github/actions/download-build-artifacts
2025-07-24T04:25:43.9398563Z with:
2025-07-24T04:25:43.9398745Z   name: linux-jammy-rocm-py3.10
2025-07-24T04:25:43.9398974Z   s3-bucket: gha-artifacts
2025-07-24T04:25:43.9399173Z env:
2025-07-24T04:25:43.9399344Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:43.9399655Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:43.9400118Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:43.9400591Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:43.9401309Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:43.9401964Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:43.9402176Z   AWS_REGION: us-east-1
2025-07-24T04:25:43.9402437Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:43.9402716Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:43.9407565Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:43.9407835Z ##[endgroup]
2025-07-24T04:25:43.9433799Z ##[group]Run seemethere/download-artifact-s3@v4
2025-07-24T04:25:43.9434052Z with:
2025-07-24T04:25:43.9434231Z   name: linux-jammy-rocm-py3.10
2025-07-24T04:25:43.9434475Z   s3-bucket: gha-artifacts
2025-07-24T04:25:43.9434691Z   region: us-east-1
2025-07-24T04:25:43.9434855Z env:
2025-07-24T04:25:43.9435024Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:25:43.9435352Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:25:43.9435822Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:25:43.9436245Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:25:43.9436955Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:25:43.9437591Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:25:43.9437802Z   AWS_REGION: us-east-1
2025-07-24T04:25:43.9438036Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:25:43.9438320Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:25:43.9442639Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:25:43.9442835Z ##[endgroup]
2025-07-24T04:25:44.4224020Z (node:1577651) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-07-24T04:25:44.4224972Z 
2025-07-24T04:25:44.4225394Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-07-24T04:25:44.4226364Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-07-24T04:25:44.4227388Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-07-24T04:25:44.5544072Z Found 1 objects with prefix pytorch/pytorch/16487210679/linux-jammy-rocm-py3.10/
2025-07-24T04:25:44.5545462Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-07-24T04:26:07.4780358Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-07-24T04:26:07.4792245Z Artifact download has finished successfully
2025-07-24T04:26:07.5376541Z ##[group]Run unzip -o artifacts.zip
2025-07-24T04:26:07.5377139Z unzip -o artifacts.zip
2025-07-24T04:26:07.5430965Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:07.5431618Z env:
2025-07-24T04:26:07.5432000Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:07.5433227Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:07.5434328Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:07.5435332Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:07.5437452Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:07.5438946Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:07.5439432Z   AWS_REGION: us-east-1
2025-07-24T04:26:07.5439998Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:07.5440663Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:07.5451688Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:07.5452152Z ##[endgroup]
2025-07-24T04:26:07.5581537Z Archive:  artifacts.zip
2025-07-24T04:26:07.5583054Z    creating: dist/
2025-07-24T04:26:12.9407186Z   inflating: dist/torch-2.9.0a0+git27c8ef1-cp310-cp310-linux_x86_64.whl  
2025-07-24T04:26:12.9534879Z   inflating: dist/.ninja_log         
2025-07-24T04:26:12.9539717Z    creating: build/custom_test_artifacts/
2025-07-24T04:26:12.9540628Z    creating: build/custom_test_artifacts/custom-op-build/
2025-07-24T04:26:12.9541572Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-07-24T04:26:12.9542619Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-07-24T04:26:12.9543798Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-24T04:26:12.9544938Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/
2025-07-24T04:26:12.9546051Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-24T04:26:12.9547229Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-24T04:26:12.9548416Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-24T04:26:12.9549745Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-24T04:26:12.9551081Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-24T04:26:12.9552339Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-24T04:26:12.9553559Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-24T04:26:12.9554725Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-24T04:26:12.9556086Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-24T04:26:12.9557493Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-24T04:26:12.9558770Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-24T04:26:12.9560174Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-24T04:26:12.9561660Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-24T04:26:12.9562941Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-07-24T04:26:12.9563955Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-07-24T04:26:12.9565234Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-07-24T04:26:12.9566346Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-07-24T04:26:12.9567551Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-07-24T04:26:12.9568947Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-07-24T04:26:12.9571025Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-07-24T04:26:12.9572309Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-07-24T04:26:12.9573607Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-07-24T04:26:12.9575318Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-07-24T04:26:12.9576619Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-07-24T04:26:12.9577906Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-07-24T04:26:12.9579180Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-07-24T04:26:12.9580660Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-07-24T04:26:12.9760319Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-07-24T04:26:12.9761686Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-07-24T04:26:12.9763073Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-07-24T04:26:12.9764809Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-07-24T04:26:12.9766246Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-07-24T04:26:12.9767586Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-07-24T04:26:12.9768983Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-07-24T04:26:12.9770361Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-07-24T04:26:12.9771735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-07-24T04:26:12.9773084Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-07-24T04:26:12.9774441Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-07-24T04:26:12.9785260Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-07-24T04:26:12.9861080Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-07-24T04:26:12.9862726Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-24T04:26:12.9864094Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-07-24T04:26:12.9865332Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-07-24T04:26:12.9866491Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-07-24T04:26:12.9867606Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-07-24T04:26:12.9868826Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json  
2025-07-24T04:26:12.9870031Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc  
2025-07-24T04:26:12.9871143Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-07-24T04:26:12.9872167Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-07-24T04:26:12.9873093Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-07-24T04:26:12.9874042Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-07-24T04:26:13.0018145Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-07-24T04:26:13.0069473Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-07-24T04:26:13.0072235Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-07-24T04:26:13.0073130Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-07-24T04:26:13.0074536Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-07-24T04:26:13.0075691Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-24T04:26:13.0076798Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/
2025-07-24T04:26:13.0077889Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-24T04:26:13.0079058Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-24T04:26:13.0080204Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-24T04:26:13.0081524Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-24T04:26:13.0082867Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-24T04:26:13.0084105Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-24T04:26:13.0085472Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-24T04:26:13.0086623Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-24T04:26:13.0087985Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-24T04:26:13.0089376Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-24T04:26:13.0090657Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-24T04:26:13.0092051Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-24T04:26:13.0093560Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-24T04:26:13.0094828Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-07-24T04:26:13.0095852Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-07-24T04:26:13.0096904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-07-24T04:26:13.0098029Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-07-24T04:26:13.0099295Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-07-24T04:26:13.0100750Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-07-24T04:26:13.0102126Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-07-24T04:26:13.0103409Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-07-24T04:26:13.0104726Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-07-24T04:26:13.0106091Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-07-24T04:26:13.0107425Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-07-24T04:26:13.0108744Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-07-24T04:26:13.0110063Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-07-24T04:26:13.0113268Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-07-24T04:26:13.0173106Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-07-24T04:26:13.0173956Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-24T04:26:13.0174671Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-07-24T04:26:13.0175564Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-07-24T04:26:13.0176239Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-07-24T04:26:13.0176843Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-07-24T04:26:13.0177419Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json  
2025-07-24T04:26:13.0178001Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc  
2025-07-24T04:26:13.0178553Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-07-24T04:26:13.0179092Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-07-24T04:26:13.0179629Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-07-24T04:26:13.0180102Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-07-24T04:26:13.0214938Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-07-24T04:26:13.0215416Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-07-24T04:26:13.0215882Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-07-24T04:26:13.0216434Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-07-24T04:26:13.0219827Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-07-24T04:26:13.0220442Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/
2025-07-24T04:26:13.0221067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-07-24T04:26:13.0221736Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/
2025-07-24T04:26:13.0222374Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-07-24T04:26:13.0223482Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-07-24T04:26:13.0224734Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-07-24T04:26:13.0225431Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-07-24T04:26:13.0226102Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-07-24T04:26:13.0226733Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-07-24T04:26:13.0228431Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-07-24T04:26:13.0229594Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-07-24T04:26:13.0230446Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-07-24T04:26:13.0231855Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-07-24T04:26:13.0233098Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-07-24T04:26:13.0233805Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-07-24T04:26:13.0234376Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-07-24T04:26:13.0234961Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-07-24T04:26:13.0235767Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-07-24T04:26:13.0236480Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-07-24T04:26:13.0237272Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-07-24T04:26:13.0238704Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-07-24T04:26:13.0239423Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-07-24T04:26:13.0240172Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-07-24T04:26:13.0240923Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-07-24T04:26:13.0241665Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-07-24T04:26:13.0242418Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-07-24T04:26:13.0243147Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-07-24T04:26:13.0243944Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-07-24T04:26:13.0355384Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-07-24T04:26:13.0356142Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-07-24T04:26:13.0356893Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-07-24T04:26:13.0357729Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-07-24T04:26:13.0358545Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-07-24T04:26:13.0359307Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-07-24T04:26:13.0360087Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-07-24T04:26:13.0360871Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-07-24T04:26:13.0361655Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-07-24T04:26:13.0362439Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-07-24T04:26:13.0363218Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-07-24T04:26:13.0380945Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-07-24T04:26:13.0431422Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-07-24T04:26:13.0432325Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-07-24T04:26:13.0433087Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-07-24T04:26:13.0433756Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-07-24T04:26:13.0434382Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-07-24T04:26:13.0434970Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-07-24T04:26:13.0435601Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json  
2025-07-24T04:26:13.0436475Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc  
2025-07-24T04:26:13.0437088Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-07-24T04:26:13.0437653Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-07-24T04:26:13.0438320Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-07-24T04:26:13.0438848Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-07-24T04:26:13.0532428Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-07-24T04:26:13.0569129Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-07-24T04:26:13.0570077Z    creating: build/lib/
2025-07-24T04:26:13.0644993Z   inflating: build/lib/libprotobuf-lite.a  
2025-07-24T04:26:13.1049817Z   inflating: build/lib/libprotobuf.a  
2025-07-24T04:26:13.1501303Z   inflating: build/lib/libprotoc.a   
2025-07-24T04:26:13.1509802Z   inflating: build/lib/libpthreadpool.a  
2025-07-24T04:26:13.1517178Z   inflating: build/lib/libcpuinfo.a  
2025-07-24T04:26:13.1524290Z   inflating: build/lib/libcpuinfo_internals.a  
2025-07-24T04:26:13.1525166Z   inflating: build/lib/libclog.a     
2025-07-24T04:26:13.1542432Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-07-24T04:26:13.1543661Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-07-24T04:26:13.1560385Z   inflating: build/lib/libnnpack.a   
2025-07-24T04:26:13.1729854Z   inflating: build/lib/libmicrokernels-prod.a  
2025-07-24T04:26:13.2523431Z   inflating: build/lib/libmicrokernels-all.a  
2025-07-24T04:26:13.2587254Z   inflating: build/lib/libgtest.a    
2025-07-24T04:26:13.2602804Z   inflating: build/lib/libgmock.a    
2025-07-24T04:26:13.2603620Z   inflating: build/lib/libgtest_main.a  
2025-07-24T04:26:13.2685264Z   inflating: build/lib/libXNNPACK.a  
2025-07-24T04:26:13.2686255Z   inflating: build/lib/libgmock_main.a  
2025-07-24T04:26:13.2754903Z   inflating: build/lib/libbenchmark.a  
2025-07-24T04:26:13.2755651Z   inflating: build/lib/libbenchmark_main.a  
2025-07-24T04:26:13.2762685Z   inflating: build/lib/libittnotify.a  
2025-07-24T04:26:13.2763396Z   inflating: build/lib/libjitprofiling.a  
2025-07-24T04:26:13.2821823Z   inflating: build/lib/libasmjit.a   
2025-07-24T04:26:13.3926734Z   inflating: build/lib/libfbgemm.a   
2025-07-24T04:26:13.3950989Z   inflating: build/lib/libtensorpipe_uv.a  
2025-07-24T04:26:13.4460093Z   inflating: build/lib/libtensorpipe.a  
2025-07-24T04:26:13.4569908Z   inflating: build/lib/libgloo.a     
2025-07-24T04:26:13.4613126Z   inflating: build/lib/libonnx_proto.a  
2025-07-24T04:26:13.5015404Z   inflating: build/lib/libgloo_hip.a  
2025-07-24T04:26:13.5674504Z   inflating: build/lib/libonnx.a     
2025-07-24T04:26:14.5147861Z   inflating: build/lib/libdnnl.a     
2025-07-24T04:26:14.5164401Z   inflating: build/lib/libfmt.a      
2025-07-24T04:26:14.5427993Z   inflating: build/lib/libkineto.a   
2025-07-24T04:26:14.5524089Z   inflating: build/lib/libc10.so     
2025-07-24T04:26:14.5525009Z   inflating: build/lib/libtorch_global_deps.so  
2025-07-24T04:26:14.5571654Z   inflating: build/lib/libc10_hip.so  
2025-07-24T04:26:14.5572364Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-07-24T04:26:17.5998434Z   inflating: build/lib/libtorch_cpu.so  
2025-07-24T04:26:17.6000894Z   inflating: build/lib/libshm.so     
2025-07-24T04:26:18.3755290Z   inflating: build/lib/libtorch_hip.so  
2025-07-24T04:26:18.3756026Z   inflating: build/lib/libtorch.so   
2025-07-24T04:26:18.3772907Z   inflating: build/lib/libjitbackend_test.so  
2025-07-24T04:26:18.3794417Z   inflating: build/lib/libbackend_with_compiler.so  
2025-07-24T04:26:18.3858151Z   inflating: build/lib/libtorchbind_test.so  
2025-07-24T04:26:18.3881909Z   inflating: build/lib/libaoti_custom_ops.so  
2025-07-24T04:26:18.5727948Z   inflating: build/lib/libtorch_python.so  
2025-07-24T04:26:18.5757702Z   inflating: build/lib/libnnapi_backend.so  
2025-07-24T04:26:18.5758426Z    creating: build/bin/
2025-07-24T04:26:18.5759634Z    creating: build/bin/CMakeFiles/
2025-07-24T04:26:18.5760269Z   inflating: build/bin/cmake_install.cmake  
2025-07-24T04:26:18.5760926Z   inflating: build/bin/CTestTestfile.cmake  
2025-07-24T04:26:18.6168238Z   inflating: build/bin/protoc-3.13.0.0  
2025-07-24T04:26:18.6576925Z   inflating: build/bin/protoc        
2025-07-24T04:26:18.6626431Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-07-24T04:26:18.6677475Z   inflating: build/bin/c10_DeviceGuard_test  
2025-07-24T04:26:18.6728625Z   inflating: build/bin/c10_Device_test  
2025-07-24T04:26:18.6787243Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-07-24T04:26:18.6840433Z   inflating: build/bin/c10_Scalar_test  
2025-07-24T04:26:18.6889409Z   inflating: build/bin/c10_StreamGuard_test  
2025-07-24T04:26:18.6939954Z   inflating: build/bin/c10_SymInt_test  
2025-07-24T04:26:18.6993838Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-07-24T04:26:18.7048913Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-07-24T04:26:18.7097665Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-07-24T04:26:18.7152818Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-07-24T04:26:18.7221423Z   inflating: build/bin/c10_cow_test  
2025-07-24T04:26:18.7270701Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-07-24T04:26:18.7323259Z   inflating: build/bin/c10_Bitset_test  
2025-07-24T04:26:18.7372488Z   inflating: build/bin/c10_ArrayRef_test  
2025-07-24T04:26:18.7428894Z   inflating: build/bin/c10_Enumerate_test  
2025-07-24T04:26:18.7481252Z   inflating: build/bin/c10_IntrusiveList_test  
2025-07-24T04:26:18.7536358Z   inflating: build/bin/c10_Metaprogramming_test  
2025-07-24T04:26:18.7586241Z   inflating: build/bin/c10_Half_test  
2025-07-24T04:26:18.7641802Z   inflating: build/bin/c10_LeftRight_test  
2025-07-24T04:26:18.7694435Z   inflating: build/bin/c10_NetworkFlow_test  
2025-07-24T04:26:18.7743854Z   inflating: build/bin/c10_Semaphore_test  
2025-07-24T04:26:18.7793634Z   inflating: build/bin/c10_Synchronized_test  
2025-07-24T04:26:18.7848760Z   inflating: build/bin/c10_ThreadLocal_test  
2025-07-24T04:26:18.7900098Z   inflating: build/bin/c10_TypeIndex_test  
2025-07-24T04:26:18.7949109Z   inflating: build/bin/c10_TypeTraits_test  
2025-07-24T04:26:18.7999562Z   inflating: build/bin/c10_TypeList_test  
2025-07-24T04:26:18.8051148Z   inflating: build/bin/c10_accumulate_test  
2025-07-24T04:26:18.8106047Z   inflating: build/bin/c10_bfloat16_test  
2025-07-24T04:26:18.8156302Z   inflating: build/bin/c10_bit_cast_test  
2025-07-24T04:26:18.8212324Z   inflating: build/bin/c10_complex_math_test  
2025-07-24T04:26:18.8261626Z   inflating: build/bin/c10_error_test  
2025-07-24T04:26:18.8316156Z   inflating: build/bin/c10_complex_test  
2025-07-24T04:26:18.8368160Z   inflating: build/bin/c10_exception_test  
2025-07-24T04:26:18.8417988Z   inflating: build/bin/c10_flags_test  
2025-07-24T04:26:18.8468109Z   inflating: build/bin/c10_generic_math_test  
2025-07-24T04:26:18.8518941Z   inflating: build/bin/c10_irange_test  
2025-07-24T04:26:18.8571790Z   inflating: build/bin/c10_lazy_test  
2025-07-24T04:26:18.8730601Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-07-24T04:26:18.8786615Z   inflating: build/bin/c10_logging_test  
2025-07-24T04:26:18.8860550Z   inflating: build/bin/c10_optional_test  
2025-07-24T04:26:18.8921269Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-07-24T04:26:18.8974319Z   inflating: build/bin/c10_registry_test  
2025-07-24T04:26:18.9025277Z   inflating: build/bin/c10_ssize_test  
2025-07-24T04:26:18.9175494Z   inflating: build/bin/c10_small_vector_test  
2025-07-24T04:26:18.9230944Z   inflating: build/bin/c10_string_util_test  
2025-07-24T04:26:18.9281020Z   inflating: build/bin/c10_tempfile_test  
2025-07-24T04:26:18.9329639Z   inflating: build/bin/c10_string_view_test  
2025-07-24T04:26:18.9373084Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-07-24T04:26:18.9428500Z   inflating: build/bin/c10_typeid_test  
2025-07-24T04:26:18.9478482Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-07-24T04:26:18.9526574Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-07-24T04:26:18.9575708Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-07-24T04:26:18.9624130Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-07-24T04:26:18.9673180Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-07-24T04:26:18.9721640Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-07-24T04:26:18.9770732Z   inflating: build/bin/c10_hip_HIPTest  
2025-07-24T04:26:18.9819578Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-07-24T04:26:19.0876198Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-07-24T04:26:19.1879688Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-07-24T04:26:19.2920359Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-07-24T04:26:19.2972605Z   inflating: build/bin/BackoffTest   
2025-07-24T04:26:19.3025427Z   inflating: build/bin/FileStoreTest  
2025-07-24T04:26:19.3081431Z   inflating: build/bin/TCPStoreTest  
2025-07-24T04:26:19.3134939Z   inflating: build/bin/HashStoreTest  
2025-07-24T04:26:19.3199546Z   inflating: build/bin/ProcessGroupGlooTest  
2025-07-24T04:26:19.3201235Z   inflating: build/bin/example_allreduce  
2025-07-24T04:26:19.3205768Z   inflating: build/bin/torch_shm_manager  
2025-07-24T04:26:19.3258622Z   inflating: build/bin/static_runtime_bench  
2025-07-24T04:26:19.3501521Z   inflating: build/bin/static_runtime_test  
2025-07-24T04:26:19.3574094Z   inflating: build/bin/Dict_test     
2025-07-24T04:26:19.3625969Z   inflating: build/bin/Dimname_test  
2025-07-24T04:26:19.3690423Z   inflating: build/bin/MaybeOwned_test  
2025-07-24T04:26:19.3746585Z   inflating: build/bin/NamedTensor_test  
2025-07-24T04:26:19.3804601Z   inflating: build/bin/apply_utils_test  
2025-07-24T04:26:19.3862508Z   inflating: build/bin/atest         
2025-07-24T04:26:19.3925387Z   inflating: build/bin/basic         
2025-07-24T04:26:19.3979836Z   inflating: build/bin/broadcast_test  
2025-07-24T04:26:19.4030525Z   inflating: build/bin/cpu_allocator_test  
2025-07-24T04:26:19.4087641Z   inflating: build/bin/cpu_generator_test  
2025-07-24T04:26:19.4140179Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-07-24T04:26:19.4229955Z   inflating: build/bin/cpu_rng_test  
2025-07-24T04:26:19.4280487Z   inflating: build/bin/dlconvertor_test  
2025-07-24T04:26:19.4337234Z   inflating: build/bin/extension_backend_test  
2025-07-24T04:26:19.4391947Z   inflating: build/bin/half_test     
2025-07-24T04:26:19.4484595Z   inflating: build/bin/ivalue_test   
2025-07-24T04:26:19.4533845Z   inflating: build/bin/lazy_tensor_test  
2025-07-24T04:26:19.4587031Z   inflating: build/bin/math_kernel_test  
2025-07-24T04:26:19.4639894Z   inflating: build/bin/memory_format_test  
2025-07-24T04:26:19.4693124Z   inflating: build/bin/memory_overlapping_test  
2025-07-24T04:26:19.4745635Z   inflating: build/bin/mobile_memory_cleanup  
2025-07-24T04:26:19.4796364Z   inflating: build/bin/operator_name_test  
2025-07-24T04:26:19.4851658Z   inflating: build/bin/native_test   
2025-07-24T04:26:19.4902197Z   inflating: build/bin/operators_test  
2025-07-24T04:26:19.4953785Z   inflating: build/bin/packedtensoraccessor_test  
2025-07-24T04:26:19.5010520Z   inflating: build/bin/quantized_test  
2025-07-24T04:26:19.5076040Z   inflating: build/bin/pow_test      
2025-07-24T04:26:19.5125929Z   inflating: build/bin/reduce_ops_test  
2025-07-24T04:26:19.5176484Z   inflating: build/bin/reportMemoryUsage_test  
2025-07-24T04:26:19.5227486Z   inflating: build/bin/StorageUtils_test  
2025-07-24T04:26:19.5282935Z   inflating: build/bin/scalar_tensor_test  
2025-07-24T04:26:19.5340954Z   inflating: build/bin/scalar_test   
2025-07-24T04:26:19.5392609Z   inflating: build/bin/stride_properties_test  
2025-07-24T04:26:19.5470215Z   inflating: build/bin/tensor_iterator_test  
2025-07-24T04:26:19.5525206Z   inflating: build/bin/type_ptr_test  
2025-07-24T04:26:19.5526427Z   inflating: build/bin/thread_init_test  
2025-07-24T04:26:19.5581096Z   inflating: build/bin/test_parallel  
2025-07-24T04:26:19.5639486Z   inflating: build/bin/type_test     
2025-07-24T04:26:19.5692000Z   inflating: build/bin/undefined_tensor_test  
2025-07-24T04:26:19.5741244Z   inflating: build/bin/verify_api_visibility  
2025-07-24T04:26:19.5809687Z   inflating: build/bin/legacy_vmap_test  
2025-07-24T04:26:19.5860168Z   inflating: build/bin/weakref_test  
2025-07-24T04:26:19.5918957Z   inflating: build/bin/IListRef_test  
2025-07-24T04:26:19.5970142Z   inflating: build/bin/wrapdim_test  
2025-07-24T04:26:19.6021473Z   inflating: build/bin/xla_tensor_test  
2025-07-24T04:26:19.6124925Z   inflating: build/bin/List_test     
2025-07-24T04:26:19.6240318Z   inflating: build/bin/kernel_function_legacy_test  
2025-07-24T04:26:19.6361231Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-07-24T04:26:19.6453482Z   inflating: build/bin/kernel_function_test  
2025-07-24T04:26:19.6518128Z   inflating: build/bin/KernelFunction_test  
2025-07-24T04:26:19.6618036Z   inflating: build/bin/kernel_lambda_test  
2025-07-24T04:26:19.6677347Z   inflating: build/bin/kernel_stackbased_test  
2025-07-24T04:26:19.6769113Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-07-24T04:26:19.6820095Z   inflating: build/bin/CppSignature_test  
2025-07-24T04:26:19.6868742Z   inflating: build/bin/op_allowlist_test  
2025-07-24T04:26:19.7165220Z   inflating: build/bin/op_registration_test  
2025-07-24T04:26:19.7230521Z   inflating: build/bin/inline_container_test  
2025-07-24T04:26:19.7285274Z   inflating: build/bin/backend_fallback_test  
2025-07-24T04:26:19.7333963Z   inflating: build/bin/hip_complex_math_test  
2025-07-24T04:26:19.7382880Z   inflating: build/bin/hip_complex_test  
2025-07-24T04:26:19.7435276Z   inflating: build/bin/hip_apply_test  
2025-07-24T04:26:19.7484237Z   inflating: build/bin/hip_distributions_test  
2025-07-24T04:26:19.7532869Z   inflating: build/bin/hip_generator_test  
2025-07-24T04:26:19.7581620Z   inflating: build/bin/hip_half_test  
2025-07-24T04:26:19.7630918Z   inflating: build/bin/hip_integer_divider_test  
2025-07-24T04:26:19.7680012Z   inflating: build/bin/hip_optional_test  
2025-07-24T04:26:19.7728740Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-07-24T04:26:19.7777683Z   inflating: build/bin/hip_vectorized_test  
2025-07-24T04:26:19.7828885Z   inflating: build/bin/hip_dlconvertor_test  
2025-07-24T04:26:19.8845482Z   inflating: build/bin/test_jit      
2025-07-24T04:26:19.9105323Z   inflating: build/bin/test_nativert  
2025-07-24T04:26:19.9834692Z   inflating: build/bin/test_tensorexpr  
2025-07-24T04:26:19.9846321Z   inflating: build/bin/tutorial_tensorexpr  
2025-07-24T04:26:19.9900625Z   inflating: build/bin/test_dist_autograd  
2025-07-24T04:26:19.9967034Z   inflating: build/bin/test_cpp_rpc  
2025-07-24T04:26:19.9968285Z   inflating: build/bin/parallel_benchmark  
2025-07-24T04:26:20.1040159Z   inflating: build/bin/test_api      
2025-07-24T04:26:20.1106339Z   inflating: build/bin/test_mobile_nnc  
2025-07-24T04:26:20.1114292Z   inflating: build/bin/aot_model_compiler_test  
2025-07-24T04:26:20.1441891Z   inflating: build/bin/test_lazy     
2025-07-24T04:26:20.1442569Z    creating: .additional_ci_files/
2025-07-24T04:26:20.1589298Z   inflating: .additional_ci_files/test-times.json  
2025-07-24T04:26:20.2150779Z   inflating: .additional_ci_files/test-class-times.json  
2025-07-24T04:26:20.2196950Z ##[group]Run rm artifacts.zip
2025-07-24T04:26:20.2197505Z rm artifacts.zip
2025-07-24T04:26:20.2248526Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:20.2249189Z env:
2025-07-24T04:26:20.2249568Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:20.2250327Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:20.2251612Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:20.2253334Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:20.2255397Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:20.2256885Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:20.2257150Z   AWS_REGION: us-east-1
2025-07-24T04:26:20.2257479Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:20.2257832Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:20.2264447Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:20.2265000Z ##[endgroup]
2025-07-24T04:26:20.5976964Z ##[group]Run df -H
2025-07-24T04:26:20.5977370Z df -H
2025-07-24T04:26:20.6022061Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:20.6022752Z env:
2025-07-24T04:26:20.6023152Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:20.6023927Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:20.6024912Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:20.6025803Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:20.6027285Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:20.6028628Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:20.6029064Z   AWS_REGION: us-east-1
2025-07-24T04:26:20.6029599Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:20.6030296Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:20.6040071Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:20.6040469Z ##[endgroup]
2025-07-24T04:26:20.6187605Z Filesystem                                          Size  Used Avail Use% Mounted on
2025-07-24T04:26:20.6188337Z tmpfs                                               109G   33M  109G   1% /run
2025-07-24T04:26:20.6189019Z /dev/nvme1n1p2                                      944G   73G  823G   9% /
2025-07-24T04:26:20.6189742Z tmpfs                                               542G   33k  542G   1% /dev/shm
2025-07-24T04:26:20.6190468Z tmpfs                                               5.3M     0  5.3M   0% /run/lock
2025-07-24T04:26:20.6191236Z /dev/nvme1n1p1                                      536M  6.4M  530M   2% /boot/efi
2025-07-24T04:26:20.6192041Z /dev/nvme0n1p1                                      3.8T  1.4T  2.3T  38% /media/4TB
2025-07-24T04:26:20.6192839Z tmpfs                                               109G   33k  109G   1% /run/user/1307800118
2025-07-24T04:26:20.6193576Z 172.18.148.8:/export/amd2                           5.5T  278G  5.3T   6% /mnt
2025-07-24T04:26:20.6194463Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch     108G  327M  108G   1% /home/amd-pytorch
2025-07-24T04:26:20.6195422Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage          165T  127T   39T  77% /groups
2025-07-24T04:26:20.6196314Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch  5.5T  1.8T  3.8T  33% /scratch
2025-07-24T04:26:20.6197327Z pure1.jax.cs.cpe.ice.amd.com:/homes/jlay            108G   19G   89G  18% /home/jlay
2025-07-24T04:26:20.6262203Z Prepare all required actions
2025-07-24T04:26:20.6263052Z Getting action download info
2025-07-24T04:26:20.8264781Z ##[group]Run ./.github/actions/download-td-artifacts
2025-07-24T04:26:20.8265387Z with:
2025-07-24T04:26:20.8265736Z env:
2025-07-24T04:26:20.8266106Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:20.8266846Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:20.8267929Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:20.8268933Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:20.8270623Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:20.8272125Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:20.8272610Z   AWS_REGION: us-east-1
2025-07-24T04:26:20.8273233Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:20.8273928Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:20.8285298Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:20.8285766Z ##[endgroup]
2025-07-24T04:26:20.8338227Z ##[group]Run seemethere/download-artifact-s3@v4
2025-07-24T04:26:20.8338750Z with:
2025-07-24T04:26:20.8339074Z   name: td_results
2025-07-24T04:26:20.8339442Z   s3-bucket: gha-artifacts
2025-07-24T04:26:20.8339857Z   region: us-east-1
2025-07-24T04:26:20.8340192Z env:
2025-07-24T04:26:20.8340518Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:20.8341161Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:20.8342193Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:20.8343086Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:20.8344621Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:20.8345994Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:20.8346432Z   AWS_REGION: us-east-1
2025-07-24T04:26:20.8346906Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:20.8347492Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:20.8356937Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:20.8357327Z ##[endgroup]
2025-07-24T04:26:21.2792833Z (node:1577710) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-07-24T04:26:21.2793764Z 
2025-07-24T04:26:21.2794112Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-07-24T04:26:21.2795050Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-07-24T04:26:21.2796030Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-07-24T04:26:21.4167294Z Found 1 objects with prefix pytorch/pytorch/16487210679/td_results/
2025-07-24T04:26:21.4168573Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-07-24T04:26:21.6992357Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-07-24T04:26:21.7002344Z Artifact download has finished successfully
2025-07-24T04:26:21.7460250Z ##[group]Run mkdir -p .additional_ci_files
2025-07-24T04:26:21.7461041Z mkdir -p .additional_ci_files
2025-07-24T04:26:21.7461928Z mv td_results.json .additional_ci_files/td_results.json || true
2025-07-24T04:26:21.7514450Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:21.7515124Z env:
2025-07-24T04:26:21.7515511Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:21.7516255Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:21.7517370Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:21.7518448Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:21.7520674Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:21.7522269Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:21.7522768Z   AWS_REGION: us-east-1
2025-07-24T04:26:21.7523357Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:21.7524040Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:21.7535657Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:21.7535915Z ##[endgroup]
2025-07-24T04:26:21.7745342Z ##[group]Run .github/scripts/parse_ref.py
2025-07-24T04:26:21.7746043Z .github/scripts/parse_ref.py
2025-07-24T04:26:21.7799749Z shell: /usr/bin/bash -e {0}
2025-07-24T04:26:21.7800271Z env:
2025-07-24T04:26:21.7800680Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:21.7801464Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:21.7802631Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:21.7803730Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:21.7806380Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:21.7807982Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:21.7808504Z   AWS_REGION: us-east-1
2025-07-24T04:26:21.7809076Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:21.7809836Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:21.7821141Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:21.7821619Z ##[endgroup]
2025-07-24T04:26:21.8277557Z Setting output tag=ciflow/trunk/149961
2025-07-24T04:26:21.8541603Z Prepare all required actions
2025-07-24T04:26:21.8542382Z Getting action download info
2025-07-24T04:26:22.0090859Z ##[group]Run ./.github/actions/filter-test-configs
2025-07-24T04:26:22.0091379Z with:
2025-07-24T04:26:22.0092006Z   github-token: ***
2025-07-24T04:26:22.0093406Z   test-matrix: {"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}
2025-07-24T04:26:22.0095134Z   job-name: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:22.0095823Z env:
2025-07-24T04:26:22.0096159Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:22.0096808Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:22.0097749Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:22.0098696Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:22.0100227Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:22.0101586Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:22.0102029Z   AWS_REGION: us-east-1
2025-07-24T04:26:22.0102511Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:22.0103163Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:22.0112550Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:22.0112927Z ##[endgroup]
2025-07-24T04:26:22.0169959Z ##[group]Run nick-fields/retry@v3.0.0
2025-07-24T04:26:22.0170442Z with:
2025-07-24T04:26:22.0170772Z   shell: bash
2025-07-24T04:26:22.0171129Z   timeout_minutes: 10
2025-07-24T04:26:22.0171497Z   max_attempts: 5
2025-07-24T04:26:22.0171865Z   retry_wait_seconds: 30
2025-07-24T04:26:22.0173062Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.1

2025-07-24T04:26:22.0174292Z   polling_interval_seconds: 1
2025-07-24T04:26:22.0174734Z   warning_on_retry: true
2025-07-24T04:26:22.0175138Z   continue_on_error: false
2025-07-24T04:26:22.0175548Z env:
2025-07-24T04:26:22.0175880Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:22.0176521Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:22.0177485Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:22.0178378Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:22.0179867Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:22.0181197Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:22.0181628Z   AWS_REGION: us-east-1
2025-07-24T04:26:22.0182103Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:22.0182699Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:22.0192131Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:22.0192713Z   GITHUB_TOKEN: ***
2025-07-24T04:26:22.0193084Z ##[endgroup]
2025-07-24T04:26:22.1244375Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.1
2025-07-24T04:26:22.4116248Z Defaulting to user installation because normal site-packages is not writeable
2025-07-24T04:26:22.5843571Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2025-07-24T04:26:22.5847248Z Requirement already satisfied: pyyaml==6.0.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.1)
2025-07-24T04:26:22.5946308Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2025-07-24T04:26:22.5952066Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2025-07-24T04:26:22.5954275Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2025-07-24T04:26:22.5965122Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2025-07-24T04:26:23.1241354Z Command completed after 1 attempt(s).
2025-07-24T04:26:23.1389786Z ##[group]Run set -x
2025-07-24T04:26:23.1390421Z set -x
2025-07-24T04:26:23.1390828Z 
2025-07-24T04:26:23.1391541Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-07-24T04:26:23.1392405Z # in runner workspace
2025-07-24T04:26:23.1393101Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-07-24T04:26:23.1451691Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:23.1452398Z env:
2025-07-24T04:26:23.1452842Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:23.1453626Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:23.1454769Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:23.1455827Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:23.1457627Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:23.1459205Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:23.1459716Z   AWS_REGION: us-east-1
2025-07-24T04:26:23.1460301Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:23.1461001Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:23.1472595Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:23.1473069Z ##[endgroup]
2025-07-24T04:26:23.1580724Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-07-24T04:26:23.1916819Z Setting output tag=ciflow/trunk/149961
2025-07-24T04:26:23.2021733Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-07-24T04:26:23.2022460Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-07-24T04:26:23.2023031Z echo "Job name: ${JOB_NAME}"
2025-07-24T04:26:23.2023578Z 
2025-07-24T04:26:23.2024239Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-07-24T04:26:23.2025095Z # in runner workspace
2025-07-24T04:26:23.2025841Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-07-24T04:26:23.2026730Z   --workflow "${GITHUB_WORKFLOW}" \
2025-07-24T04:26:23.2027409Z   --job-name "${JOB_NAME}" \
2025-07-24T04:26:23.2029221Z   --test-matrix "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}" \
2025-07-24T04:26:23.2030942Z   --selected-test-configs "" \
2025-07-24T04:26:23.2031524Z   --pr-number "${PR_NUMBER}" \
2025-07-24T04:26:23.2032054Z   --tag "${TAG}" \
2025-07-24T04:26:23.2032557Z   --event-name "${EVENT_NAME}" \
2025-07-24T04:26:23.2033545Z   --schedule "${SCHEDULE}" \
2025-07-24T04:26:23.2034083Z   --branch "${HEAD_BRANCH}"
2025-07-24T04:26:23.2089309Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:23.2089972Z env:
2025-07-24T04:26:23.2090354Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:23.2091086Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:23.2092159Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:23.2093150Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:23.2095276Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:23.2096789Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:23.2097280Z   AWS_REGION: us-east-1
2025-07-24T04:26:23.2097854Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:23.2098520Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:23.2109457Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:23.2110153Z   GITHUB_TOKEN: ***
2025-07-24T04:26:23.2110855Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:23.2111634Z   PR_NUMBER: 
2025-07-24T04:26:23.2112044Z   TAG: ciflow/trunk/149961
2025-07-24T04:26:23.2112501Z   EVENT_NAME: push
2025-07-24T04:26:23.2112893Z   SCHEDULE: 
2025-07-24T04:26:23.2113267Z   HEAD_BRANCH: 
2025-07-24T04:26:23.2113652Z ##[endgroup]
2025-07-24T04:26:23.2209785Z Workflow: trunk
2025-07-24T04:26:23.2210717Z Job name: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:23.5341795Z INFO:root:Found no test-config label on the PR, so all test configs are included
2025-07-24T04:26:23.8947350Z Setting output keep-going=False
2025-07-24T04:26:23.8948061Z Setting output ci-verbose-test-logs=False
2025-07-24T04:26:23.8948756Z Setting output ci-test-showlocals=False
2025-07-24T04:26:23.8949346Z Setting output ci-no-test-timeout=False
2025-07-24T04:26:23.8949902Z Setting output ci-no-td=False
2025-07-24T04:26:23.8950429Z Setting output ci-td-distributed=False
2025-07-24T04:26:23.8950989Z Setting output is-unstable=False
2025-07-24T04:26:23.8951510Z Setting output reenabled-issues=
2025-07-24T04:26:23.8953271Z Setting output test-matrix={"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}
2025-07-24T04:26:23.8955106Z Setting output is-test-matrix-empty=False
2025-07-24T04:26:23.9216699Z ##[group]Run echo "Filtered matrix:"
2025-07-24T04:26:23.9217293Z echo "Filtered matrix:"
2025-07-24T04:26:23.9218922Z echo "{"include": [{"config": "default", "shard": 1, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "default", "shard": 2, "num_shards": 2, "runner": "linux.rocm.gpu.2"}, {"config": "distributed", "shard": 1, "num_shards": 1, "runner": "linux.rocm.gpu.4"}]}"
2025-07-24T04:26:23.9220579Z 
2025-07-24T04:26:23.9220934Z echo
2025-07-24T04:26:23.9221401Z echo "Is the current job unstable? False"
2025-07-24T04:26:23.9222042Z 
2025-07-24T04:26:23.9222446Z echo
2025-07-24T04:26:23.9222967Z echo "Is keep-going label set? False"
2025-07-24T04:26:23.9223597Z 
2025-07-24T04:26:23.9223987Z echo
2025-07-24T04:26:23.9224454Z echo "Reenabled issues? "
2025-07-24T04:26:23.9278811Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:23.9279520Z env:
2025-07-24T04:26:23.9279886Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:23.9280632Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:23.9281705Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:23.9283224Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:23.9285290Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:23.9286835Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:23.9287313Z   AWS_REGION: us-east-1
2025-07-24T04:26:23.9287866Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:23.9288527Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:23.9300252Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:23.9301108Z ##[endgroup]
2025-07-24T04:26:23.9398131Z Filtered matrix:
2025-07-24T04:26:23.9399752Z {include: [{config: default, shard: 1, num_shards: 2, runner: linux.rocm.gpu.2}, {config: default, shard: 2, num_shards: 2, runner: linux.rocm.gpu.2}, {config: distributed, shard: 1, num_shards: 1, runner: linux.rocm.gpu.4}]}
2025-07-24T04:26:23.9401183Z 
2025-07-24T04:26:23.9401440Z Is the current job unstable? False
2025-07-24T04:26:23.9401809Z 
2025-07-24T04:26:23.9402020Z Is keep-going label set? False
2025-07-24T04:26:23.9402368Z 
2025-07-24T04:26:23.9402673Z Reenabled issues? 
2025-07-24T04:26:23.9481884Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-07-24T04:26:23.9482901Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-07-24T04:26:23.9539751Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:26:23.9540440Z env:
2025-07-24T04:26:23.9540840Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:23.9541607Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:23.9542749Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:23.9543845Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:23.9545648Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:23.9547260Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:23.9547824Z   AWS_REGION: us-east-1
2025-07-24T04:26:23.9548400Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:23.9549097Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:23.9560364Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:23.9560852Z   JOB_TIMEOUT: 300
2025-07-24T04:26:23.9561273Z ##[endgroup]
2025-07-24T04:26:23.9745667Z ##[group]Run set -x
2025-07-24T04:26:23.9746358Z set -x
2025-07-24T04:26:23.9770492Z 
2025-07-24T04:26:23.9771014Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-07-24T04:26:23.9771741Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-07-24T04:26:23.9772494Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-07-24T04:26:23.9773154Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-07-24T04:26:23.9773700Z else
2025-07-24T04:26:23.9774202Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-07-24T04:26:23.9774743Z fi
2025-07-24T04:26:23.9775102Z 
2025-07-24T04:26:23.9775711Z # detached container should get cleaned up by teardown_ec2_linux
2025-07-24T04:26:23.9776624Z # TODO: Stop building test binaries as part of the build phase
2025-07-24T04:26:23.9777416Z # Used for GPU_FLAG since that doesn't play nice
2025-07-24T04:26:23.9778179Z # shellcheck disable=SC2086,SC2090
2025-07-24T04:26:23.9778777Z container_name=$(docker run \
2025-07-24T04:26:23.9779327Z   ${GPU_FLAG:-} \
2025-07-24T04:26:23.9779805Z   -e BUILD_ENVIRONMENT \
2025-07-24T04:26:23.9780340Z   -e PR_NUMBER \
2025-07-24T04:26:23.9780832Z   -e GITHUB_ACTIONS \
2025-07-24T04:26:23.9781339Z   -e GITHUB_REPOSITORY \
2025-07-24T04:26:23.9781862Z   -e GITHUB_WORKFLOW \
2025-07-24T04:26:23.9782360Z   -e GITHUB_JOB \
2025-07-24T04:26:23.9783406Z   -e GITHUB_RUN_ID \
2025-07-24T04:26:23.9783905Z   -e GITHUB_RUN_NUMBER \
2025-07-24T04:26:23.9784430Z   -e GITHUB_RUN_ATTEMPT \
2025-07-24T04:26:23.9784944Z   -e JOB_ID \
2025-07-24T04:26:23.9785378Z   -e JOB_NAME \
2025-07-24T04:26:23.9785825Z   -e BRANCH \
2025-07-24T04:26:23.9786255Z   -e SHA1 \
2025-07-24T04:26:23.9786696Z   -e AWS_DEFAULT_REGION \
2025-07-24T04:26:23.9787213Z   -e IN_WHEEL_TEST \
2025-07-24T04:26:23.9787687Z   -e SHARD_NUMBER \
2025-07-24T04:26:23.9788153Z   -e TEST_CONFIG \
2025-07-24T04:26:23.9788618Z   -e NUM_TEST_SHARDS \
2025-07-24T04:26:23.9789109Z   -e REENABLED_ISSUES \
2025-07-24T04:26:23.9789615Z   -e CONTINUE_THROUGH_ERROR \
2025-07-24T04:26:23.9790159Z   -e VERBOSE_TEST_LOGS \
2025-07-24T04:26:23.9790659Z   -e TEST_SHOWLOCALS \
2025-07-24T04:26:23.9791144Z   -e NO_TEST_TIMEOUT \
2025-07-24T04:26:23.9791622Z   -e NO_TD \
2025-07-24T04:26:23.9792105Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-07-24T04:26:23.9792737Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-07-24T04:26:23.9793367Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-07-24T04:26:23.9793953Z   -e TESTS_TO_INCLUDE \
2025-07-24T04:26:23.9794451Z   -e DASHBOARD_TAG \
2025-07-24T04:26:23.9795083Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-07-24T04:26:23.9795786Z   --ulimit stack=10485760:83886080 \
2025-07-24T04:26:23.9796334Z   --ulimit core=0 \
2025-07-24T04:26:23.9796866Z   --security-opt seccomp=unconfined \
2025-07-24T04:26:23.9797457Z   --cap-add=SYS_PTRACE \
2025-07-24T04:26:23.9797956Z   --shm-size="8g" \
2025-07-24T04:26:23.9798422Z   --tty \
2025-07-24T04:26:23.9798834Z   --detach \
2025-07-24T04:26:23.9799294Z   --name="${container_name}" \
2025-07-24T04:26:23.9799842Z   --user jenkins \
2025-07-24T04:26:23.9800438Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-07-24T04:26:23.9801119Z   -w /var/lib/jenkins/workspace \
2025-07-24T04:26:23.9801666Z   "${DOCKER_IMAGE}"
2025-07-24T04:26:23.9802108Z )
2025-07-24T04:26:23.9802536Z # save container name for later step
2025-07-24T04:26:23.9803670Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-07-24T04:26:23.9805067Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-07-24T04:26:23.9806669Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-07-24T04:26:23.9861158Z shell: /usr/bin/bash -e {0}
2025-07-24T04:26:23.9861641Z env:
2025-07-24T04:26:23.9862039Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:26:23.9862820Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:26:23.9863947Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:26:23.9864981Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:26:23.9866695Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:26:23.9868196Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:26:23.9868690Z   AWS_REGION: us-east-1
2025-07-24T04:26:23.9869269Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:26:23.9869926Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:26:23.9880637Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:26:23.9881179Z   BUILD_ENVIRONMENT: linux-jammy-rocm-py3.10
2025-07-24T04:26:23.9881750Z   PR_NUMBER: 
2025-07-24T04:26:23.9882190Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-07-24T04:26:23.9883145Z   GITHUB_WORKFLOW: trunk
2025-07-24T04:26:23.9883597Z   GITHUB_JOB: test
2025-07-24T04:26:23.9884010Z   GITHUB_RUN_ID: 16487210679
2025-07-24T04:26:23.9884681Z   GITHUB_RUN_NUMBER: 137322
2025-07-24T04:26:23.9885151Z   GITHUB_RUN_ATTEMPT: 1
2025-07-24T04:26:23.9885580Z   JOB_ID: 46615439125
2025-07-24T04:26:23.9886273Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:23.9887066Z   BRANCH: 
2025-07-24T04:26:23.9887517Z   SHA1: 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:23.9888109Z   CONTINUE_THROUGH_ERROR: False
2025-07-24T04:26:23.9888603Z   VERBOSE_TEST_LOGS: False
2025-07-24T04:26:23.9889065Z   TEST_SHOWLOCALS: False
2025-07-24T04:26:23.9889493Z   NO_TEST_TIMEOUT: False
2025-07-24T04:26:23.9889918Z   NO_TD: False
2025-07-24T04:26:23.9890323Z   TEST_CONFIG: distributed
2025-07-24T04:26:23.9890768Z   SHARD_NUMBER: 1
2025-07-24T04:26:23.9891187Z   NUM_TEST_SHARDS: 1
2025-07-24T04:26:23.9891606Z   REENABLED_ISSUES: 
2025-07-24T04:26:23.9892855Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:26:23.9894220Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-07-24T04:26:23.9894784Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-07-24T04:26:23.9896282Z   TESTS_TO_INCLUDE: test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-07-24T04:26:23.9897734Z   DASHBOARD_TAG: 
2025-07-24T04:26:23.9898132Z ##[endgroup]
2025-07-24T04:26:23.9986376Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2025-07-24T04:26:23.9987094Z + [[ linux-jammy-rocm-py3.10 == *onnx* ]]
2025-07-24T04:26:23.9987740Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-07-24T04:26:24.0006871Z +++ nproc --ignore=2
2025-07-24T04:26:24.0043149Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_16487210679 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-f28e9a7a8bc459f8b121db7008296463e7d7437f
2025-07-24T04:26:24.2871542Z + container_name=0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:26:24.2872842Z + echo CONTAINER_NAME=0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:26:24.2878855Z + docker exec -t 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-07-24T04:26:40.6171033Z Processing ./dist/torch-2.9.0a0+git27c8ef1-cp310-cp310-linux_x86_64.whl
2025-07-24T04:26:41.3930678Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (3.18.0)
2025-07-24T04:26:41.3934490Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (4.14.1)
2025-07-24T04:26:41.3937275Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (1.13.3)
2025-07-24T04:26:41.3940740Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (2.8.8)
2025-07-24T04:26:41.3943190Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (3.1.6)
2025-07-24T04:26:41.3946907Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git27c8ef1) (2025.5.1)
2025-07-24T04:26:41.4251042Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git27c8ef1) (1.3.0)
2025-07-24T04:26:41.4296362Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.9.0a0+git27c8ef1) (3.0.2)
2025-07-24T04:26:41.9668125Z Installing collected packages: torch
2025-07-24T04:26:53.0608634Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-07-24T04:26:53.0610389Z helion 0.0.10 requires filecheck, which is not installed.
2025-07-24T04:26:53.0611387Z timm 1.0.14 requires torchvision, which is not installed.
2025-07-24T04:26:53.0612347Z Successfully installed torch-2.9.0a0+git27c8ef1
2025-07-24T04:26:53.1203493Z + export TERM=vt100
2025-07-24T04:26:53.1204022Z + TERM=vt100
2025-07-24T04:26:53.1214642Z ++ dirname .ci/pytorch/test.sh
2025-07-24T04:26:53.1248686Z + source .ci/pytorch/common.sh
2025-07-24T04:26:53.1261451Z +++ dirname .ci/pytorch/common.sh
2025-07-24T04:26:53.1291836Z ++ source .ci/pytorch/common_utils.sh
2025-07-24T04:26:53.1292720Z +++ declare -f -t trap_add
2025-07-24T04:26:53.1304136Z ++ set -ex -o pipefail
2025-07-24T04:26:53.1304747Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:26:53.1305398Z ++ unset HIP_PLATFORM
2025-07-24T04:26:53.1305902Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-07-24T04:26:53.1306461Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-07-24T04:26:53.1306960Z ++ BUILD_TEST_LIBTORCH=0
2025-07-24T04:26:53.1317383Z ++ dirname .ci/pytorch/test.sh
2025-07-24T04:26:53.1348202Z + source .ci/pytorch/common-build.sh
2025-07-24T04:26:53.1351760Z ++ [[ linux-jammy-rocm-py3.10 != *win-* ]]
2025-07-24T04:26:53.1373406Z ++++ dirname .ci/pytorch/common-build.sh
2025-07-24T04:26:53.1400441Z +++ cd .ci/pytorch
2025-07-24T04:26:53.1401370Z +++ pwd -P
2025-07-24T04:26:53.1407471Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch
2025-07-24T04:26:53.1408292Z ++ [[ linux-jammy-rocm-py3.10 == *-pch* ]]
2025-07-24T04:26:53.1408899Z ++ which sccache
2025-07-24T04:26:53.1442720Z ++ [[ -z '' ]]
2025-07-24T04:26:53.1446797Z ++ unset SCCACHE_BUCKET
2025-07-24T04:26:53.1447341Z ++ unset SCCACHE_REGION
2025-07-24T04:26:53.1447841Z ++ sccache --stop-server
2025-07-24T04:26:53.1512981Z ++ true
2025-07-24T04:26:53.1516698Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-07-24T04:26:53.1543163Z ++ trap_add sccache_epilogue EXIT
2025-07-24T04:26:53.1544132Z ++ trap_add_cmd=sccache_epilogue
2025-07-24T04:26:53.1544849Z ++ shift
2025-07-24T04:26:53.1545323Z ++ for trap_add_name in "$@"
2025-07-24T04:26:53.1561872Z ++++ trap -p EXIT
2025-07-24T04:26:53.1570530Z +++ eval 'extract_trap_cmd '
2025-07-24T04:26:53.1571139Z ++++ extract_trap_cmd
2025-07-24T04:26:53.1571653Z ++++ printf '%s\n' ''
2025-07-24T04:26:53.1572161Z +++ printf '%s\n' sccache_epilogue
2025-07-24T04:26:53.1578317Z ++ trap -- '
2025-07-24T04:26:53.1578839Z sccache_epilogue' EXIT
2025-07-24T04:26:53.1579348Z ++ [[ -n '' ]]
2025-07-24T04:26:53.1579829Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:26:53.1580559Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-07-24T04:26:53.1581236Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-07-24T04:26:53.1581710Z ++ sccache --start-server
2025-07-24T04:26:53.1639325Z sccache: Starting the server...
2025-07-24T04:26:53.2345439Z sccache: Listening on address 127.0.0.1:4226
2025-07-24T04:26:53.2373382Z ++ sccache --zero-stats
2025-07-24T04:26:53.2427875Z Statistics zeroed.
2025-07-24T04:26:53.2438228Z ++ which ccache
2025-07-24T04:26:53.2470088Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-07-24T04:26:53.2470737Z + echo 'Environment variables:'
2025-07-24T04:26:53.2471276Z Environment variables:
2025-07-24T04:26:53.2471716Z + env
2025-07-24T04:26:53.2497128Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-24T04:26:53.2498066Z CONTINUE_THROUGH_ERROR=False
2025-07-24T04:26:53.2498659Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-07-24T04:26:53.2499305Z HOSTNAME=gpu9455.jax.cs.cpe.ice.amd.com
2025-07-24T04:26:53.2500551Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2501735Z GITHUB_ACTION=__self
2025-07-24T04:26:53.2502214Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-07-24T04:26:53.2502783Z GITHUB_RUN_NUMBER=137322
2025-07-24T04:26:53.2503267Z TEST_CONFIG=distributed
2025-07-24T04:26:53.2503757Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-07-24T04:26:53.2504321Z AWS_DEFAULT_REGION=us-east-1
2025-07-24T04:26:53.2504863Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-07-24T04:26:53.2505429Z GITHUB_REF_TYPE=tag
2025-07-24T04:26:53.2506169Z ***
2025-07-24T04:26:53.2506570Z GITHUB_REPOSITORY_ID=65600975
2025-07-24T04:26:53.2507096Z GITHUB_ACTIONS=true
2025-07-24T04:26:53.2507611Z SHA1=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2508282Z GITHUB_SHA=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2509314Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/149961
2025-07-24T04:26:53.2510270Z UCC_HOME=/usr
2025-07-24T04:26:53.2510692Z VERBOSE_TEST_LOGS=False
2025-07-24T04:26:53.2511197Z GITHUB_REF=refs/tags/ciflow/trunk/149961
2025-07-24T04:26:53.2511759Z SHARD_NUMBER=1
2025-07-24T04:26:53.2512185Z GITHUB_REF_PROTECTED=false
2025-07-24T04:26:53.2512679Z HOME=/var/lib/jenkins
2025-07-24T04:26:53.2513197Z GITHUB_API_URL=https://api.github.com
2025-07-24T04:26:53.2513814Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-07-24T04:26:53.2514345Z LANG=C.UTF-8
2025-07-24T04:26:53.2514836Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-07-24T04:26:53.2515475Z PYTORCH_TEST_WITH_ROCM=1
2025-07-24T04:26:53.2515932Z NUM_TEST_SHARDS=1
2025-07-24T04:26:53.2516327Z UCX_HOME=/usr
2025-07-24T04:26:53.2517945Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2519499Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:53.2520349Z MAGMA_HOME=/opt/rocm/magma
2025-07-24T04:26:53.2521465Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2523025Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-07-24T04:26:53.2523970Z GITHUB_EVENT_NAME=push
2025-07-24T04:26:53.2524780Z DASHBOARD_TAG=
2025-07-24T04:26:53.2525200Z GITHUB_RUN_ID=16487210679
2025-07-24T04:26:53.2526393Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2527665Z GITHUB_ACTOR=pytorch-bot[bot]
2025-07-24T04:26:53.2528153Z PR_NUMBER=
2025-07-24T04:26:53.2528533Z GITHUB_RUN_ATTEMPT=1
2025-07-24T04:26:53.2529006Z ANACONDA_PYTHON_VERSION=3.10
2025-07-24T04:26:53.2529646Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-07-24T04:26:53.2530273Z TERM=vt100
2025-07-24T04:26:53.2530656Z INSTALLED_VISION=yes
2025-07-24T04:26:53.2531086Z BRANCH=
2025-07-24T04:26:53.2531492Z OPENSSL_ROOT_DIR=/opt/openssl
2025-07-24T04:26:53.2533007Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-07-24T04:26:53.2535485Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-07-24T04:26:53.2536605Z GITHUB_SERVER_URL=https://github.com
2025-07-24T04:26:53.2537201Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-07-24T04:26:53.2537771Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-07-24T04:26:53.2538365Z REENABLED_ISSUES=
2025-07-24T04:26:53.2538764Z SHLVL=1
2025-07-24T04:26:53.2539118Z MAX_JOBS=126
2025-07-24T04:26:53.2539527Z GITHUB_ACTOR_ID=54816060
2025-07-24T04:26:53.2540144Z GITHUB_WORKFLOW_SHA=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2540856Z GITHUB_REF_NAME=ciflow/trunk/149961
2025-07-24T04:26:53.2541380Z ROCM_PATH=/opt/rocm
2025-07-24T04:26:53.2541790Z GITHUB_JOB=test
2025-07-24T04:26:53.2542185Z NO_TEST_TIMEOUT=False
2025-07-24T04:26:53.2542660Z GITHUB_REPOSITORY=pytorch/pytorch
2025-07-24T04:26:53.2543173Z LC_ALL=C.UTF-8
2025-07-24T04:26:53.2543587Z GITHUB_RETENTION_DAYS=90
2025-07-24T04:26:53.2544068Z OPENSSL_DIR=/opt/openssl
2025-07-24T04:26:53.2544545Z GITHUB_ACTION_REPOSITORY=
2025-07-24T04:26:53.2546298Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-24T04:26:53.2548063Z GITHUB_BASE_REF=
2025-07-24T04:26:53.2548459Z CI=true
2025-07-24T04:26:53.2548872Z GITHUB_REPOSITORY_OWNER=pytorch
2025-07-24T04:26:53.2549363Z JOB_ID=46615439125
2025-07-24T04:26:53.2549766Z GITHUB_HEAD_REF=
2025-07-24T04:26:53.2550166Z GITHUB_ACTION_REF=
2025-07-24T04:26:53.2550580Z TEST_SHOWLOCALS=False
2025-07-24T04:26:53.2551026Z GITHUB_WORKFLOW=trunk
2025-07-24T04:26:53.2551489Z DEBIAN_FRONTEND=noninteractive
2025-07-24T04:26:53.2552681Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2553865Z NO_TD=False
2025-07-24T04:26:53.2554256Z OLDPWD=/var/lib/jenkins
2025-07-24T04:26:53.2554685Z _=/usr/bin/env
2025-07-24T04:26:53.2555242Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-07-24T04:26:53.2813361Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-07-24T04:26:53.2814520Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:26:53.2815491Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-07-24T04:26:53.2816994Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-07-24T04:26:53.2817763Z + BUILD_DIR=build
2025-07-24T04:26:53.2818199Z + BUILD_RENAMED_DIR=build_renamed
2025-07-24T04:26:53.2818728Z + BUILD_BIN_DIR=build/bin
2025-07-24T04:26:53.2819164Z + SHARD_NUMBER=1
2025-07-24T04:26:53.2819545Z + NUM_TEST_SHARDS=1
2025-07-24T04:26:53.2819971Z + export TORCH_SERIALIZATION_DEBUG=1
2025-07-24T04:26:53.2820507Z + TORCH_SERIALIZATION_DEBUG=1
2025-07-24T04:26:53.2820982Z + export VALGRIND=ON
2025-07-24T04:26:53.2821401Z + VALGRIND=ON
2025-07-24T04:26:53.2821831Z + [[ linux-jammy-rocm-py3.10 == *clang9* ]]
2025-07-24T04:26:53.2822445Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-07-24T04:26:53.2823096Z + [[ linux-jammy-rocm-py3.10 == *s390x* ]]
2025-07-24T04:26:53.2823609Z + [[ 0 == \1 ]]
2025-07-24T04:26:53.2823990Z + [[ False == \1 ]]
2025-07-24T04:26:53.2824414Z + [[ linux-jammy-rocm-py3.10 != *bazel* ]]
2025-07-24T04:26:53.2824993Z ++ realpath build/custom_test_artifacts
2025-07-24T04:26:53.2853014Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-07-24T04:26:53.2854957Z + [[ -n test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl ]]
2025-07-24T04:26:53.2856377Z + echo 'Setting INCLUDE_CLAUSE'
2025-07-24T04:26:53.2856879Z Setting INCLUDE_CLAUSE
2025-07-24T04:26:53.2858337Z + INCLUDE_CLAUSE='--include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl'
2025-07-24T04:26:53.2860395Z + echo 'Environment variables'
2025-07-24T04:26:53.2860871Z Environment variables
2025-07-24T04:26:53.2861275Z + env
2025-07-24T04:26:53.2880568Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-24T04:26:53.2881532Z CONTINUE_THROUGH_ERROR=False
2025-07-24T04:26:53.2882137Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-07-24T04:26:53.2882790Z HOSTNAME=gpu9455.jax.cs.cpe.ice.amd.com
2025-07-24T04:26:53.2883972Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2885226Z GITHUB_ACTION=__self
2025-07-24T04:26:53.2885697Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-07-24T04:26:53.2886216Z GITHUB_RUN_NUMBER=137322
2025-07-24T04:26:53.2886656Z TEST_CONFIG=distributed
2025-07-24T04:26:53.2887108Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-07-24T04:26:53.2887650Z AWS_DEFAULT_REGION=us-east-1
2025-07-24T04:26:53.2888165Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2025-07-24T04:26:53.2888691Z GITHUB_REF_TYPE=tag
2025-07-24T04:26:53.2889190Z ***
2025-07-24T04:26:53.2889583Z GITHUB_REPOSITORY_ID=65600975
2025-07-24T04:26:53.2890085Z GITHUB_ACTIONS=true
2025-07-24T04:26:53.2890550Z SHA1=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2891190Z GITHUB_SHA=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2892174Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/149961
2025-07-24T04:26:53.2893065Z UCC_HOME=/usr
2025-07-24T04:26:53.2893472Z TORCH_SERIALIZATION_DEBUG=1
2025-07-24T04:26:53.2893938Z VERBOSE_TEST_LOGS=False
2025-07-24T04:26:53.2894402Z GITHUB_REF=refs/tags/ciflow/trunk/149961
2025-07-24T04:26:53.2894919Z SHARD_NUMBER=1
2025-07-24T04:26:53.2895313Z GITHUB_REF_PROTECTED=false
2025-07-24T04:26:53.2895753Z HOME=/var/lib/jenkins
2025-07-24T04:26:53.2896229Z GITHUB_API_URL=https://api.github.com
2025-07-24T04:26:53.2896807Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-07-24T04:26:53.2897300Z LANG=C.UTF-8
2025-07-24T04:26:53.2897759Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-07-24T04:26:53.2898361Z PYTORCH_TEST_WITH_ROCM=1
2025-07-24T04:26:53.2898795Z NUM_TEST_SHARDS=1
2025-07-24T04:26:53.2899172Z UCX_HOME=/usr
2025-07-24T04:26:53.2900192Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2902059Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:26:53.2902937Z MAGMA_HOME=/opt/rocm/magma
2025-07-24T04:26:53.2904187Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2905686Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-07-24T04:26:53.2906581Z GITHUB_EVENT_NAME=push
2025-07-24T04:26:53.2907012Z DASHBOARD_TAG=
2025-07-24T04:26:53.2907412Z GITHUB_RUN_ID=16487210679
2025-07-24T04:26:53.2908570Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2909790Z GITHUB_ACTOR=pytorch-bot[bot]
2025-07-24T04:26:53.2910251Z PR_NUMBER=
2025-07-24T04:26:53.2910621Z GITHUB_RUN_ATTEMPT=1
2025-07-24T04:26:53.2911014Z VALGRIND=ON
2025-07-24T04:26:53.2911400Z ANACONDA_PYTHON_VERSION=3.10
2025-07-24T04:26:53.2911978Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-07-24T04:26:53.2912556Z TERM=vt100
2025-07-24T04:26:53.2912908Z INSTALLED_VISION=yes
2025-07-24T04:26:53.2913303Z BRANCH=
2025-07-24T04:26:53.2913676Z OPENSSL_ROOT_DIR=/opt/openssl
2025-07-24T04:26:53.2915093Z TESTS_TO_INCLUDE=test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl
2025-07-24T04:26:53.2917022Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-07-24T04:26:53.2918408Z GITHUB_SERVER_URL=https://github.com
2025-07-24T04:26:53.2918950Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-07-24T04:26:53.2919491Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-07-24T04:26:53.2920055Z REENABLED_ISSUES=
2025-07-24T04:26:53.2920420Z SHLVL=1
2025-07-24T04:26:53.2920748Z MAX_JOBS=126
2025-07-24T04:26:53.2921122Z GITHUB_ACTOR_ID=54816060
2025-07-24T04:26:53.2921706Z GITHUB_WORKFLOW_SHA=27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:26:53.2922373Z GITHUB_REF_NAME=ciflow/trunk/149961
2025-07-24T04:26:53.2922864Z ROCM_PATH=/opt/rocm
2025-07-24T04:26:53.2923240Z GITHUB_JOB=test
2025-07-24T04:26:53.2923618Z NO_TEST_TIMEOUT=False
2025-07-24T04:26:53.2924065Z GITHUB_REPOSITORY=pytorch/pytorch
2025-07-24T04:26:53.2924892Z LC_ALL=C.UTF-8
2025-07-24T04:26:53.2925277Z GITHUB_RETENTION_DAYS=90
2025-07-24T04:26:53.2925727Z OPENSSL_DIR=/opt/openssl
2025-07-24T04:26:53.2926250Z GITHUB_ACTION_REPOSITORY=
2025-07-24T04:26:53.2928206Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-24T04:26:53.2929869Z GITHUB_BASE_REF=
2025-07-24T04:26:53.2930237Z CI=true
2025-07-24T04:26:53.2930605Z GITHUB_REPOSITORY_OWNER=pytorch
2025-07-24T04:26:53.2931069Z JOB_ID=46615439125
2025-07-24T04:26:53.2931444Z GITHUB_HEAD_REF=
2025-07-24T04:26:53.2931828Z GITHUB_ACTION_REF=
2025-07-24T04:26:53.2932219Z TEST_SHOWLOCALS=False
2025-07-24T04:26:53.2932630Z GITHUB_WORKFLOW=trunk
2025-07-24T04:26:53.2933057Z DEBIAN_FRONTEND=noninteractive
2025-07-24T04:26:53.2934174Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_e623c425-eaae-487f-b43c-8ede55cc5e0c
2025-07-24T04:26:53.2935294Z NO_TD=False
2025-07-24T04:26:53.2935653Z OLDPWD=/var/lib/jenkins
2025-07-24T04:26:53.2936062Z _=/usr/bin/env
2025-07-24T04:26:53.2936483Z + echo 'Testing pytorch'
2025-07-24T04:26:53.2936906Z Testing pytorch
2025-07-24T04:26:53.2937295Z + export LANG=C.UTF-8
2025-07-24T04:26:53.2937696Z + LANG=C.UTF-8
2025-07-24T04:26:53.2938057Z + PR_NUMBER=
2025-07-24T04:26:53.2938452Z + [[ distributed == \d\e\f\a\u\l\t ]]
2025-07-24T04:26:53.2938998Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-07-24T04:26:53.2939564Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:26:53.2940117Z + export HIP_VISIBLE_DEVICES=0,1,2,3
2025-07-24T04:26:53.2940978Z + HIP_VISIBLE_DEVICES=0,1,2,3
2025-07-24T04:26:53.2941484Z + [[ distributed == \s\l\o\w ]]
2025-07-24T04:26:53.2942029Z + [[ linux-jammy-rocm-py3.10 == *slow-gradcheck* ]]
2025-07-24T04:26:53.2942659Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-07-24T04:26:53.2943310Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:26:53.2943975Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-07-24T04:26:53.2944604Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-07-24T04:26:53.2945139Z + [[ distributed == *crossref* ]]
2025-07-24T04:26:53.2945652Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:26:53.2946170Z + export VALGRIND=OFF
2025-07-24T04:26:53.2946570Z + VALGRIND=OFF
2025-07-24T04:26:53.2946928Z + rocminfo
2025-07-24T04:26:53.3058303Z ROCk module version 6.10.5 is loaded
2025-07-24T04:26:53.5063677Z =====================    
2025-07-24T04:26:53.5064335Z HSA System Attributes    
2025-07-24T04:26:53.5064809Z =====================    
2025-07-24T04:26:53.5065280Z Runtime Version:         1.15
2025-07-24T04:26:53.5065766Z Runtime Ext Version:     1.7
2025-07-24T04:26:53.5066320Z System Timestamp Freq.:  1000.000000MHz
2025-07-24T04:26:53.5067145Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-07-24T04:26:53.5068043Z Machine Model:           LARGE                              
2025-07-24T04:26:53.5068778Z System Endianness:       LITTLE                             
2025-07-24T04:26:53.5069436Z Mwaitx:                  DISABLED
2025-07-24T04:26:53.5070474Z XNACK enabled:           NO
2025-07-24T04:26:53.5070966Z DMAbuf Support:          YES
2025-07-24T04:26:53.5071438Z VMM Support:             YES
2025-07-24T04:26:53.5071727Z 
2025-07-24T04:26:53.5071897Z ==========               
2025-07-24T04:26:53.5072343Z HSA Agents               
2025-07-24T04:26:53.5072753Z ==========               
2025-07-24T04:26:53.5073150Z *******                  
2025-07-24T04:26:53.5073546Z Agent 1                  
2025-07-24T04:26:53.5073958Z *******                  
2025-07-24T04:26:53.5074475Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.5075160Z   Uuid:                    CPU-XX                             
2025-07-24T04:26:53.5075846Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.5076541Z   Vendor Name:             CPU                                
2025-07-24T04:26:53.5077198Z   Feature:                 None specified                     
2025-07-24T04:26:53.5077855Z   Profile:                 FULL_PROFILE                       
2025-07-24T04:26:53.5078538Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5079219Z   Max Queue Number:        0(0x0)                             
2025-07-24T04:26:53.5079872Z   Queue Min Size:          0(0x0)                             
2025-07-24T04:26:53.5080512Z   Queue Max Size:          0(0x0)                             
2025-07-24T04:26:53.5081156Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5081779Z   Node:                    0                                  
2025-07-24T04:26:53.5082380Z   Device Type:             CPU                                
2025-07-24T04:26:53.5082969Z   Cache Info:              
2025-07-24T04:26:53.5083466Z     L1:                      32768(0x8000) KB                   
2025-07-24T04:26:53.5084062Z   Chip ID:                 0(0x0)                             
2025-07-24T04:26:53.5084889Z   ASIC Revision:           0(0x0)                             
2025-07-24T04:26:53.5085561Z   Cacheline Size:          64(0x40)                           
2025-07-24T04:26:53.5086235Z   Max Clock Freq. (MHz):   2000                               
2025-07-24T04:26:53.5086884Z   BDFID:                   0                                  
2025-07-24T04:26:53.5087524Z   Internal Node ID:        0                                  
2025-07-24T04:26:53.5088180Z   Compute Unit:            64                                 
2025-07-24T04:26:53.5089161Z   SIMDs per CU:            0                                  
2025-07-24T04:26:53.5089822Z   Shader Engines:          0                                  
2025-07-24T04:26:53.5090498Z   Shader Arrs. per Eng.:   0                                  
2025-07-24T04:26:53.5091191Z   WatchPts on Addr. Ranges:1                                  
2025-07-24T04:26:53.5091817Z   Memory Properties:       
2025-07-24T04:26:53.5092304Z   Features:                None
2025-07-24T04:26:53.5092767Z   Pool Info:               
2025-07-24T04:26:53.5093208Z     Pool 1                   
2025-07-24T04:26:53.5093771Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5094442Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:26:53.5095091Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5095766Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5096492Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5097213Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5097921Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5098516Z     Pool 2                   
2025-07-24T04:26:53.5099071Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5099732Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:26:53.5100391Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5101417Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5102128Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5102857Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5103553Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5104148Z     Pool 3                   
2025-07-24T04:26:53.5104707Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-24T04:26:53.5105349Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:26:53.5105985Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5106654Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5107360Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5108065Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5108766Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5109353Z     Pool 4                   
2025-07-24T04:26:53.5109880Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5110514Z       Size:                    528196732(0x1f7ba47c) KB           
2025-07-24T04:26:53.5111139Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5111816Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5112514Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5113214Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5113899Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5114495Z   ISA Info:                
2025-07-24T04:26:53.5114926Z *******                  
2025-07-24T04:26:53.5115346Z Agent 2                  
2025-07-24T04:26:53.5116010Z *******                  
2025-07-24T04:26:53.5116502Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.5117140Z   Uuid:                    CPU-XX                             
2025-07-24T04:26:53.5117823Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.5118519Z   Vendor Name:             CPU                                
2025-07-24T04:26:53.5119451Z   Feature:                 None specified                     
2025-07-24T04:26:53.5120124Z   Profile:                 FULL_PROFILE                       
2025-07-24T04:26:53.5120800Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5121487Z   Max Queue Number:        0(0x0)                             
2025-07-24T04:26:53.5122147Z   Queue Min Size:          0(0x0)                             
2025-07-24T04:26:53.5122791Z   Queue Max Size:          0(0x0)                             
2025-07-24T04:26:53.5123440Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5124051Z   Node:                    1                                  
2025-07-24T04:26:53.5124996Z   Device Type:             CPU                                
2025-07-24T04:26:53.5125580Z   Cache Info:              
2025-07-24T04:26:53.5126068Z     L1:                      32768(0x8000) KB                   
2025-07-24T04:26:53.5126660Z   Chip ID:                 0(0x0)                             
2025-07-24T04:26:53.5127309Z   ASIC Revision:           0(0x0)                             
2025-07-24T04:26:53.5127985Z   Cacheline Size:          64(0x40)                           
2025-07-24T04:26:53.5128660Z   Max Clock Freq. (MHz):   2000                               
2025-07-24T04:26:53.5129290Z   BDFID:                   0                                  
2025-07-24T04:26:53.5129924Z   Internal Node ID:        1                                  
2025-07-24T04:26:53.5130587Z   Compute Unit:            64                                 
2025-07-24T04:26:53.5131569Z   SIMDs per CU:            0                                  
2025-07-24T04:26:53.5132218Z   Shader Engines:          0                                  
2025-07-24T04:26:53.5132887Z   Shader Arrs. per Eng.:   0                                  
2025-07-24T04:26:53.5133582Z   WatchPts on Addr. Ranges:1                                  
2025-07-24T04:26:53.5134203Z   Memory Properties:       
2025-07-24T04:26:53.5134669Z   Features:                None
2025-07-24T04:26:53.5135145Z   Pool Info:               
2025-07-24T04:26:53.5135586Z     Pool 1                   
2025-07-24T04:26:53.5136145Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5136811Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:26:53.5137466Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5138144Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5138863Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5139598Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5140293Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5140898Z     Pool 2                   
2025-07-24T04:26:53.5141450Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5142108Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:26:53.5142761Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5143440Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5144146Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5144856Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5145543Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5146140Z     Pool 3                   
2025-07-24T04:26:53.5146683Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-07-24T04:26:53.5147318Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:26:53.5147944Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5148616Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5149320Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5150357Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5151096Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5151694Z     Pool 4                   
2025-07-24T04:26:53.5152236Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5152871Z       Size:                    528455488(0x1f7f9740) KB           
2025-07-24T04:26:53.5153504Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5154187Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5154891Z       Alloc Recommended Granule:4KB                                
2025-07-24T04:26:53.5155596Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5156281Z       Accessible by all:       TRUE                               
2025-07-24T04:26:53.5156883Z   ISA Info:                
2025-07-24T04:26:53.5157329Z *******                  
2025-07-24T04:26:53.5157749Z Agent 3                  
2025-07-24T04:26:53.5158163Z *******                  
2025-07-24T04:26:53.5158624Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5159256Z   Uuid:                    GPU-18ffdfe035d4f207               
2025-07-24T04:26:53.5159937Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5160625Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5161283Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5162242Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5162917Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5163591Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5164248Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5165016Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5165670Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5166272Z   Node:                    2                                  
2025-07-24T04:26:53.5166881Z   Device Type:             GPU                                
2025-07-24T04:26:53.5167462Z   Cache Info:              
2025-07-24T04:26:53.5167947Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5168520Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5169113Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5169748Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5170411Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5171084Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5171703Z   BDFID:                   12800                              
2025-07-24T04:26:53.5172340Z   Internal Node ID:        2                                  
2025-07-24T04:26:53.5173004Z   Compute Unit:            104                                
2025-07-24T04:26:53.5173638Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5174288Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5174970Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5175669Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5176383Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5177006Z   Memory Properties:       
2025-07-24T04:26:53.5177506Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5178141Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5178822Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5179504Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5180496Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5181032Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5181576Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5182108Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5182705Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5183376Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5184055Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5184655Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5185144Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5185706Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5186257Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5186892Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5187657Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5188366Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5189044Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5189624Z   Pool Info:               
2025-07-24T04:26:53.5190068Z     Pool 1                   
2025-07-24T04:26:53.5190618Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5191618Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5192267Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5192939Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5193644Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5194364Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5195051Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5195642Z     Pool 2                   
2025-07-24T04:26:53.5196184Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5196832Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5197460Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5198125Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5198827Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5199536Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5200218Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5200808Z     Pool 3                   
2025-07-24T04:26:53.5201331Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5201962Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5202592Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5203259Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5203948Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5204770Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5205485Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5206088Z     Pool 4                   
2025-07-24T04:26:53.5206602Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5207215Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5207853Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5208523Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5209557Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5210263Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5210948Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5230131Z   ISA Info:                
2025-07-24T04:26:53.5230648Z     ISA 1                    
2025-07-24T04:26:53.5231314Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5232160Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5232931Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5233651Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5234355Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5235004Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5235648Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5236306Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5236880Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5237443Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5237972Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5238568Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5239162Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5240150Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5240706Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5241287Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5241943Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5242530Z *******                  
2025-07-24T04:26:53.5242956Z Agent 4                  
2025-07-24T04:26:53.5243357Z *******                  
2025-07-24T04:26:53.5243835Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5244875Z   Uuid:                    GPU-1fc9e435699e2f6c               
2025-07-24T04:26:53.5245731Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5246427Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5247079Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5247723Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5248366Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5249013Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5249647Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5250262Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5250883Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5251473Z   Node:                    3                                  
2025-07-24T04:26:53.5252068Z   Device Type:             GPU                                
2025-07-24T04:26:53.5252632Z   Cache Info:              
2025-07-24T04:26:53.5253117Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5253667Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5254229Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5254868Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5255519Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5256179Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5256800Z   BDFID:                   13568                              
2025-07-24T04:26:53.5257419Z   Internal Node ID:        3                                  
2025-07-24T04:26:53.5258468Z   Compute Unit:            104                                
2025-07-24T04:26:53.5259128Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5259763Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5260428Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5261105Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5261800Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5262419Z   Memory Properties:       
2025-07-24T04:26:53.5262911Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5263542Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5264212Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5264868Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5265480Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5266017Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5266554Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5267084Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5267779Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5268561Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5269240Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5270166Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5270637Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5271176Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5271698Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5272334Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5273109Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5273921Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5274700Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5275337Z   Pool Info:               
2025-07-24T04:26:53.5275768Z     Pool 1                   
2025-07-24T04:26:53.5276326Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5277010Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5277683Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5278372Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5279102Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5279807Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5280487Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5281080Z     Pool 2                   
2025-07-24T04:26:53.5281632Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5282285Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5282910Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5283571Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5284254Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5285074Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5285752Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5286338Z     Pool 3                   
2025-07-24T04:26:53.5286857Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5287483Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5288434Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5289113Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5289795Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5290489Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5291149Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5291730Z     Pool 4                   
2025-07-24T04:26:53.5292247Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5292845Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5293491Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5294148Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5294883Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5295581Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5296252Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5296843Z   ISA Info:                
2025-07-24T04:26:53.5297290Z     ISA 1                    
2025-07-24T04:26:53.5297841Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5298566Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5299257Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5300265Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5300975Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5301628Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5302272Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5302899Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5303451Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5304039Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5304579Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5305175Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5305784Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5306305Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5306863Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5307402Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5308019Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5308601Z *******                  
2025-07-24T04:26:53.5309013Z Agent 5                  
2025-07-24T04:26:53.5309407Z *******                  
2025-07-24T04:26:53.5309883Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5310504Z   Uuid:                    GPU-8fcbcaf6dd10fade               
2025-07-24T04:26:53.5311179Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5311849Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5312499Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5313126Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5313898Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5314683Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5315369Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5315999Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5316623Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5317486Z   Node:                    4                                  
2025-07-24T04:26:53.5318106Z   Device Type:             GPU                                
2025-07-24T04:26:53.5318674Z   Cache Info:              
2025-07-24T04:26:53.5319148Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5319712Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5320279Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5320904Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5321569Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5322229Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5322843Z   BDFID:                   4352                               
2025-07-24T04:26:53.5323464Z   Internal Node ID:        4                                  
2025-07-24T04:26:53.5324125Z   Compute Unit:            104                                
2025-07-24T04:26:53.5324845Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5325488Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5326155Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5326872Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5327696Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5328404Z   Memory Properties:       
2025-07-24T04:26:53.5329314Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5329937Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5330610Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5331279Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5331899Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5332424Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5332959Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5333483Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5334072Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5334736Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5335382Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5335947Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5336420Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5336958Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5337484Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5338106Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5338814Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5339501Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5340190Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5340860Z   Pool Info:               
2025-07-24T04:26:53.5341357Z     Pool 1                   
2025-07-24T04:26:53.5341940Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5342588Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5343214Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5343889Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5344573Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5345263Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5345939Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5346519Z     Pool 2                   
2025-07-24T04:26:53.5347365Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5348006Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5348623Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5349267Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5349944Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5350627Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5351304Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5351877Z     Pool 3                   
2025-07-24T04:26:53.5352388Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5352999Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5353610Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5354265Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5354943Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5355629Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5356290Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5356864Z     Pool 4                   
2025-07-24T04:26:53.5357353Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5358219Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5358826Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5359477Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5360152Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5360841Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5361513Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5362097Z   ISA Info:                
2025-07-24T04:26:53.5362505Z     ISA 1                    
2025-07-24T04:26:53.5363045Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5363767Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5364747Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5365440Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5366150Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5366790Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5367428Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5368050Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5368584Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5369067Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5369532Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5370054Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5370563Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5371012Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5371479Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5371953Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5372507Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5373014Z *******                  
2025-07-24T04:26:53.5373371Z Agent 6                  
2025-07-24T04:26:53.5373714Z *******                  
2025-07-24T04:26:53.5374115Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5374987Z   Uuid:                    GPU-61b30b7b05420286               
2025-07-24T04:26:53.5375583Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5376177Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5376735Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5377297Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5377870Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5378461Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5379027Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5379583Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5380130Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5380659Z   Node:                    5                                  
2025-07-24T04:26:53.5381191Z   Device Type:             GPU                                
2025-07-24T04:26:53.5381687Z   Cache Info:              
2025-07-24T04:26:53.5382086Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5382561Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5383052Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5383598Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5384465Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5385044Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5385576Z   BDFID:                   5120                               
2025-07-24T04:26:53.5386110Z   Internal Node ID:        5                                  
2025-07-24T04:26:53.5386675Z   Compute Unit:            104                                
2025-07-24T04:26:53.5387225Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5387780Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5388361Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5388952Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5389550Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5390080Z   Memory Properties:       
2025-07-24T04:26:53.5390498Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5391046Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5391627Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5392211Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5392751Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5393200Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5393669Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5394125Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5394631Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5395210Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5395781Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5396293Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5396701Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5397185Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5397651Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5398203Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5398831Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5399428Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5400248Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5400767Z   Pool Info:               
2025-07-24T04:26:53.5401127Z     Pool 1                   
2025-07-24T04:26:53.5401597Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5402160Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5402721Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5403296Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5403916Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5404639Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5405236Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5405752Z     Pool 2                   
2025-07-24T04:26:53.5406221Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5406785Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5407324Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5407891Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5408485Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5409085Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5409671Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5410479Z     Pool 3                   
2025-07-24T04:26:53.5410941Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5411487Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5412024Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5412599Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5413209Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5413817Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5414401Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5414903Z     Pool 4                   
2025-07-24T04:26:53.5415340Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5415861Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5416406Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5416979Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5417580Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5418179Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5418764Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5419280Z   ISA Info:                
2025-07-24T04:26:53.5419642Z     ISA 1                    
2025-07-24T04:26:53.5420112Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5420740Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5421335Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5421932Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5422551Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5423119Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5423680Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5424220Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5424690Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5425459Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5425936Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5426446Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5426961Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5427398Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5427872Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5428343Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5428886Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5429383Z *******                  
2025-07-24T04:26:53.5429735Z Agent 7                  
2025-07-24T04:26:53.5430081Z *******                  
2025-07-24T04:26:53.5430481Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5431030Z   Uuid:                    GPU-5f75bf9184112d67               
2025-07-24T04:26:53.5431650Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5432253Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5432828Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5433384Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5433956Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5434534Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5435346Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5435902Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5436464Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5436985Z   Node:                    6                                  
2025-07-24T04:26:53.5437510Z   Device Type:             GPU                                
2025-07-24T04:26:53.5438018Z   Cache Info:              
2025-07-24T04:26:53.5438438Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5438934Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5439433Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5439972Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5440543Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5441122Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5441655Z   BDFID:                   44544                              
2025-07-24T04:26:53.5442192Z   Internal Node ID:        6                                  
2025-07-24T04:26:53.5442755Z   Compute Unit:            104                                
2025-07-24T04:26:53.5443303Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5443859Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5444656Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5445263Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5445864Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5446399Z   Memory Properties:       
2025-07-24T04:26:53.5446825Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5447363Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5447955Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5448545Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5449084Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5449534Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5450000Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5450764Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5451294Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5451874Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5452445Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5452956Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5453373Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5453845Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5454323Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5454871Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5455496Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5456095Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5456684Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5457189Z   Pool Info:               
2025-07-24T04:26:53.5457567Z     Pool 1                   
2025-07-24T04:26:53.5458052Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5458639Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5459217Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5459805Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5460814Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5461457Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5462077Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5462596Z     Pool 2                   
2025-07-24T04:26:53.5463085Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5463670Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5464238Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5464837Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5465455Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5466079Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5466681Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5467205Z     Pool 3                   
2025-07-24T04:26:53.5467667Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5468212Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5468746Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5469326Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5469942Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5470558Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5471155Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5471664Z     Pool 4                   
2025-07-24T04:26:53.5472104Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5472625Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5473161Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5473745Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5474342Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5474954Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5475544Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5476056Z   ISA Info:                
2025-07-24T04:26:53.5476675Z     ISA 1                    
2025-07-24T04:26:53.5477176Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5477814Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5478425Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5479027Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5479651Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5480233Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5480808Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5481356Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5481835Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5482314Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5482784Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5483308Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5483828Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5484278Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5484850Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5485322Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5486148Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5486651Z *******                  
2025-07-24T04:26:53.5487016Z Agent 8                  
2025-07-24T04:26:53.5487362Z *******                  
2025-07-24T04:26:53.5487765Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5488299Z   Uuid:                    GPU-5fa6c987997a0603               
2025-07-24T04:26:53.5488880Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5489466Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5490023Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5490581Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5491145Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5491718Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5492276Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5492841Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5493382Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5493897Z   Node:                    7                                  
2025-07-24T04:26:53.5494416Z   Device Type:             GPU                                
2025-07-24T04:26:53.5494911Z   Cache Info:              
2025-07-24T04:26:53.5495349Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5495860Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5496364Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5496908Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5497478Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5498052Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5498597Z   BDFID:                   45824                              
2025-07-24T04:26:53.5499125Z   Internal Node ID:        7                                  
2025-07-24T04:26:53.5499694Z   Compute Unit:            104                                
2025-07-24T04:26:53.5500267Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5500840Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5501719Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5502350Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5502969Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5503511Z   Memory Properties:       
2025-07-24T04:26:53.5503943Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5504477Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5505063Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5505654Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5506194Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5506652Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5507117Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5507576Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5508107Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5508690Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5509271Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5509788Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5510204Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5510680Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5511407Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5511963Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5512593Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5513200Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5513792Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5514304Z   Pool Info:               
2025-07-24T04:26:53.5514690Z     Pool 1                   
2025-07-24T04:26:53.5515180Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5515759Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5516320Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5516908Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5517515Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5518146Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5518747Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5519264Z     Pool 2                   
2025-07-24T04:26:53.5519732Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5520290Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5520851Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5521430Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5522032Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5522642Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5523233Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5523736Z     Pool 3                   
2025-07-24T04:26:53.5524199Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5524993Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5525535Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5526114Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5526716Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5527618Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5528226Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5528739Z     Pool 4                   
2025-07-24T04:26:53.5529175Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5529696Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5530237Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5530818Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5531417Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5532048Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5532648Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5533170Z   ISA Info:                
2025-07-24T04:26:53.5533545Z     ISA 1                    
2025-07-24T04:26:53.5534045Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5534683Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5535287Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5535889Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5536513Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5537075Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5537942Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5538496Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5538979Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5539461Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5539930Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5540464Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5540989Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5541442Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5541932Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5542440Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5543003Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5543539Z *******                  
2025-07-24T04:26:53.5543913Z Agent 9                  
2025-07-24T04:26:53.5544286Z *******                  
2025-07-24T04:26:53.5544710Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5545268Z   Uuid:                    GPU-805d7be51a59017b               
2025-07-24T04:26:53.5545861Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5546456Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5547029Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5547596Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5548171Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5548745Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5549318Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5549891Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5550448Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5550972Z   Node:                    8                                  
2025-07-24T04:26:53.5551503Z   Device Type:             GPU                                
2025-07-24T04:26:53.5552002Z   Cache Info:              
2025-07-24T04:26:53.5552424Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5553170Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5553700Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5554261Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5554841Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5555423Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5555974Z   BDFID:                   36352                              
2025-07-24T04:26:53.5556542Z   Internal Node ID:        8                                  
2025-07-24T04:26:53.5557119Z   Compute Unit:            104                                
2025-07-24T04:26:53.5557679Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5558245Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5558845Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5559458Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5560073Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5560611Z   Memory Properties:       
2025-07-24T04:26:53.5561033Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5561589Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5562207Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5562812Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5563672Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5564143Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5564723Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5565187Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5565718Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5566320Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5566909Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5567428Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5567850Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5568347Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5568828Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5569394Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5570040Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5570654Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5571251Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5571767Z   Pool Info:               
2025-07-24T04:26:53.5572154Z     Pool 1                   
2025-07-24T04:26:53.5572653Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5573233Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5573824Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5574422Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5575048Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5575679Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5576296Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5576819Z     Pool 2                   
2025-07-24T04:26:53.5577297Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5577870Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5578424Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5579300Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5579928Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5580559Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5581166Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5581684Z     Pool 3                   
2025-07-24T04:26:53.5582151Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5582707Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5583260Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5583844Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5584462Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5585084Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5585695Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5586210Z     Pool 4                   
2025-07-24T04:26:53.5586647Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5587192Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5587745Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5588330Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5589239Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5589855Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5590454Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5590974Z   ISA Info:                
2025-07-24T04:26:53.5591349Z     ISA 1                    
2025-07-24T04:26:53.5591845Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5592499Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5593111Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5593722Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5594345Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5594927Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5595505Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5596072Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5596561Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5597052Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5597526Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5598061Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5598594Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5599041Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5599524Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5600004Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5600551Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5601063Z *******                  
2025-07-24T04:26:53.5601434Z Agent 10                 
2025-07-24T04:26:53.5601789Z *******                  
2025-07-24T04:26:53.5602208Z   Name:                    gfx90a                             
2025-07-24T04:26:53.5602752Z   Uuid:                    GPU-7fd720b66acc078e               
2025-07-24T04:26:53.5603338Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.5603930Z   Vendor Name:             AMD                                
2025-07-24T04:26:53.5604864Z   Feature:                 KERNEL_DISPATCH                    
2025-07-24T04:26:53.5605447Z   Profile:                 BASE_PROFILE                       
2025-07-24T04:26:53.5606036Z   Float Round Mode:        NEAR                               
2025-07-24T04:26:53.5606625Z   Max Queue Number:        128(0x80)                          
2025-07-24T04:26:53.5607200Z   Queue Min Size:          64(0x40)                           
2025-07-24T04:26:53.5607760Z   Queue Max Size:          131072(0x20000)                    
2025-07-24T04:26:53.5608330Z   Queue Type:              MULTI                              
2025-07-24T04:26:53.5608856Z   Node:                    9                                  
2025-07-24T04:26:53.5609395Z   Device Type:             GPU                                
2025-07-24T04:26:53.5609897Z   Cache Info:              
2025-07-24T04:26:53.5610306Z     L1:                      16(0x10) KB                        
2025-07-24T04:26:53.5610800Z     L2:                      8192(0x2000) KB                    
2025-07-24T04:26:53.5611306Z   Chip ID:                 29708(0x740c)                      
2025-07-24T04:26:53.5611858Z   ASIC Revision:           1(0x1)                             
2025-07-24T04:26:53.5612435Z   Cacheline Size:          128(0x80)                          
2025-07-24T04:26:53.5613017Z   Max Clock Freq. (MHz):   1700                               
2025-07-24T04:26:53.5613558Z   BDFID:                   37632                              
2025-07-24T04:26:53.5614102Z   Internal Node ID:        9                                  
2025-07-24T04:26:53.5614965Z   Compute Unit:            104                                
2025-07-24T04:26:53.5615523Z   SIMDs per CU:            4                                  
2025-07-24T04:26:53.5616090Z   Shader Engines:          8                                  
2025-07-24T04:26:53.5616674Z   Shader Arrs. per Eng.:   1                                  
2025-07-24T04:26:53.5617277Z   WatchPts on Addr. Ranges:4                                  
2025-07-24T04:26:53.5617897Z   Coherent Host Access:    FALSE                              
2025-07-24T04:26:53.5618438Z   Memory Properties:       
2025-07-24T04:26:53.5618864Z   Features:                KERNEL_DISPATCH 
2025-07-24T04:26:53.5619405Z   Fast F16 Operation:      TRUE                               
2025-07-24T04:26:53.5620000Z   Wavefront Size:          64(0x40)                           
2025-07-24T04:26:53.5620590Z   Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5621140Z   Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5621601Z     x                        1024(0x400)                        
2025-07-24T04:26:53.5622075Z     y                        1024(0x400)                        
2025-07-24T04:26:53.5622537Z     z                        1024(0x400)                        
2025-07-24T04:26:53.5623053Z   Max Waves Per CU:        32(0x20)                           
2025-07-24T04:26:53.5623646Z   Max Work-item Per CU:    2048(0x800)                        
2025-07-24T04:26:53.5624229Z   Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5624746Z   Grid Max Size per Dimension:
2025-07-24T04:26:53.5625166Z     x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5625649Z     y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5626130Z     z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5626686Z   Max fbarriers/Workgrp:   32                                 
2025-07-24T04:26:53.5627330Z   Packet Processor uCode:: 92                                 
2025-07-24T04:26:53.5627941Z   SDMA engine uCode::      9                                  
2025-07-24T04:26:53.5628544Z   IOMMU Support::          None                               
2025-07-24T04:26:53.5639315Z   Pool Info:               
2025-07-24T04:26:53.5639826Z     Pool 1                   
2025-07-24T04:26:53.5640344Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-07-24T04:26:53.5641271Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5641901Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5642494Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5643125Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5643758Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5644367Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5645215Z     Pool 2                   
2025-07-24T04:26:53.5645705Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-07-24T04:26:53.5646263Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5646814Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5647385Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5648042Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5648762Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5649452Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5650060Z     Pool 3                   
2025-07-24T04:26:53.5650609Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-07-24T04:26:53.5651246Z       Size:                    67092480(0x3ffc000) KB             
2025-07-24T04:26:53.5652264Z       Allocatable:             TRUE                               
2025-07-24T04:26:53.5652946Z       Alloc Granule:           4KB                                
2025-07-24T04:26:53.5653662Z       Alloc Recommended Granule:2048KB                             
2025-07-24T04:26:53.5654311Z       Alloc Alignment:         4KB                                
2025-07-24T04:26:53.5654636Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5654901Z     Pool 4                   
2025-07-24T04:26:53.5655108Z       Segment:                 GROUP                              
2025-07-24T04:26:53.5655353Z       Size:                    64(0x40) KB                        
2025-07-24T04:26:53.5655603Z       Allocatable:             FALSE                              
2025-07-24T04:26:53.5655870Z       Alloc Granule:           0KB                                
2025-07-24T04:26:53.5656148Z       Alloc Recommended Granule:0KB                                
2025-07-24T04:26:53.5656436Z       Alloc Alignment:         0KB                                
2025-07-24T04:26:53.5656711Z       Accessible by all:       FALSE                              
2025-07-24T04:26:53.5656951Z   ISA Info:                
2025-07-24T04:26:53.5657124Z     ISA 1                    
2025-07-24T04:26:53.5657347Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-07-24T04:26:53.5657644Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-07-24T04:26:53.5657927Z       Profiles:                HSA_PROFILE_BASE                   
2025-07-24T04:26:53.5658206Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5658492Z       Default Rounding Mode:   NEAR                               
2025-07-24T04:26:53.5658756Z       Fast f16:                TRUE                               
2025-07-24T04:26:53.5659021Z       Workgroup Max Size:      1024(0x400)                        
2025-07-24T04:26:53.5659279Z       Workgroup Max Size per Dimension:
2025-07-24T04:26:53.5659506Z         x                        1024(0x400)                        
2025-07-24T04:26:53.5659730Z         y                        1024(0x400)                        
2025-07-24T04:26:53.5659943Z         z                        1024(0x400)                        
2025-07-24T04:26:53.5660186Z       Grid Max Size:           4294967295(0xffffffff)             
2025-07-24T04:26:53.5660426Z       Grid Max Size per Dimension:
2025-07-24T04:26:53.5660792Z         x                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5661026Z         y                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5661247Z         z                        4294967295(0xffffffff)             
2025-07-24T04:26:53.5661498Z       FBarrier Max Size:       32                                 
2025-07-24T04:26:53.5661737Z *** Done ***             
2025-07-24T04:26:53.5661912Z + rocminfo
2025-07-24T04:26:53.5662074Z + grep -E 'Name:.*\sgfx|Marketing'
2025-07-24T04:26:53.7486127Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.7486879Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-07-24T04:26:53.7487462Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7488077Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7488654Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7489223Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7489755Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7490286Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7490813Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7491391Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7492014Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7493212Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7493829Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7494440Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7495057Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7495613Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7495979Z   Name:                    gfx90a                             
2025-07-24T04:26:53.7496302Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-07-24T04:26:53.7672927Z + MAYBE_ROCM=rocm/
2025-07-24T04:26:53.7673856Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-07-24T04:26:53.7674558Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-07-24T04:26:53.7675149Z + pip_install ninja==1.10.2
2025-07-24T04:26:53.7675824Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-07-24T04:26:53.7676717Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-07-24T04:26:54.2594517Z Collecting ninja==1.10.2
2025-07-24T04:26:54.2883486Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-07-24T04:26:54.3030703Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-07-24T04:26:54.8125595Z Installing collected packages: ninja
2025-07-24T04:26:54.8127695Z   Attempting uninstall: ninja
2025-07-24T04:26:54.8132951Z     Found existing installation: ninja 1.11.1.3
2025-07-24T04:26:54.8155713Z     Uninstalling ninja-1.11.1.3:
2025-07-24T04:26:54.8263301Z       Successfully uninstalled ninja-1.11.1.3
2025-07-24T04:26:54.8670054Z Successfully installed ninja-1.10.2
2025-07-24T04:26:54.9352170Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-24T04:26:54.9356035Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-07-24T04:26:54.9358007Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-07-24T04:26:54.9358630Z + [[ linux-jammy-rocm-py3.10 == *asan* ]]
2025-07-24T04:26:54.9360156Z + [[ linux-jammy-rocm-py3.10 == *-debug* ]]
2025-07-24T04:26:54.9360845Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-07-24T04:26:54.9361768Z + echo 'We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass'
2025-07-24T04:26:54.9362852Z We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass
2025-07-24T04:26:54.9363609Z + cd test
2025-07-24T04:26:54.9364333Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-07-24T04:26:56.5636083Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-07-24T04:26:56.5636867Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-07-24T04:26:56.5637588Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-07-24T04:26:56.5644696Z + DYNAMO_BENCHMARK_FLAGS=()
2025-07-24T04:26:56.5645495Z + [[ distributed == *pr_time_benchmarks* ]]
2025-07-24T04:26:56.5646147Z + [[ distributed == *dynamo_eager* ]]
2025-07-24T04:26:56.5646686Z + [[ distributed == *aot_eager* ]]
2025-07-24T04:26:56.5647248Z + [[ distributed == *aot_inductor* ]]
2025-07-24T04:26:56.5647810Z + [[ distributed == *max_autotune_inductor* ]]
2025-07-24T04:26:56.5648378Z + [[ distributed == *inductor* ]]
2025-07-24T04:26:56.5648888Z + [[ distributed == *dynamic* ]]
2025-07-24T04:26:56.5649388Z + [[ distributed == *cpu* ]]
2025-07-24T04:26:56.5649940Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-07-24T04:26:56.5688044Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-07-24T04:26:56.5688773Z + [[ linux-jammy-rocm-py3.10 == *-bazel-* ]]
2025-07-24T04:26:56.5696997Z + cd test
2025-07-24T04:26:56.5698582Z + python -c 'import torch; print(torch.__config__.show())'
2025-07-24T04:26:57.9273464Z PyTorch built with:
2025-07-24T04:26:57.9274016Z   - GCC 11.4
2025-07-24T04:26:57.9274445Z   - C++ Version: 201703
2025-07-24T04:26:57.9275550Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-07-24T04:26:57.9277039Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-07-24T04:26:57.9277876Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-07-24T04:26:57.9278464Z   - LAPACK is enabled (usually provided by MKL)
2025-07-24T04:26:57.9279030Z   - NNPACK is enabled
2025-07-24T04:26:57.9279494Z   - CPU capability usage: AVX2
2025-07-24T04:26:57.9279989Z   - HIP Runtime 6.4.43484
2025-07-24T04:26:57.9280418Z   - MIOpen 3.4.0
2025-07-24T04:26:57.9280807Z   - Magma 2.7.2
2025-07-24T04:26:57.9288554Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=27c8ef1839e0747b36769efb192befc9308ff93c, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-07-24T04:26:57.9296706Z 
2025-07-24T04:26:58.1859300Z + cd test
2025-07-24T04:26:58.1859989Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-07-24T04:26:59.3399338Z ATen/Parallel:
2025-07-24T04:26:59.3399895Z 	at::get_num_threads() : 128
2025-07-24T04:26:59.3400466Z 	at::get_num_interop_threads() : 128
2025-07-24T04:26:59.3401027Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-07-24T04:26:59.3401555Z 	omp_get_max_threads() : 128
2025-07-24T04:26:59.3412043Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-07-24T04:26:59.3413348Z 	mkl_get_max_threads() : 128
2025-07-24T04:26:59.3414063Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-07-24T04:26:59.3414861Z std::thread::hardware_concurrency() : 128
2025-07-24T04:26:59.3415409Z Environment variables:
2025-07-24T04:26:59.3415858Z 	OMP_NUM_THREADS : [not set]
2025-07-24T04:26:59.3416318Z 	MKL_NUM_THREADS : [not set]
2025-07-24T04:26:59.3416803Z ATen parallel backend: OpenMP
2025-07-24T04:26:59.3417128Z 
2025-07-24T04:26:59.7919600Z + [[ distributed == *numpy_2* ]]
2025-07-24T04:26:59.7920271Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-07-24T04:26:59.7920945Z + [[ distributed == *backward* ]]
2025-07-24T04:26:59.7921545Z + [[ distributed == *xla* ]]
2025-07-24T04:26:59.7922122Z + [[ distributed == *executorch* ]]
2025-07-24T04:26:59.7922775Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2025-07-24T04:26:59.7923490Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-07-24T04:26:59.7924085Z + [[ distributed == distributed ]]
2025-07-24T04:26:59.7924738Z + test_distributed
2025-07-24T04:26:59.7925200Z + echo 'Testing distributed python tests'
2025-07-24T04:26:59.7925769Z Testing distributed python tests
2025-07-24T04:26:59.7927639Z + python test/run_test.py --distributed-tests --shard 1 1 --include test_nn test_torch test_cuda test_ops test_unary_ufuncs test_binary_ufuncs test_autograd inductor/test_torchinductor distributed/test_c10d_common distributed/test_c10d_nccl --verbose
2025-07-24T04:27:02.4492485Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:27:02.4495233Z   import pkg_resources
2025-07-24T04:27:04.3246766Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:27:04.5091918Z Ignoring disabled issues:  ['']
2025-07-24T04:27:04.5572409Z Found test times from artifacts
2025-07-24T04:27:04.6495684Z Found test times from artifacts
2025-07-24T04:27:04.6529988Z Running 25% of tests based on TD
2025-07-24T04:27:04.6533271Z Running parallel tests on 8 processes
2025-07-24T04:27:04.6534080Z Name: tests to run (est. time: 1.82min)
2025-07-24T04:27:04.6534643Z   Serial tests (1):
2025-07-24T04:27:04.6535092Z     distributed/test_c10d_common 1/1
2025-07-24T04:27:04.6535644Z   Parallel tests (0):
2025-07-24T04:27:04.6536104Z Name: excluded (est. time: 21.97min)
2025-07-24T04:27:04.6536638Z   Serial tests (3):
2025-07-24T04:27:04.6537078Z     distributed/test_c10d_nccl 1/3
2025-07-24T04:27:04.6537606Z     distributed/test_c10d_nccl 2/3
2025-07-24T04:27:04.6538123Z     distributed/test_c10d_nccl 3/3
2025-07-24T04:27:04.6538664Z   Parallel tests (0):
2025-07-24T04:27:04.6539341Z Running distributed/test_c10d_common 1/1 ... [2025-07-24 04:27:04.653230]
2025-07-24T04:27:04.6540114Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:27:04.6542052Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-24 04:27:04.653618]
2025-07-24T04:30:31.5204818Z 
2025-07-24T04:30:31.5206645Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_f950c278c6069932_.log
2025-07-24T04:30:31.5243151Z Running 27 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-07-24T04:30:31.5264228Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier
2025-07-24T04:30:31.5265840Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype
2025-07-24T04:30:31.5267581Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype
2025-07-24T04:30:31.5269336Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype
2025-07-24T04:30:31.5271117Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype
2025-07-24T04:30:31.5272676Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level
2025-07-24T04:30:31.5274088Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort
2025-07-24T04:30:31.5275741Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr
2025-07-24T04:30:31.5277470Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config
2025-07-24T04:30:31.5279356Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper
2025-07-24T04:30:31.5281042Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives
2025-07-24T04:30:31.5282681Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name
2025-07-24T04:30:31.5285049Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends
2025-07-24T04:30:31.5287009Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available
2025-07-24T04:30:31.5288682Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv
2025-07-24T04:30:31.5290267Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown
2025-07-24T04:30:31.5292126Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-07-24T04:30:31.5294315Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends
2025-07-24T04:30:31.5296589Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-07-24T04:30:31.5298535Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop
2025-07-24T04:30:31.5300034Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable
2025-07-24T04:30:31.5301588Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal
2025-07-24T04:30:31.5303226Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle
2025-07-24T04:30:31.5305020Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank
2025-07-24T04:30:31.5306522Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback
2025-07-24T04:30:31.5308001Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv
2025-07-24T04:30:31.5309415Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-07-24T04:30:31.5310263Z 
2025-07-24T04:30:31.5310737Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:30:31.5311602Z Uploading artifacts took 0.00 seconds
2025-07-24T04:30:34.5680673Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.5683619Z   import pkg_resources
2025-07-24T04:30:34.5686590Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.5689292Z   import pkg_resources
2025-07-24T04:30:34.5947023Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.5949814Z   import pkg_resources
2025-07-24T04:30:34.6691803Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.6694534Z   import pkg_resources
2025-07-24T04:30:34.7013094Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.7015979Z   import pkg_resources
2025-07-24T04:30:34.7121582Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.7124632Z   import pkg_resources
2025-07-24T04:30:34.7158974Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.7161848Z   import pkg_resources
2025-07-24T04:30:34.7237961Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:34.7241007Z   import pkg_resources
2025-07-24T04:30:36.0389830Z Running test batch 'tests to run' cost 211.39 seconds
2025-07-24T04:30:36.8174767Z 
2025-07-24T04:30:36.8175218Z real	3m37.025s
2025-07-24T04:30:36.8175718Z user	6m15.192s
2025-07-24T04:30:36.8176124Z sys	9m37.244s
2025-07-24T04:30:36.8176524Z + assert_git_not_dirty
2025-07-24T04:30:36.8177042Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-07-24T04:30:36.8177634Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-07-24T04:30:36.8178200Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-07-24T04:30:36.8178738Z + [[ 1 == 1 ]]
2025-07-24T04:30:36.8179153Z + echo 'Testing distributed C++ tests'
2025-07-24T04:30:36.8179705Z Testing distributed C++ tests
2025-07-24T04:30:36.8184199Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:30:36.8221104Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:30:36.8257374Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:30:36.8258560Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:30:36.8259521Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest
2025-07-24T04:30:39.4899619Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:39.4902492Z   import pkg_resources
2025-07-24T04:30:41.2718136Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:30:41.2921628Z Found test times from artifacts
2025-07-24T04:30:41.3780464Z Found test times from artifacts
2025-07-24T04:30:41.3815253Z Running 25% of tests based on TD
2025-07-24T04:30:41.3817881Z Running parallel tests on 8 processes
2025-07-24T04:30:41.3818526Z Name: tests to run (est. time: 0.0min)
2025-07-24T04:30:41.3819055Z   Serial tests (1):
2025-07-24T04:30:41.3819501Z     cpp/FileStoreTest 1/1
2025-07-24T04:30:41.3819976Z   Parallel tests (0):
2025-07-24T04:30:41.3820420Z Name: excluded (est. time: 0.0min)
2025-07-24T04:30:41.3820905Z   Serial tests (0):
2025-07-24T04:30:41.3821302Z   Parallel tests (0):
2025-07-24T04:30:41.3822729Z Running cpp/FileStoreTest 1/1 ... [2025-07-24 04:30:41.381879]
2025-07-24T04:30:41.3823463Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:30:41.3831150Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-8aee367754df5219.xml', '-x', '--reruns=2'] ... [2025-07-24 04:30:41.382477]
2025-07-24T04:30:43.8562908Z 
2025-07-24T04:30:43.8564700Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_f1124047cc752ba5_.log
2025-07-24T04:30:43.8565893Z 
2025-07-24T04:30:43.8566389Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:30:43.8567263Z Uploading artifacts took 0.00 seconds
2025-07-24T04:30:47.0473644Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.0477062Z   import pkg_resources
2025-07-24T04:30:47.0669654Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.0672210Z   import pkg_resources
2025-07-24T04:30:47.0733362Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.0735938Z   import pkg_resources
2025-07-24T04:30:47.1073815Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.1076421Z   import pkg_resources
2025-07-24T04:30:47.1177832Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.1180463Z   import pkg_resources
2025-07-24T04:30:47.1270023Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.1272547Z   import pkg_resources
2025-07-24T04:30:47.1336303Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.1338989Z   import pkg_resources
2025-07-24T04:30:47.1341393Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:47.1343796Z   import pkg_resources
2025-07-24T04:30:48.8059840Z Running test batch 'tests to run' cost 7.42 seconds
2025-07-24T04:30:49.3614676Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest
2025-07-24T04:30:52.0526381Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:52.0529204Z   import pkg_resources
2025-07-24T04:30:53.8432759Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:30:53.8636659Z Found test times from artifacts
2025-07-24T04:30:53.9497661Z Found test times from artifacts
2025-07-24T04:30:53.9531600Z Running 25% of tests based on TD
2025-07-24T04:30:53.9535278Z Running parallel tests on 8 processes
2025-07-24T04:30:53.9537053Z Name: tests to run (est. time: 0.0min)
2025-07-24T04:30:53.9537634Z   Serial tests (1):
2025-07-24T04:30:53.9538097Z     cpp/HashStoreTest 1/1
2025-07-24T04:30:53.9538592Z   Parallel tests (0):
2025-07-24T04:30:53.9539063Z Name: excluded (est. time: 0.0min)
2025-07-24T04:30:53.9539595Z   Serial tests (0):
2025-07-24T04:30:53.9540017Z   Parallel tests (0):
2025-07-24T04:30:53.9540642Z Running cpp/HashStoreTest 1/1 ... [2025-07-24 04:30:53.953283]
2025-07-24T04:30:53.9541429Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:30:53.9543880Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-79f6982067aec5db.xml', '-x', '--reruns=2'] ... [2025-07-24 04:30:53.953839]
2025-07-24T04:30:56.1761409Z 
2025-07-24T04:30:56.1763264Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_d984431efa8a7c81_.log
2025-07-24T04:30:56.1764655Z 
2025-07-24T04:30:56.1765108Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:30:56.1765923Z Uploading artifacts took 0.00 seconds
2025-07-24T04:30:59.3893435Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.3894997Z   import pkg_resources
2025-07-24T04:30:59.4131126Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4132885Z   import pkg_resources
2025-07-24T04:30:59.4200332Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4201997Z   import pkg_resources
2025-07-24T04:30:59.4292500Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4294285Z   import pkg_resources
2025-07-24T04:30:59.4328769Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4330135Z   import pkg_resources
2025-07-24T04:30:59.4403549Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4404968Z   import pkg_resources
2025-07-24T04:30:59.4443629Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4445418Z   import pkg_resources
2025-07-24T04:30:59.4471310Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:30:59.4472626Z   import pkg_resources
2025-07-24T04:31:01.0912418Z Running test batch 'tests to run' cost 7.14 seconds
2025-07-24T04:31:01.6942043Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest
2025-07-24T04:31:04.3788064Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:04.3790721Z   import pkg_resources
2025-07-24T04:31:06.1451260Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:31:06.1653493Z Found test times from artifacts
2025-07-24T04:31:06.2512426Z Found test times from artifacts
2025-07-24T04:31:06.2543721Z Running 25% of tests based on TD
2025-07-24T04:31:06.2546983Z Running parallel tests on 8 processes
2025-07-24T04:31:06.2547629Z Name: tests to run (est. time: 0.0min)
2025-07-24T04:31:06.2548221Z   Serial tests (1):
2025-07-24T04:31:06.2548689Z     cpp/TCPStoreTest 1/1
2025-07-24T04:31:06.2549178Z   Parallel tests (0):
2025-07-24T04:31:06.2549663Z Name: excluded (est. time: 0.0min)
2025-07-24T04:31:06.2550188Z   Serial tests (0):
2025-07-24T04:31:06.2550602Z   Parallel tests (0):
2025-07-24T04:31:06.2551219Z Running cpp/TCPStoreTest 1/1 ... [2025-07-24 04:31:06.254533]
2025-07-24T04:31:06.2551960Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:31:06.2555820Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-d136dcb3f4c64f63.xml', '-x', '--reruns=2'] ... [2025-07-24 04:31:06.255030]
2025-07-24T04:31:18.4027066Z 
2025-07-24T04:31:18.4028695Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_d1fe964ab6e64ad5_.log
2025-07-24T04:31:18.4030078Z 
2025-07-24T04:31:18.4031477Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:31:18.4032395Z Uploading artifacts took 0.00 seconds
2025-07-24T04:31:21.5409263Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.5410874Z   import pkg_resources
2025-07-24T04:31:21.5775256Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.5776884Z   import pkg_resources
2025-07-24T04:31:21.6906574Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.6907944Z   import pkg_resources
2025-07-24T04:31:21.7023048Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.7025244Z   import pkg_resources
2025-07-24T04:31:21.7057371Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.7059000Z   import pkg_resources
2025-07-24T04:31:21.7095604Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.7097290Z   import pkg_resources
2025-07-24T04:31:21.7122083Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.7123369Z   import pkg_resources
2025-07-24T04:31:21.7146859Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:21.7148546Z   import pkg_resources
2025-07-24T04:31:23.0420640Z Running test batch 'tests to run' cost 16.79 seconds
2025-07-24T04:31:23.6407618Z + echo 'Testing multi-GPU linalg tests'
2025-07-24T04:31:23.6408611Z Testing multi-GPU linalg tests
2025-07-24T04:31:23.6410352Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose
2025-07-24T04:31:26.2840729Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:26.2844225Z   import pkg_resources
2025-07-24T04:31:28.0756989Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:31:28.0957115Z Found test times from artifacts
2025-07-24T04:31:28.1816515Z Found test times from artifacts
2025-07-24T04:31:28.1847850Z Running 25% of tests based on TD
2025-07-24T04:31:28.1851909Z Running parallel tests on 8 processes
2025-07-24T04:31:28.1852569Z Name: tests to run (est. time: 0.19min)
2025-07-24T04:31:28.1853179Z   Serial tests (1):
2025-07-24T04:31:28.1853603Z     test_linalg 1/1
2025-07-24T04:31:28.1854059Z   Parallel tests (0):
2025-07-24T04:31:28.1854537Z Name: excluded (est. time: 0.0min)
2025-07-24T04:31:28.1855071Z   Serial tests (0):
2025-07-24T04:31:28.1855497Z   Parallel tests (0):
2025-07-24T04:31:28.1856062Z Running test_linalg 1/1 ... [2025-07-24 04:31:28.184885]
2025-07-24T04:31:28.1856751Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:31:28.1858835Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-07-24 04:31:28.185154]
2025-07-24T04:31:43.6878086Z 
2025-07-24T04:31:43.6879642Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_817dc1031940fc10_.log
2025-07-24T04:31:43.6882423Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32
2025-07-24T04:31:43.6883316Z 
2025-07-24T04:31:43.6883797Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:31:43.6885313Z Uploading artifacts took 0.00 seconds
2025-07-24T04:31:46.9628610Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9631150Z   import pkg_resources
2025-07-24T04:31:46.9695988Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9698579Z   import pkg_resources
2025-07-24T04:31:46.9745871Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9748452Z   import pkg_resources
2025-07-24T04:31:46.9904581Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9907309Z   import pkg_resources
2025-07-24T04:31:46.9921936Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9924645Z   import pkg_resources
2025-07-24T04:31:46.9978348Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9980948Z   import pkg_resources
2025-07-24T04:31:46.9995787Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:46.9998613Z   import pkg_resources
2025-07-24T04:31:47.0024414Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:47.0026966Z   import pkg_resources
2025-07-24T04:31:48.5645229Z Running test batch 'tests to run' cost 20.38 seconds
2025-07-24T04:31:49.1387438Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-07-24T04:31:49.1388071Z + [[ 1 == 1 ]]
2025-07-24T04:31:49.1388457Z + test_rpc
2025-07-24T04:31:49.1388860Z + echo 'Testing RPC C++ tests'
2025-07-24T04:31:49.1389351Z Testing RPC C++ tests
2025-07-24T04:31:49.1393429Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:31:49.1438822Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:31:49.1478287Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-07-24T04:31:49.1479330Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc
2025-07-24T04:31:51.7793327Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:51.7795833Z   import pkg_resources
2025-07-24T04:31:53.5708763Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-07-24T04:31:53.5906973Z Found test times from artifacts
2025-07-24T04:31:53.6753604Z Found test times from artifacts
2025-07-24T04:31:53.6784283Z Running 25% of tests based on TD
2025-07-24T04:31:53.6785450Z Running parallel tests on 8 processes
2025-07-24T04:31:53.6786103Z Name: tests to run (est. time: 0.0min)
2025-07-24T04:31:53.6786856Z   Serial tests (1):
2025-07-24T04:31:53.6787373Z     cpp/test_cpp_rpc 1/1
2025-07-24T04:31:53.6787925Z   Parallel tests (0):
2025-07-24T04:31:53.6788413Z Name: excluded (est. time: 0.0min)
2025-07-24T04:31:53.6788915Z   Serial tests (0):
2025-07-24T04:31:53.6789386Z   Parallel tests (0):
2025-07-24T04:31:53.6790024Z Running cpp/test_cpp_rpc 1/1 ... [2025-07-24 04:31:53.678557]
2025-07-24T04:31:53.6790744Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-07-24T04:31:53.6795730Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-2f96e33c56520b83.xml', '-x', '--reruns=2'] ... [2025-07-24 04:31:53.678958]
2025-07-24T04:31:55.0495377Z 
2025-07-24T04:31:55.0496856Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_ba4fe5ad70c177ad_.log
2025-07-24T04:31:55.0497921Z 
2025-07-24T04:31:55.0498380Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-07-24T04:31:55.0499242Z Uploading artifacts took 0.00 seconds
2025-07-24T04:31:58.1380473Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.1381829Z   import pkg_resources
2025-07-24T04:31:58.2698686Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.2700200Z   import pkg_resources
2025-07-24T04:31:58.3051697Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3053868Z   import pkg_resources
2025-07-24T04:31:58.3067012Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3068293Z   import pkg_resources
2025-07-24T04:31:58.3072917Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3074443Z   import pkg_resources
2025-07-24T04:31:58.3212045Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3213716Z   import pkg_resources
2025-07-24T04:31:58.3258467Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3260108Z   import pkg_resources
2025-07-24T04:31:58.3266239Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-07-24T04:31:58.3267796Z   import pkg_resources
2025-07-24T04:31:59.8803031Z Running test batch 'tests to run' cost 6.2 seconds
2025-07-24T04:32:00.4418159Z + sccache_epilogue
2025-07-24T04:32:00.4418804Z + echo '::group::Sccache Compilation Log'
2025-07-24T04:32:00.4421472Z ##[group]Sccache Compilation Log
2025-07-24T04:32:00.4422145Z + echo '=================== sccache compilation log ==================='
2025-07-24T04:32:00.4423728Z =================== sccache compilation log ===================
2025-07-24T04:32:00.4424860Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-07-24T04:32:00.4713811Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-07-24T04:32:00.4715206Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-07-24T04:32:00.4716127Z + sccache --show-stats
2025-07-24T04:32:00.4776174Z Compile requests                      0
2025-07-24T04:32:00.4776895Z Compile requests executed             0
2025-07-24T04:32:00.4777539Z Cache hits                            0
2025-07-24T04:32:00.4778196Z Cache misses                          0
2025-07-24T04:32:00.4778853Z Cache hits rate                       -
2025-07-24T04:32:00.4779482Z Cache timeouts                        0
2025-07-24T04:32:00.4780068Z Cache read errors                     0
2025-07-24T04:32:00.4780620Z Forced recaches                       0
2025-07-24T04:32:00.4781165Z Cache write errors                    0
2025-07-24T04:32:00.4781710Z Cache errors                          0
2025-07-24T04:32:00.4782240Z Compilations                          0
2025-07-24T04:32:00.4782775Z Compilation failures                  0
2025-07-24T04:32:00.4783330Z Non-cacheable compilations            0
2025-07-24T04:32:00.4783882Z Non-cacheable calls                   0
2025-07-24T04:32:00.4784430Z Non-compilation calls                 0
2025-07-24T04:32:00.4785667Z Unsupported compiler calls            0
2025-07-24T04:32:00.4786244Z Average cache write               0.000 s
2025-07-24T04:32:00.4786830Z Average compiler                  0.000 s
2025-07-24T04:32:00.4787402Z Average cache read hit            0.000 s
2025-07-24T04:32:00.4787972Z Failed distributed compilations       0
2025-07-24T04:32:00.4788703Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-07-24T04:32:00.4789491Z Use direct/preprocessor mode?   yes
2025-07-24T04:32:00.4790076Z Version (client)                0.10.0
2025-07-24T04:32:00.4790644Z Max cache size                       10 GiB
2025-07-24T04:32:00.4791224Z + sccache --stop-server
2025-07-24T04:32:00.4847723Z Stopping sccache server...
2025-07-24T04:32:00.4854051Z Compile requests                      0
2025-07-24T04:32:00.4854794Z Compile requests executed             0
2025-07-24T04:32:00.4855361Z Cache hits                            0
2025-07-24T04:32:00.4855897Z Cache misses                          0
2025-07-24T04:32:00.4856492Z Cache hits rate                       -
2025-07-24T04:32:00.4857035Z Cache timeouts                        0
2025-07-24T04:32:00.4857566Z Cache read errors                     0
2025-07-24T04:32:00.4858119Z Forced recaches                       0
2025-07-24T04:32:00.4858668Z Cache write errors                    0
2025-07-24T04:32:00.4859184Z Cache errors                          0
2025-07-24T04:32:00.4859723Z Compilations                          0
2025-07-24T04:32:00.4860270Z Compilation failures                  0
2025-07-24T04:32:00.4860815Z Non-cacheable compilations            0
2025-07-24T04:32:00.4861370Z Non-cacheable calls                   0
2025-07-24T04:32:00.4861909Z Non-compilation calls                 0
2025-07-24T04:32:00.4862454Z Unsupported compiler calls            0
2025-07-24T04:32:00.4863019Z Average cache write               0.000 s
2025-07-24T04:32:00.4863611Z Average compiler                  0.000 s
2025-07-24T04:32:00.4864204Z Average cache read hit            0.000 s
2025-07-24T04:32:00.4864780Z Failed distributed compilations       0
2025-07-24T04:32:00.4865525Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-07-24T04:32:00.4866325Z Use direct/preprocessor mode?   yes
2025-07-24T04:32:00.4866914Z Version (client)                0.10.0
2025-07-24T04:32:00.4867486Z Max cache size                       10 GiB
2025-07-24T04:32:00.4871450Z + echo ::endgroup::
2025-07-24T04:32:00.4872983Z ##[endgroup]
2025-07-24T04:32:00.5034079Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-07-24T04:32:00.5035596Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-07-24T04:32:00.5037361Z docker exec -t "0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-07-24T04:32:00.5090536Z shell: /usr/bin/bash -e {0}
2025-07-24T04:32:00.5091006Z env:
2025-07-24T04:32:00.5106714Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:00.5107625Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:00.5108755Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:00.5109795Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:00.5111596Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:00.5113129Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:00.5113612Z   AWS_REGION: us-east-1
2025-07-24T04:32:00.5114162Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:00.5114823Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:00.5125989Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:00.5126758Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:00.5128065Z ##[endgroup]
2025-07-24T04:32:00.6633780Z ##[group]Run cat test/**/*_toprint.log || true
2025-07-24T04:32:00.6634470Z cat test/**/*_toprint.log || true
2025-07-24T04:32:00.6689392Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:00.6690045Z env:
2025-07-24T04:32:00.6690421Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:00.6691143Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:00.6692253Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:00.6693246Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:00.6694924Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:00.6696423Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:00.6696929Z   AWS_REGION: us-east-1
2025-07-24T04:32:00.6697540Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:00.6698207Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:00.6708885Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:00.6709635Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:00.6710451Z ##[endgroup]
2025-07-24T04:32:00.7021633Z cat: 'test/**/*_toprint.log': No such file or directory
2025-07-24T04:32:00.7246162Z Prepare all required actions
2025-07-24T04:32:00.7247282Z Getting action download info
2025-07-24T04:32:00.9586306Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-07-24T04:32:01.6206043Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-07-24T04:32:02.3796498Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-07-24T04:32:02.3796755Z with:
2025-07-24T04:32:02.3796916Z   use-gha: true
2025-07-24T04:32:02.3797189Z   file-suffix: test-distributed-1-1-linux.rocm.gpu.4_46615439125
2025-07-24T04:32:02.3797479Z   s3-bucket: gha-artifacts
2025-07-24T04:32:02.3797670Z env:
2025-07-24T04:32:02.3797829Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:02.3798132Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:02.3798570Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:02.3799031Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:02.3799732Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:02.3800362Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:02.3800572Z   AWS_REGION: us-east-1
2025-07-24T04:32:02.3800808Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:02.3801087Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:02.3805494Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:02.3805796Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:02.3806187Z ##[endgroup]
2025-07-24T04:32:02.3870572Z ##[group]Run actions/upload-artifact@v4
2025-07-24T04:32:02.3870836Z with:
2025-07-24T04:32:02.3871225Z   name: test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip
2025-07-24T04:32:02.3871747Z   retention-days: 14
2025-07-24T04:32:02.3872003Z   if-no-files-found: warn
2025-07-24T04:32:02.3872268Z   path: test/**/*.json
2025-07-24T04:32:02.3872512Z   compression-level: 6
2025-07-24T04:32:02.3872751Z   overwrite: false
2025-07-24T04:32:02.3873003Z   include-hidden-files: false
2025-07-24T04:32:02.3873246Z env:
2025-07-24T04:32:02.3873427Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:02.3873769Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:02.3874207Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:02.3874813Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:02.3875505Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:02.3876115Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:02.3876316Z   AWS_REGION: us-east-1
2025-07-24T04:32:02.3876526Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:02.3876790Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:02.3881133Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:02.3881439Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:02.3881762Z ##[endgroup]
2025-07-24T04:32:03.2037904Z With the provided path, there will be 11 files uploaded
2025-07-24T04:32:03.2043546Z Artifact name is valid!
2025-07-24T04:32:03.2044118Z Root directory input is valid!
2025-07-24T04:32:03.4302914Z Beginning upload of artifact content to blob storage
2025-07-24T04:32:03.6973829Z Uploaded bytes 43612
2025-07-24T04:32:03.7438738Z Finished uploading artifact content to blob storage!
2025-07-24T04:32:03.7443740Z SHA256 digest of uploaded artifact zip is a000e2e4ea6371bf9464e83716d7da2fb9f57fc2b2dc0eda185003ca910c7f65
2025-07-24T04:32:03.7446181Z Finalizing artifact upload
2025-07-24T04:32:03.8437119Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip.zip successfully finalized. Artifact ID 3603072926
2025-07-24T04:32:03.8439431Z Artifact test-jsons-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip has been successfully uploaded! Final size is 43612 bytes. Artifact ID is 3603072926
2025-07-24T04:32:03.8449711Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16487210679/artifacts/3603072926
2025-07-24T04:32:03.8807644Z ##[group]Run actions/upload-artifact@v4
2025-07-24T04:32:03.8808345Z with:
2025-07-24T04:32:03.8809208Z   name: test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip
2025-07-24T04:32:03.8810223Z   retention-days: 14
2025-07-24T04:32:03.8810728Z   if-no-files-found: ignore
2025-07-24T04:32:03.8811287Z   path: test/**/*.xml
test/**/*.csv

2025-07-24T04:32:03.8811880Z   compression-level: 6
2025-07-24T04:32:03.8812361Z   overwrite: false
2025-07-24T04:32:03.8812830Z   include-hidden-files: false
2025-07-24T04:32:03.8813335Z env:
2025-07-24T04:32:03.8813760Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:03.8814548Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:03.8815738Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:03.8816816Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:03.8818630Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:03.8820487Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:03.8821083Z   AWS_REGION: us-east-1
2025-07-24T04:32:03.8821683Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:03.8822382Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:03.8833718Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:03.8834538Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:03.8835413Z ##[endgroup]
2025-07-24T04:32:04.7623946Z With the provided path, there will be 34 files uploaded
2025-07-24T04:32:04.7629978Z Artifact name is valid!
2025-07-24T04:32:04.7630547Z Root directory input is valid!
2025-07-24T04:32:04.9416420Z Beginning upload of artifact content to blob storage
2025-07-24T04:32:05.1814685Z Uploaded bytes 17498
2025-07-24T04:32:05.2352343Z Finished uploading artifact content to blob storage!
2025-07-24T04:32:05.2357928Z SHA256 digest of uploaded artifact zip is ae15c61a2341ba89670d42708baa1bcdc30664b1aa7518915a0f7f041b17ee67
2025-07-24T04:32:05.2360224Z Finalizing artifact upload
2025-07-24T04:32:05.3435986Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip.zip successfully finalized. Artifact ID 3603073028
2025-07-24T04:32:05.3438322Z Artifact test-reports-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip has been successfully uploaded! Final size is 17498 bytes. Artifact ID is 3603073028
2025-07-24T04:32:05.3449946Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16487210679/artifacts/3603073028
2025-07-24T04:32:05.3839940Z ##[group]Run actions/upload-artifact@v4
2025-07-24T04:32:05.3840557Z with:
2025-07-24T04:32:05.3841305Z   name: logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip
2025-07-24T04:32:05.3842206Z   retention-days: 14
2025-07-24T04:32:05.3842707Z   if-no-files-found: ignore
2025-07-24T04:32:05.3843265Z   path: usage_log.txt
test/**/*.log

2025-07-24T04:32:05.3843848Z   compression-level: 6
2025-07-24T04:32:05.3844333Z   overwrite: false
2025-07-24T04:32:05.3845234Z   include-hidden-files: false
2025-07-24T04:32:05.3845755Z env:
2025-07-24T04:32:05.3846157Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:05.3846942Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:05.3848108Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:05.3849695Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:05.3851557Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:05.3853221Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:05.3853784Z   AWS_REGION: us-east-1
2025-07-24T04:32:05.3854402Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:05.3855137Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:05.3866477Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:05.3867324Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:05.3868219Z ##[endgroup]
2025-07-24T04:32:06.3015382Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-07-24T04:32:06.3018333Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-07-24T04:32:06.3019862Z With the provided path, there will be 6 files uploaded
2025-07-24T04:32:06.3023033Z Artifact name is valid!
2025-07-24T04:32:06.3024020Z Root directory input is valid!
2025-07-24T04:32:06.4782365Z Beginning upload of artifact content to blob storage
2025-07-24T04:32:06.6604317Z Uploaded bytes 9222
2025-07-24T04:32:06.7136498Z Finished uploading artifact content to blob storage!
2025-07-24T04:32:06.7141435Z SHA256 digest of uploaded artifact zip is 2e9c790e3337ff5e9b340a03870fb76c89ec96f8bb5cc5bb0f248b96c29aebf0
2025-07-24T04:32:06.7143765Z Finalizing artifact upload
2025-07-24T04:32:06.8280560Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip.zip successfully finalized. Artifact ID 3603073126
2025-07-24T04:32:06.8282700Z Artifact logs-runattempt1-test-distributed-1-1-linux.rocm.gpu.4_46615439125.zip has been successfully uploaded! Final size is 9222 bytes. Artifact ID is 3603073126
2025-07-24T04:32:06.8296743Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16487210679/artifacts/3603073126
2025-07-24T04:32:06.8680681Z ##[group]Run # shellcheck disable=SC2156
2025-07-24T04:32:06.8681553Z # shellcheck disable=SC2156
2025-07-24T04:32:06.8682673Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-07-24T04:32:06.8740053Z shell: /usr/bin/bash -e {0}
2025-07-24T04:32:06.8740610Z env:
2025-07-24T04:32:06.8741056Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:06.8741901Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:06.8743634Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:06.8744764Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:06.8746637Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:06.8748364Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:06.8748923Z   AWS_REGION: us-east-1
2025-07-24T04:32:06.8749553Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:06.8750296Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:06.8761688Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:06.8762526Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:06.8763424Z ##[endgroup]
2025-07-24T04:32:07.2721005Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-07-24T04:32:07.2722044Z with:
2025-07-24T04:32:07.2722840Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
2025-07-24T04:32:07.2723771Z   role-duration-seconds: 18000
2025-07-24T04:32:07.2724339Z   aws-region: us-east-1
2025-07-24T04:32:07.2725272Z   audience: sts.amazonaws.com
2025-07-24T04:32:07.2725808Z env:
2025-07-24T04:32:07.2726222Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:07.2727042Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:07.2728240Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:07.2729359Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:07.2731285Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:07.2732938Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:07.2733502Z   AWS_REGION: us-east-1
2025-07-24T04:32:07.2734045Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:07.2734672Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:07.2744208Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:07.2744919Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:07.2745688Z ##[endgroup]
2025-07-24T04:32:07.5840494Z Assuming role with OIDC
2025-07-24T04:32:07.7560630Z Authenticated as assumedRoleId AROAUPVRELQNA5GQHA6IA:GitHubActions
2025-07-24T04:32:07.8275689Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-07-24T04:32:07.8276597Z with:
2025-07-24T04:32:07.8277097Z   benchmark-results-dir: test/test-reports
2025-07-24T04:32:07.8277721Z   dry-run: false
2025-07-24T04:32:07.8278183Z   schema-version: v3
2025-07-24T04:32:07.8279201Z   github-token: ***
2025-07-24T04:32:07.8279681Z env:
2025-07-24T04:32:07.8280109Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:07.8280929Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:07.8282116Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:07.8283213Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:07.8285161Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:07.8286939Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:07.8287611Z   AWS_REGION: us-east-1
2025-07-24T04:32:07.8288309Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:07.8289063Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:07.8299054Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:07.8299566Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:07.8300107Z ##[endgroup]
2025-07-24T04:32:07.8334136Z ##[group]Run set -eux
2025-07-24T04:32:07.8334728Z set -eux
2025-07-24T04:32:07.8335857Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-07-24T04:32:07.8336643Z 
2025-07-24T04:32:07.8337128Z DEVICE_NAME=""
2025-07-24T04:32:07.8337729Z DEVICE_TYPE=""
2025-07-24T04:32:07.8338291Z 
2025-07-24T04:32:07.8338843Z if command -v nvidia-smi; then
2025-07-24T04:32:07.8339840Z   # NB: I'm using PyTorch here to get the device name, however, it needs to
2025-07-24T04:32:07.8341090Z   # install the correct version of PyTorch manually for now. Any PyTorch
2025-07-24T04:32:07.8341849Z   # version is fine, I just use 2.7.1 to satify PYPIDEP linter
2025-07-24T04:32:07.8342340Z   python3 -mpip install torch==2.7.1
2025-07-24T04:32:07.8342738Z elif command -v rocminfo; then
2025-07-24T04:32:07.8343238Z   # NB: Installing torch on ROCm runner with pip here causes CI to fail
2025-07-24T04:32:07.8343855Z   # with a memoryview is too large error only on MI300 runners. Is pip
2025-07-24T04:32:07.8344472Z   # version on ROCm runner there too old? As a workaround, let's use the
2025-07-24T04:32:07.8345028Z   # GPU device name coming from rocminfo instead
2025-07-24T04:32:07.8345450Z   DEVICE_NAME=rocm
2025-07-24T04:32:07.8345992Z   DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs)
2025-07-24T04:32:07.8346549Z fi
2025-07-24T04:32:07.8346812Z 
2025-07-24T04:32:07.8347147Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV
2025-07-24T04:32:07.8347703Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV
2025-07-24T04:32:07.8398825Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:07.8399553Z env:
2025-07-24T04:32:07.8399984Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:07.8400803Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:07.8401980Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:07.8403086Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:07.8405034Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:07.8406662Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:07.8407826Z   AWS_REGION: us-east-1
2025-07-24T04:32:07.8408563Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:07.8409351Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:07.8419325Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:07.8419831Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:07.8420369Z ##[endgroup]
2025-07-24T04:32:07.8509926Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-07-24T04:32:08.1369894Z Defaulting to user installation because normal site-packages is not writeable
2025-07-24T04:32:08.2974850Z Requirement already satisfied: boto3==1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (1.35.33)
2025-07-24T04:32:08.2978590Z Requirement already satisfied: psutil==7.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (7.0.0)
2025-07-24T04:32:08.2984217Z Requirement already satisfied: pynvml==12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (12.0.0)
2025-07-24T04:32:08.3018042Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from boto3==1.35.33) (0.10.0)
2025-07-24T04:32:08.3023882Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (1.35.99)
2025-07-24T04:32:08.3027434Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (0.10.4)
2025-07-24T04:32:08.3179654Z Requirement already satisfied: nvidia-ml-py<13.0.0a0,>=12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from pynvml==12.0.0) (12.575.51)
2025-07-24T04:32:08.3227495Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.26.5)
2025-07-24T04:32:08.3231851Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.9.0.post0)
2025-07-24T04:32:08.3274747Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.16.0)
2025-07-24T04:32:08.6053672Z + DEVICE_NAME=
2025-07-24T04:32:08.6054222Z + DEVICE_TYPE=
2025-07-24T04:32:08.6054717Z + command -v nvidia-smi
2025-07-24T04:32:08.6055256Z + command -v rocminfo
2025-07-24T04:32:08.6055759Z + DEVICE_NAME=rocm
2025-07-24T04:32:08.6056255Z /opt/rocm/bin/rocminfo
2025-07-24T04:32:08.6073471Z ++ rocminfo
2025-07-24T04:32:08.6076669Z ++ grep 'Marketing Name'
2025-07-24T04:32:08.6080564Z ++ tail -n1
2025-07-24T04:32:08.6083727Z ++ awk -F: '{print $2}'
2025-07-24T04:32:08.6087243Z ++ xargs
2025-07-24T04:32:08.7640846Z + DEVICE_TYPE='AMD Instinct MI250X/MI250'
2025-07-24T04:32:08.7641570Z + echo DEVICE_NAME=rocm
2025-07-24T04:32:08.7642152Z + echo 'DEVICE_TYPE=AMD Instinct MI250X/MI250'
2025-07-24T04:32:08.7692611Z ##[group]Run set -eux
2025-07-24T04:32:08.7693157Z set -eux
2025-07-24T04:32:08.7693641Z 
2025-07-24T04:32:08.7694127Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-07-24T04:32:08.7694924Z   echo "Missing github-token input"
2025-07-24T04:32:08.7695644Z   exit 1
2025-07-24T04:32:08.7696153Z fi
2025-07-24T04:32:08.7754113Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:08.7754846Z env:
2025-07-24T04:32:08.7755290Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:08.7756094Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:08.7757308Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:08.7758410Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:08.7760208Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:08.7762241Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:08.7762826Z   AWS_REGION: us-east-1
2025-07-24T04:32:08.7763464Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:08.7764220Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:08.7774938Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:08.7775931Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:08.7776986Z   DEVICE_NAME: rocm
2025-07-24T04:32:08.7777592Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:08.7778491Z   GITHUB_TOKEN: ***
2025-07-24T04:32:08.7778982Z ##[endgroup]
2025-07-24T04:32:08.7880026Z + [[ -z *** ]]
2025-07-24T04:32:08.7967981Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-07-24T04:32:08.7968825Z with:
2025-07-24T04:32:08.7969524Z   github-token: ***
2025-07-24T04:32:08.7970015Z env:
2025-07-24T04:32:08.7970491Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:08.7971323Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:08.7972534Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:08.7973623Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:08.7975400Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:08.7976996Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:08.7978024Z   AWS_REGION: us-east-1
2025-07-24T04:32:08.7978600Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:08.7979341Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:08.7989463Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:08.7990288Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:08.7991184Z   DEVICE_NAME: rocm
2025-07-24T04:32:08.7991707Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:08.7992298Z ##[endgroup]
2025-07-24T04:32:08.8056419Z ##[group]Run set -eux
2025-07-24T04:32:08.8056961Z set -eux
2025-07-24T04:32:08.8057407Z 
2025-07-24T04:32:08.8058288Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-07-24T04:32:08.8111684Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:08.8112331Z env:
2025-07-24T04:32:08.8112722Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:08.8113426Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:08.8114471Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:08.8115434Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:08.8116984Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:08.8118394Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:08.8118898Z   AWS_REGION: us-east-1
2025-07-24T04:32:08.8119419Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:08.8120091Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:08.8129317Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:08.8130063Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:08.8130847Z   DEVICE_NAME: rocm
2025-07-24T04:32:08.8131312Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:08.8132015Z   GITHUB_TOKEN: ***
2025-07-24T04:32:08.8132453Z ##[endgroup]
2025-07-24T04:32:08.8220967Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16487210679 gpu9455
2025-07-24T04:32:09.7572325Z setting job-id=46615439125
2025-07-24T04:32:09.7573297Z setting job-name=linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:32:09.7818947Z ##[group]Run set -eux
2025-07-24T04:32:09.7819897Z set -eux
2025-07-24T04:32:09.7820360Z 
2025-07-24T04:32:09.7821115Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-07-24T04:32:09.7822075Z   --schema-version "${SCHEMA_VERSION}" \
2025-07-24T04:32:09.7822741Z   --repo "${REPO}" \
2025-07-24T04:32:09.7823368Z   --head-branch "${HEAD_BRANCH}" \
2025-07-24T04:32:09.7824005Z   --head-sha "${HEAD_SHA}" \
2025-07-24T04:32:09.7824642Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-07-24T04:32:09.7825363Z   --run-attempt "${RUN_ATTEMPT}" \
2025-07-24T04:32:09.7825964Z   --job-id "${JOB_ID}" \
2025-07-24T04:32:09.7826537Z   --job-name "${JOB_NAME}"
2025-07-24T04:32:09.7879209Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:09.7879916Z env:
2025-07-24T04:32:09.7880350Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:09.7881162Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:09.7882305Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:09.7883363Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:09.7885426Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:09.7887015Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:09.7888013Z   AWS_REGION: us-east-1
2025-07-24T04:32:09.7888613Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:09.7889351Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:09.7899244Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:09.7900084Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:09.7900976Z   DEVICE_NAME: rocm
2025-07-24T04:32:09.7901512Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:09.7902128Z   SCHEMA_VERSION: v3
2025-07-24T04:32:09.7902629Z   REPO: pytorch/pytorch
2025-07-24T04:32:09.7903183Z   HEAD_BRANCH: refs/tags/ciflow/trunk/149961
2025-07-24T04:32:09.7903891Z   HEAD_SHA: 27c8ef1839e0747b36769efb192befc9308ff93c
2025-07-24T04:32:09.7904570Z   WORKFLOW_RUN_ID: 16487210679
2025-07-24T04:32:09.7905099Z   RUN_ATTEMPT: 1
2025-07-24T04:32:09.7905561Z   JOB_ID: 46615439125
2025-07-24T04:32:09.7906310Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)
2025-07-24T04:32:09.7907153Z ##[endgroup]
2025-07-24T04:32:09.8003785Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/tags/ciflow/trunk/149961 --head-sha 27c8ef1839e0747b36769efb192befc9308ff93c --workflow-id 16487210679 --run-attempt 1 --job-id 46615439125 --job-name 'linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)'
2025-07-24T04:32:09.8597150Z ##[group]Run set -eux
2025-07-24T04:32:09.8597689Z set -eux
2025-07-24T04:32:09.8598154Z 
2025-07-24T04:32:09.8598910Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py"
2025-07-24T04:32:09.8652707Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:09.8653583Z env:
2025-07-24T04:32:09.8654117Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:09.8655096Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:09.8656544Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:09.8657858Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:09.8659731Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:09.8661745Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:09.8662339Z   AWS_REGION: us-east-1
2025-07-24T04:32:09.8662992Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:09.8663788Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:09.8674367Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:09.8675211Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:09.8676112Z   DEVICE_NAME: rocm
2025-07-24T04:32:09.8676628Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:09.8677246Z ##[endgroup]
2025-07-24T04:32:09.8766543Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py
2025-07-24T04:32:10.6318276Z /var/home/pytorchci/.local/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:276: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.)
2025-07-24T04:32:10.6320591Z   cpu = _conversion_method_template(device=torch.device("cpu"))
2025-07-24T04:32:11.2489244Z ##[group]Run set -eux
2025-07-24T04:32:11.2489887Z set -eux
2025-07-24T04:32:11.2490433Z 
2025-07-24T04:32:11.2491017Z # TODO (huydhn): Implement this part
2025-07-24T04:32:11.2491913Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-07-24T04:32:11.2548446Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:11.2549177Z env:
2025-07-24T04:32:11.2549630Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:11.2550966Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:11.2552171Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:11.2553291Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:11.2555135Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:11.2556740Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:11.2557322Z   AWS_REGION: us-east-1
2025-07-24T04:32:11.2557948Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:11.2558703Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:11.2568816Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:11.2569806Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:11.2570846Z   DEVICE_NAME: rocm
2025-07-24T04:32:11.2571474Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:11.2572177Z ##[endgroup]
2025-07-24T04:32:11.2662271Z + echo 'dependencies={}'
2025-07-24T04:32:11.2696088Z ##[group]Run set -eux
2025-07-24T04:32:11.2696416Z set -eux
2025-07-24T04:32:11.2696664Z 
2025-07-24T04:32:11.2696947Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-07-24T04:32:11.2697388Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-07-24T04:32:11.2697894Z   # We don't want the job to fail if the directory doesn't exist
2025-07-24T04:32:11.2698292Z   exit 0
2025-07-24T04:32:11.2698527Z fi
2025-07-24T04:32:11.2698762Z 
2025-07-24T04:32:11.2699015Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-07-24T04:32:11.2699481Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-07-24T04:32:11.2700031Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-07-24T04:32:11.2700557Z     --metadata "${BENCHMARK_METADATA}" \
2025-07-24T04:32:11.2701270Z     --runners "${RUNNER_INFO}" \
2025-07-24T04:32:11.2702066Z     --dependencies "${DEPENDENCIES}" \
2025-07-24T04:32:11.2702795Z     --dry-run
2025-07-24T04:32:11.2703377Z else
2025-07-24T04:32:11.2704098Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-07-24T04:32:11.2705557Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-07-24T04:32:11.2706426Z     --metadata "${BENCHMARK_METADATA}" \
2025-07-24T04:32:11.2707107Z     --runners "${RUNNER_INFO}" \
2025-07-24T04:32:11.2707785Z     --dependencies "${DEPENDENCIES}"
2025-07-24T04:32:11.2708399Z fi
2025-07-24T04:32:11.2757975Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:11.2758723Z env:
2025-07-24T04:32:11.2759162Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:11.2759987Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:11.2761201Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:11.2762288Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:11.2764098Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:11.2766078Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:11.2766649Z   AWS_REGION: us-east-1
2025-07-24T04:32:11.2767291Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:11.2768043Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:11.2776808Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:11.2777261Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:11.2777726Z   DEVICE_NAME: rocm
2025-07-24T04:32:11.2777992Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:11.2778615Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-07-24T04:32:11.2778932Z   DRY_RUN: false
2025-07-24T04:32:11.2780145Z   BENCHMARK_METADATA: {"timestamp": 1753331529, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/trunk/149961", "head_sha": "27c8ef1839e0747b36769efb192befc9308ff93c", "workflow_id": 16487210679, "run_attempt": 1, "job_id": 46615439125}
2025-07-24T04:32:11.2783216Z   RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpu9455.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]
2025-07-24T04:32:11.2784619Z   DEPENDENCIES: {}
2025-07-24T04:32:11.2785087Z ##[endgroup]
2025-07-24T04:32:11.2871698Z + [[ ! -d test/test-reports ]]
2025-07-24T04:32:11.2872255Z + [[ false == \t\r\u\e ]]
2025-07-24T04:32:11.2876701Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1753331529, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 1, 1, linux.rocm.gpu.4)", "repo": "pytorch/pytorch", "head_branch": "refs/tags/ciflow/trunk/149961", "head_sha": "27c8ef1839e0747b36769efb192befc9308ff93c", "workflow_id": 16487210679, "run_attempt": 1, "job_id": 46615439125}' --runners '[{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpu9455.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]' --dependencies '{}'
2025-07-24T04:32:11.4510277Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/FileStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-d1677fd015cd7446b76b.json is not a benchmark record, skipping
2025-07-24T04:32:11.4513359Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4521504Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'test_linalg'}], 'excluded': []} from test/test-reports/td_exclusions-5c22e902b87075fb4963.json is not a benchmark record, skipping
2025-07-24T04:32:11.4524369Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4527724Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'distributed/test_c10d_common'}], 'excluded': [{'test_file': 'distributed/test_c10d_nccl'}]} from test/test-reports/td_exclusions-34f65f903a5df8ce93e0.json is not a benchmark record, skipping
2025-07-24T04:32:11.4530880Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4533699Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/TCPStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-7d1f426b340cbd71c042.json is not a benchmark record, skipping
2025-07-24T04:32:11.4536511Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4539295Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/test_cpp_rpc'}], 'excluded': []} from test/test-reports/td_exclusions-09d1b9b169715b5b89a0.json is not a benchmark record, skipping
2025-07-24T04:32:11.4542102Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4545315Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/HashStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-571b3b3d8511e117ab7a.json is not a benchmark record, skipping
2025-07-24T04:32:11.4548119Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-07-24T04:32:11.4744129Z Prepare all required actions
2025-07-24T04:32:11.4745020Z Getting action download info
2025-07-24T04:32:11.4795806Z ##[group]Run ./.github/actions/teardown-rocm
2025-07-24T04:32:11.4796431Z env:
2025-07-24T04:32:11.4796880Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:11.4797688Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:11.4798847Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:11.4799960Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:11.4801738Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:11.4803383Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:11.4803961Z   AWS_REGION: us-east-1
2025-07-24T04:32:11.4804942Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:11.4805792Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:11.4816444Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:11.4817434Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:11.4818303Z   DEVICE_NAME: rocm
2025-07-24T04:32:11.4818618Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:11.4818950Z ##[endgroup]
2025-07-24T04:32:11.4845774Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-07-24T04:32:11.4846804Z # ignore expansion of "docker ps -q" since it could be empty
2025-07-24T04:32:11.4847606Z # shellcheck disable=SC2046
2025-07-24T04:32:11.4848286Z docker stop $(docker ps -q) || true
2025-07-24T04:32:11.4848987Z # Prune all stopped containers.
2025-07-24T04:32:11.4849619Z docker container prune -f
2025-07-24T04:32:11.4899298Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:11.4899674Z env:
2025-07-24T04:32:11.4899910Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:11.4900333Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:11.4900921Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:11.4901481Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:11.4902390Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:11.4903204Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:11.4903503Z   AWS_REGION: us-east-1
2025-07-24T04:32:11.4903819Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:11.4904202Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:11.4914077Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:11.4914892Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:11.4915757Z   DEVICE_NAME: rocm
2025-07-24T04:32:11.4916935Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:11.4917845Z ##[endgroup]
2025-07-24T04:32:22.2450570Z 0675e8934e3d
2025-07-24T04:32:29.4498049Z Deleted Containers:
2025-07-24T04:32:29.4498548Z 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:29.4498886Z 
2025-07-24T04:32:29.4499005Z Total reclaimed space: 10.03GB
2025-07-24T04:32:29.4610916Z Prepare all required actions
2025-07-24T04:32:29.4671591Z ##[group]Run ./.github/actions/diskspace-cleanup
2025-07-24T04:32:29.4672751Z with:
2025-07-24T04:32:29.4673234Z   diskspace-cutoff: 70
2025-07-24T04:32:29.4673739Z env:
2025-07-24T04:32:29.4674185Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:29.4675039Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:29.4676260Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:29.4677402Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:29.4679773Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:29.4681499Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:29.4682096Z   AWS_REGION: us-east-1
2025-07-24T04:32:29.4682760Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:29.4683555Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:29.4694453Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:29.4695378Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:29.4696314Z   DEVICE_NAME: rocm
2025-07-24T04:32:29.4696869Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:29.4697495Z ##[endgroup]
2025-07-24T04:32:29.4730240Z ##[group]Run set -ex
2025-07-24T04:32:29.4730832Z set -ex
2025-07-24T04:32:29.4731334Z diskspace_cutoff=70
2025-07-24T04:32:29.4732121Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-07-24T04:32:29.4732952Z if [ ! -d "$docker_root_dir" ]; then
2025-07-24T04:32:29.4734020Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-07-24T04:32:29.4735004Z     exit 0
2025-07-24T04:32:29.4735489Z fi
2025-07-24T04:32:29.4736365Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-24T04:32:29.4738110Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-07-24T04:32:29.4739617Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-07-24T04:32:29.4740400Z     docker system prune -af
2025-07-24T04:32:29.4741395Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-07-24T04:32:29.4742525Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-07-24T04:32:29.4743707Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-07-24T04:32:29.4744733Z         echo "$msg"
2025-07-24T04:32:29.4745294Z         exit 1
2025-07-24T04:32:29.4745802Z     else
2025-07-24T04:32:29.4746402Z         difference=$((diskspace - diskspace_new))
2025-07-24T04:32:29.4747233Z         echo "Diskspace saved: $difference percent"
2025-07-24T04:32:29.4747908Z     fi
2025-07-24T04:32:29.4748379Z fi
2025-07-24T04:32:29.4804345Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-07-24T04:32:29.4805507Z env:
2025-07-24T04:32:29.4805998Z   GIT_DEFAULT_BRANCH: main
2025-07-24T04:32:29.4806858Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-07-24T04:32:29.4808089Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-07-24T04:32:29.4809253Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-07-24T04:32:29.4811095Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-07-24T04:32:29.4812610Z   AWS_DEFAULT_REGION: us-east-1
2025-07-24T04:32:29.4813096Z   AWS_REGION: us-east-1
2025-07-24T04:32:29.4813624Z   AWS_ACCESS_KEY_ID: ***
2025-07-24T04:32:29.4814282Z   AWS_SECRET_ACCESS_KEY: ***
2025-07-24T04:32:29.4823362Z   AWS_SESSION_TOKEN: ***
2025-07-24T04:32:29.4824091Z   CONTAINER_NAME: 0675e8934e3d3835c770adc3bddee15f7949bb638a9e09182bd94bb1b081b2f8
2025-07-24T04:32:29.4824865Z   DEVICE_NAME: rocm
2025-07-24T04:32:29.4825330Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-07-24T04:32:29.4825849Z ##[endgroup]
2025-07-24T04:32:29.4913014Z + diskspace_cutoff=70
2025-07-24T04:32:29.4921908Z ++ docker info -f '{{.DockerRootDir}}'
2025-07-24T04:32:29.5805430Z + docker_root_dir=/media/4TB/docker-rootless
2025-07-24T04:32:29.5806251Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-07-24T04:32:29.5821016Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-07-24T04:32:29.5823420Z ++ sed -n 2p
2025-07-24T04:32:29.5827621Z ++ sed s/%//
2025-07-24T04:32:29.5830304Z ++ sed 's/ //'
2025-07-24T04:32:29.5873965Z + diskspace=38
2025-07-24T04:32:29.5875231Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-07-24T04:32:29.5876507Z + [[ 38 -ge 70 ]]
2025-07-24T04:32:29.5970003Z Post job cleanup.
2025-07-24T04:32:29.6063676Z Post job cleanup.
2025-07-24T04:32:29.7499468Z Post job cleanup.
2025-07-24T04:32:29.8189023Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-07-24T04:32:29.8811210Z Post job cleanup.
2025-07-24T04:32:30.0362013Z Post job cleanup.
2025-07-24T04:32:30.0444156Z Post job cleanup.
2025-07-24T04:32:30.1595934Z [command]/usr/bin/git version
2025-07-24T04:32:30.1653395Z git version 2.34.1
2025-07-24T04:32:30.1686598Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/748aa094-b8aa-40fc-b55d-4277927bbc69/.gitconfig'
2025-07-24T04:32:30.1699151Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/748aa094-b8aa-40fc-b55d-4277927bbc69' before making global git config changes
2025-07-24T04:32:30.1701122Z Adding repository directory to the temporary git global config as a safe directory
2025-07-24T04:32:30.1714003Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-07-24T04:32:30.1785368Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-07-24T04:32:30.1853545Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-07-24T04:32:30.2563724Z Entering 'android/libs/fbjni'
2025-07-24T04:32:30.2713530Z Entering 'third_party/FP16'
2025-07-24T04:32:30.2840825Z Entering 'third_party/FXdiv'
2025-07-24T04:32:30.2970214Z Entering 'third_party/NNPACK'
2025-07-24T04:32:30.3096214Z Entering 'third_party/NVTX'
2025-07-24T04:32:30.3223260Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:32:30.3348775Z Entering 'third_party/XNNPACK'
2025-07-24T04:32:30.3512078Z Entering 'third_party/aiter'
2025-07-24T04:32:30.3637984Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:32:30.3790209Z Entering 'third_party/benchmark'
2025-07-24T04:32:30.3919270Z Entering 'third_party/composable_kernel'
2025-07-24T04:32:30.4062227Z Entering 'third_party/cpp-httplib'
2025-07-24T04:32:30.4188812Z Entering 'third_party/cpuinfo'
2025-07-24T04:32:30.4321295Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:32:30.4447723Z Entering 'third_party/cutlass'
2025-07-24T04:32:30.4595328Z Entering 'third_party/fbgemm'
2025-07-24T04:32:30.4721845Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:32:30.4843533Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:32:30.4981870Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:32:30.5101177Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:32:30.5240967Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:32:30.5361834Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:32:30.5480146Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:32:30.5609002Z Entering 'third_party/flash-attention'
2025-07-24T04:32:30.5733865Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:32:30.5869927Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:32:30.6011509Z Entering 'third_party/flatbuffers'
2025-07-24T04:32:30.6146097Z Entering 'third_party/fmt'
2025-07-24T04:32:30.6275653Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:32:30.6403524Z Entering 'third_party/gloo'
2025-07-24T04:32:30.6530681Z Entering 'third_party/googletest'
2025-07-24T04:32:30.6657642Z Entering 'third_party/ideep'
2025-07-24T04:32:30.6777946Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:32:30.6922824Z Entering 'third_party/ittapi'
2025-07-24T04:32:30.7049423Z Entering 'third_party/kineto'
2025-07-24T04:32:30.7173122Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:32:30.7289541Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:32:30.7413427Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:32:30.7531085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:32:30.7650010Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:32:30.7770518Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:32:30.7897159Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:32:30.8014084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:32:30.8131047Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:32:30.8253245Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:32:30.8376297Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:32:30.8489493Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:32:30.8612464Z Entering 'third_party/kleidiai'
2025-07-24T04:32:30.8740534Z Entering 'third_party/mimalloc'
2025-07-24T04:32:30.8866977Z Entering 'third_party/nlohmann'
2025-07-24T04:32:30.8997882Z Entering 'third_party/onnx'
2025-07-24T04:32:30.9157206Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:32:30.9291639Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:32:30.9415440Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:32:30.9533529Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:32:30.9649189Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:32:30.9764043Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:32:30.9886696Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:32:31.0004683Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:32:31.0121006Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:32:31.0232785Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:32:31.0356104Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:32:31.0478646Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:32:31.0647348Z Entering 'third_party/pocketfft'
2025-07-24T04:32:31.0770418Z Entering 'third_party/protobuf'
2025-07-24T04:32:31.0896292Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:32:31.1013164Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:32:31.1137420Z Entering 'third_party/psimd'
2025-07-24T04:32:31.1267253Z Entering 'third_party/pthreadpool'
2025-07-24T04:32:31.1396643Z Entering 'third_party/pybind11'
2025-07-24T04:32:31.1521533Z Entering 'third_party/python-peachpy'
2025-07-24T04:32:31.1651535Z Entering 'third_party/sleef'
2025-07-24T04:32:31.1781118Z Entering 'third_party/tensorpipe'
2025-07-24T04:32:31.1905298Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:32:31.2023188Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:32:31.2141197Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:32:31.2258412Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:32:31.2372687Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:32:31.2547788Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-07-24T04:32:31.2600916Z http.https://github.com/.extraheader
2025-07-24T04:32:31.2623807Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-07-24T04:32:31.2702406Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-07-24T04:32:31.3414891Z Entering 'android/libs/fbjni'
2025-07-24T04:32:31.3484991Z http.https://github.com/.extraheader
2025-07-24T04:32:31.3584611Z Entering 'third_party/FP16'
2025-07-24T04:32:31.3651351Z http.https://github.com/.extraheader
2025-07-24T04:32:31.3748959Z Entering 'third_party/FXdiv'
2025-07-24T04:32:31.3818853Z http.https://github.com/.extraheader
2025-07-24T04:32:31.3912685Z Entering 'third_party/NNPACK'
2025-07-24T04:32:31.3980134Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4073205Z Entering 'third_party/NVTX'
2025-07-24T04:32:31.4140756Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4236202Z Entering 'third_party/VulkanMemoryAllocator'
2025-07-24T04:32:31.4303163Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4399848Z Entering 'third_party/XNNPACK'
2025-07-24T04:32:31.4466432Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4597304Z Entering 'third_party/aiter'
2025-07-24T04:32:31.4663257Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4753987Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-07-24T04:32:31.4816933Z http.https://github.com/.extraheader
2025-07-24T04:32:31.4935948Z Entering 'third_party/benchmark'
2025-07-24T04:32:31.5000817Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5101608Z Entering 'third_party/composable_kernel'
2025-07-24T04:32:31.5171639Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5286015Z Entering 'third_party/cpp-httplib'
2025-07-24T04:32:31.5352832Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5449361Z Entering 'third_party/cpuinfo'
2025-07-24T04:32:31.5515967Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5615361Z Entering 'third_party/cudnn_frontend'
2025-07-24T04:32:31.5683915Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5782985Z Entering 'third_party/cutlass'
2025-07-24T04:32:31.5850916Z http.https://github.com/.extraheader
2025-07-24T04:32:31.5968683Z Entering 'third_party/fbgemm'
2025-07-24T04:32:31.6037106Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6135348Z Entering 'third_party/fbgemm/external/asmjit'
2025-07-24T04:32:31.6198472Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6292219Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-07-24T04:32:31.6353144Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6463558Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-07-24T04:32:31.6524729Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6618706Z Entering 'third_party/fbgemm/external/cutlass'
2025-07-24T04:32:31.6679664Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6791868Z Entering 'third_party/fbgemm/external/googletest'
2025-07-24T04:32:31.6852883Z http.https://github.com/.extraheader
2025-07-24T04:32:31.6948132Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-07-24T04:32:31.7010983Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7100179Z Entering 'third_party/fbgemm/external/json'
2025-07-24T04:32:31.7163641Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7268176Z Entering 'third_party/flash-attention'
2025-07-24T04:32:31.7338009Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7435001Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-07-24T04:32:31.7495785Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7601924Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-07-24T04:32:31.7661309Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7780614Z Entering 'third_party/flatbuffers'
2025-07-24T04:32:31.7843785Z http.https://github.com/.extraheader
2025-07-24T04:32:31.7946961Z Entering 'third_party/fmt'
2025-07-24T04:32:31.8015021Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8113170Z Entering 'third_party/gemmlowp/gemmlowp'
2025-07-24T04:32:31.8177937Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8274553Z Entering 'third_party/gloo'
2025-07-24T04:32:31.8337909Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8432493Z Entering 'third_party/googletest'
2025-07-24T04:32:31.8495007Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8590739Z Entering 'third_party/ideep'
2025-07-24T04:32:31.8655283Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8741410Z Entering 'third_party/ideep/mkl-dnn'
2025-07-24T04:32:31.8800844Z http.https://github.com/.extraheader
2025-07-24T04:32:31.8913582Z Entering 'third_party/ittapi'
2025-07-24T04:32:31.8980317Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9081640Z Entering 'third_party/kineto'
2025-07-24T04:32:31.9146204Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9242364Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-07-24T04:32:31.9306375Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9395260Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-07-24T04:32:31.9456072Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9553234Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-07-24T04:32:31.9614492Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9707855Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-07-24T04:32:31.9768027Z http.https://github.com/.extraheader
2025-07-24T04:32:31.9859845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-07-24T04:32:31.9918806Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0005510Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-07-24T04:32:32.0066594Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0163668Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-07-24T04:32:32.0223473Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0315551Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-07-24T04:32:32.0374391Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0466994Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-07-24T04:32:32.0526973Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0618801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-07-24T04:32:32.0677541Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0774619Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-07-24T04:32:32.0834745Z http.https://github.com/.extraheader
2025-07-24T04:32:32.0923724Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-07-24T04:32:32.0983109Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1081448Z Entering 'third_party/kleidiai'
2025-07-24T04:32:32.1145602Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1243235Z Entering 'third_party/mimalloc'
2025-07-24T04:32:32.1315187Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1409276Z Entering 'third_party/nlohmann'
2025-07-24T04:32:32.1473528Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1571557Z Entering 'third_party/onnx'
2025-07-24T04:32:32.1637961Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1762167Z Entering 'third_party/onnx/third_party/pybind11'
2025-07-24T04:32:32.1823354Z http.https://github.com/.extraheader
2025-07-24T04:32:32.1923440Z Entering 'third_party/opentelemetry-cpp'
2025-07-24T04:32:32.1986509Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2078472Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-07-24T04:32:32.2137363Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2223693Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-07-24T04:32:32.2281327Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2370564Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-07-24T04:32:32.2429220Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2515979Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-07-24T04:32:32.2576069Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2669916Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-07-24T04:32:32.2729373Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2817930Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-07-24T04:32:32.2877426Z http.https://github.com/.extraheader
2025-07-24T04:32:32.2966705Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-07-24T04:32:32.3025752Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3111620Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-07-24T04:32:32.3170631Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3262813Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-07-24T04:32:32.3323415Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3420146Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-07-24T04:32:32.3482029Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3617422Z Entering 'third_party/pocketfft'
2025-07-24T04:32:32.3682100Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3779509Z Entering 'third_party/protobuf'
2025-07-24T04:32:32.3844355Z http.https://github.com/.extraheader
2025-07-24T04:32:32.3947873Z Entering 'third_party/protobuf/third_party/benchmark'
2025-07-24T04:32:32.4010290Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4100860Z Entering 'third_party/protobuf/third_party/googletest'
2025-07-24T04:32:32.4160260Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4256648Z Entering 'third_party/psimd'
2025-07-24T04:32:32.4320678Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4415054Z Entering 'third_party/pthreadpool'
2025-07-24T04:32:32.4478906Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4569288Z Entering 'third_party/pybind11'
2025-07-24T04:32:32.4634502Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4727586Z Entering 'third_party/python-peachpy'
2025-07-24T04:32:32.4792554Z http.https://github.com/.extraheader
2025-07-24T04:32:32.4883694Z Entering 'third_party/sleef'
2025-07-24T04:32:32.4949073Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5040600Z Entering 'third_party/tensorpipe'
2025-07-24T04:32:32.5107927Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5197384Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-07-24T04:32:32.5259210Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5348061Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-07-24T04:32:32.5408755Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5496546Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-07-24T04:32:32.5555949Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5652034Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-07-24T04:32:32.5706042Z http.https://github.com/.extraheader
2025-07-24T04:32:32.5793582Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-07-24T04:32:32.5852941Z http.https://github.com/.extraheader
2025-07-24T04:32:32.6267314Z Cleaning up orphan processes