2025-08-14T21:23:39.7890237Z Current runner version: '2.328.0'
2025-08-14T21:23:39.7896650Z Runner name: 'gpud515'
2025-08-14T21:23:39.7897491Z Runner group name: 'linux.rocm.gpu.group'
2025-08-14T21:23:39.7898418Z Machine name: 'gpud515'
2025-08-14T21:23:39.7901365Z ##[group]GITHUB_TOKEN Permissions
2025-08-14T21:23:39.7903413Z Contents: read
2025-08-14T21:23:39.7904052Z Metadata: read
2025-08-14T21:23:39.7904663Z ##[endgroup]
2025-08-14T21:23:39.7906979Z Secret source: Actions
2025-08-14T21:23:39.7907799Z Prepare workflow directory
2025-08-14T21:23:40.2204282Z Prepare all required actions
2025-08-14T21:23:40.2286978Z Getting action download info
2025-08-14T21:23:40.6728329Z Download action repository 'pytorch/pytorch@main' (SHA:3be70dc30e893b552fc0f23ca06cd8f7949b6d08)
2025-08-14T21:23:45.9118169Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-08-14T21:23:46.6163524Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-08-14T21:23:47.2068598Z Download action repository 'pytorch/test-infra@main' (SHA:83f58f391e939c10dcb8cb6d745e4cefa3b98a84)
2025-08-14T21:23:48.3251914Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-08-14T21:23:49.1771634Z Getting action download info
2025-08-14T21:23:49.3932441Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955)
2025-08-14T21:23:50.0735732Z Getting action download info
2025-08-14T21:23:50.2361126Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-08-14T21:23:51.0066122Z Getting action download info
2025-08-14T21:23:51.1747468Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/main (1fc683cf17c8c673044538d10266c00f92987be2)
2025-08-14T21:23:51.1751995Z ##[group] Inputs
2025-08-14T21:23:51.1752365Z   build-environment: linux-jammy-rocm-py3.10
2025-08-14T21:23:51.1753738Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:23:51.1755351Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:23:51.1756006Z   sync-tag: 
2025-08-14T21:23:51.1756765Z   timeout-minutes: 300
2025-08-14T21:23:51.1757035Z   tests-to-include: 
2025-08-14T21:23:51.1757259Z   dashboard-tag: 
2025-08-14T21:23:51.1757791Z   disable-monitor: true
2025-08-14T21:23:51.1758070Z   monitor-log-interval: 5
2025-08-14T21:23:51.1758341Z   monitor-data-collect-interval: 1
2025-08-14T21:23:51.1758612Z ##[endgroup]
2025-08-14T21:23:51.1759083Z Complete job name: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:23:51.3688377Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-08-14T21:23:51.3689109Z with:
2025-08-14T21:23:51.3689307Z   no-sudo: true
2025-08-14T21:23:51.3689528Z   submodules: recursive
2025-08-14T21:23:51.3689746Z   fetch-depth: 0
2025-08-14T21:23:51.3690140Z env:
2025-08-14T21:23:51.3690357Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:23:51.3690586Z ##[endgroup]
2025-08-14T21:23:51.3774538Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-08-14T21:23:51.3775387Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-08-14T21:23:51.3801498Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:23:51.3802295Z env:
2025-08-14T21:23:51.3802481Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:23:51.3802706Z ##[endgroup]
2025-08-14T21:23:51.4030812Z ##[group]Run # Use all available CPUs for fetching
2025-08-14T21:23:51.4031250Z # Use all available CPUs for fetching
2025-08-14T21:23:51.4031573Z cd "${GITHUB_WORKSPACE}"
2025-08-14T21:23:51.4031892Z git config --global fetch.parallel 0
2025-08-14T21:23:51.4032230Z git config --global submodule.fetchJobs 0
2025-08-14T21:23:51.4032541Z 
2025-08-14T21:23:51.4032851Z # Clean workspace. The default checkout action should also do this, but
2025-08-14T21:23:51.4033249Z # do it here as well just in case
2025-08-14T21:23:51.4033544Z if [[ -d .git ]]; then
2025-08-14T21:23:51.4033804Z   if [ -z "${NO_SUDO}" ]; then
2025-08-14T21:23:51.4034068Z     sudo git clean -ffdx
2025-08-14T21:23:51.4034316Z   else
2025-08-14T21:23:51.4034538Z     git clean -ffdx
2025-08-14T21:23:51.4034778Z   fi
2025-08-14T21:23:51.4034972Z fi
2025-08-14T21:23:51.4055570Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:23:51.4055979Z env:
2025-08-14T21:23:51.4056209Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:23:51.4056473Z   NO_SUDO: true
2025-08-14T21:23:51.4056670Z ##[endgroup]
2025-08-14T21:23:51.8918335Z Removing .additional_ci_files/
2025-08-14T21:23:51.8918748Z Removing build/
2025-08-14T21:23:51.8918976Z Removing dist/
2025-08-14T21:23:51.8919201Z Removing test/test-reports/
2025-08-14T21:23:51.8985701Z ##[group]Run actions/checkout@v4
2025-08-14T21:23:51.8985996Z with:
2025-08-14T21:23:51.8986242Z   ref: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:23:51.8986536Z   fetch-depth: 0
2025-08-14T21:23:51.8986764Z   submodules: recursive
2025-08-14T21:23:51.8986990Z   show-progress: false
2025-08-14T21:23:51.8987227Z   repository: pytorch/pytorch
2025-08-14T21:23:51.8987631Z   token: ***
2025-08-14T21:23:51.8987838Z   ssh-strict: true
2025-08-14T21:23:51.8988034Z   ssh-user: git
2025-08-14T21:23:51.8988258Z   persist-credentials: true
2025-08-14T21:23:51.8988531Z   clean: true
2025-08-14T21:23:51.8988756Z   sparse-checkout-cone-mode: true
2025-08-14T21:23:51.8989015Z   fetch-tags: false
2025-08-14T21:23:51.8989198Z   lfs: false
2025-08-14T21:23:51.8989396Z   set-safe-directory: true
2025-08-14T21:23:51.8989612Z env:
2025-08-14T21:23:51.8989793Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:23:51.8989998Z ##[endgroup]
2025-08-14T21:23:52.0089655Z Syncing repository: pytorch/pytorch
2025-08-14T21:23:52.0090933Z ##[group]Getting Git version info
2025-08-14T21:23:52.0091421Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2025-08-14T21:23:52.0092025Z [command]/usr/bin/git version
2025-08-14T21:23:52.0103521Z git version 2.34.1
2025-08-14T21:23:52.0128772Z ##[endgroup]
2025-08-14T21:23:52.0137701Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/cae2fede-9e89-4dcc-a27c-09ba6a4448f2/.gitconfig'
2025-08-14T21:23:52.0148183Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/cae2fede-9e89-4dcc-a27c-09ba6a4448f2' before making global git config changes
2025-08-14T21:23:52.0149069Z Adding repository directory to the temporary git global config as a safe directory
2025-08-14T21:23:52.0161051Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:23:52.0215406Z [command]/usr/bin/git config --local --get remote.origin.url
2025-08-14T21:23:52.0242335Z https://github.com/pytorch/pytorch
2025-08-14T21:23:52.0295680Z ##[group]Removing previously created refs, to avoid conflicts
2025-08-14T21:23:52.0296204Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-08-14T21:23:52.0296614Z HEAD
2025-08-14T21:23:52.0345371Z ##[endgroup]
2025-08-14T21:23:52.0348866Z [command]/usr/bin/git submodule status
2025-08-14T21:23:52.0785446Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-08-14T21:23:52.0914929Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-08-14T21:23:52.1034682Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-08-14T21:23:52.1170252Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-08-14T21:23:52.1232648Z  2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07 third_party/NVTX (v3.1.0-263-g2942f16)
2025-08-14T21:23:52.1332164Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-08-14T21:23:52.1896149Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-08-14T21:23:52.1945449Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-08-14T21:23:52.1986841Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-08-14T21:23:52.2086899Z  7fe50dc3da2069d6645d9deb8c017a876472a977 third_party/composable_kernel (rocm-6.4.3-459-g7fe50dc3d)
2025-08-14T21:23:52.2252539Z  3af7f2c16147f3fbc6e4d717032daf505dc1652c third_party/cpp-httplib (v0.20.1)
2025-08-14T21:23:52.2422953Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-08-14T21:23:52.2475702Z  f937055efc6d414d11f4c6577e3977fe74f35fb6 third_party/cudnn_frontend (v0.5-52-gf937055)
2025-08-14T21:23:52.2601843Z  e51efbfe18fe4f4cbb66ab814c55bf4aa0185491 third_party/cutlass (v4.1.0)
2025-08-14T21:23:52.2666526Z  21c7d30c526c0f1ad873ecc632dca6cfa8a69067 third_party/fbgemm (v1.3.0-rc1-165-g21c7d30c)
2025-08-14T21:23:52.2783817Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-08-14T21:23:52.2821331Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-08-14T21:23:52.3260442Z  40626af88bd7df9a5fb80be7b25ac85b122d6c21 third_party/fmt (11.2.0)
2025-08-14T21:23:52.3415926Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-08-14T21:23:52.3589410Z  c7b7b022c124d9643957d9bd55f57ac59fce8fa2 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-33-gc7b7b02)
2025-08-14T21:23:52.3840493Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-08-14T21:23:52.3954260Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-08-14T21:23:52.4040783Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-08-14T21:23:52.4350115Z  5e7501833f1021ce6f618572d3baf657b6319658 third_party/kineto (remotes/origin/sraikund/test-98-g5e75018)
2025-08-14T21:23:52.4392069Z  cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7 third_party/kleidiai (v1.8.0)
2025-08-14T21:23:52.4430999Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-08-14T21:23:52.4469264Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-08-14T21:23:52.4831770Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-08-14T21:23:52.4871156Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-08-14T21:23:52.4915479Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-08-14T21:23:52.5330890Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-08-14T21:23:52.5463033Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-08-14T21:23:52.5545923Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-08-14T21:23:52.5611775Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-08-14T21:23:52.5736931Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-08-14T21:23:52.5842092Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-08-14T21:23:52.5956726Z  dacda0567d9f23d4bc503e1c4f84aa65f33ac38a third_party/tensorpipe (heads/main)
2025-08-14T21:23:52.5979794Z ##[group]Cleaning the repository
2025-08-14T21:23:52.5985010Z [command]/usr/bin/git clean -ffdx
2025-08-14T21:23:52.6352132Z [command]/usr/bin/git reset --hard HEAD
2025-08-14T21:23:52.8477914Z HEAD is now at 884e299f170 Update on "[associative_scan] support gen_schema for associative_scan"
2025-08-14T21:23:52.8522297Z ##[endgroup]
2025-08-14T21:23:52.8524027Z ##[group]Disabling automatic garbage collection
2025-08-14T21:23:52.8528685Z [command]/usr/bin/git config --local gc.auto 0
2025-08-14T21:23:52.8568136Z ##[endgroup]
2025-08-14T21:23:52.8568822Z ##[group]Setting up auth
2025-08-14T21:23:52.8573533Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-08-14T21:23:52.8617703Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-08-14T21:23:52.8999285Z Entering 'android/libs/fbjni'
2025-08-14T21:23:52.9077558Z Entering 'third_party/FP16'
2025-08-14T21:23:52.9146880Z Entering 'third_party/FXdiv'
2025-08-14T21:23:52.9220143Z Entering 'third_party/NNPACK'
2025-08-14T21:23:52.9298484Z Entering 'third_party/NVTX'
2025-08-14T21:23:52.9378227Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:23:52.9446941Z Entering 'third_party/XNNPACK'
2025-08-14T21:23:52.9532192Z Entering 'third_party/aiter'
2025-08-14T21:23:52.9600131Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:23:52.9689471Z Entering 'third_party/benchmark'
2025-08-14T21:23:52.9759274Z Entering 'third_party/composable_kernel'
2025-08-14T21:23:52.9838650Z Entering 'third_party/cpp-httplib'
2025-08-14T21:23:52.9904401Z Entering 'third_party/cpuinfo'
2025-08-14T21:23:52.9980272Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:23:53.0058390Z Entering 'third_party/cutlass'
2025-08-14T21:23:53.0137151Z Entering 'third_party/fbgemm'
2025-08-14T21:23:53.0208330Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:23:53.0280157Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:23:53.0357055Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:23:53.0422765Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:23:53.0490740Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:23:53.0560333Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:23:53.0625879Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:23:53.0699231Z Entering 'third_party/flash-attention'
2025-08-14T21:23:53.0763753Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:23:53.0834426Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:23:53.0919560Z Entering 'third_party/flatbuffers'
2025-08-14T21:23:53.0991292Z Entering 'third_party/fmt'
2025-08-14T21:23:53.1060679Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:23:53.1124726Z Entering 'third_party/gloo'
2025-08-14T21:23:53.1192615Z Entering 'third_party/googletest'
2025-08-14T21:23:53.1256100Z Entering 'third_party/ideep'
2025-08-14T21:23:53.1329431Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:23:53.1405697Z Entering 'third_party/ittapi'
2025-08-14T21:23:53.1473920Z Entering 'third_party/kineto'
2025-08-14T21:23:53.1544948Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:23:53.1620236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:23:53.1695895Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:23:53.1771936Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:23:53.1834670Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:23:53.1905101Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:23:53.1988386Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:23:53.2054988Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:23:53.2132282Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:23:53.2201809Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:23:53.2271670Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:23:53.2336149Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:23:53.2398563Z Entering 'third_party/kleidiai'
2025-08-14T21:23:53.2470556Z Entering 'third_party/mimalloc'
2025-08-14T21:23:53.2543648Z Entering 'third_party/nlohmann'
2025-08-14T21:23:53.2617073Z Entering 'third_party/onnx'
2025-08-14T21:23:53.2710951Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:23:53.2785688Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:23:53.2857713Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:23:53.2928190Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:23:53.2990707Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:23:53.3054851Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:23:53.3118508Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:23:53.3189917Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:23:53.3257152Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:23:53.3330685Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:23:53.3399231Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:23:53.3476709Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:23:53.3573606Z Entering 'third_party/pocketfft'
2025-08-14T21:23:53.3643008Z Entering 'third_party/protobuf'
2025-08-14T21:23:53.3711720Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:23:53.3786032Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:23:53.3863629Z Entering 'third_party/psimd'
2025-08-14T21:23:53.3937389Z Entering 'third_party/pthreadpool'
2025-08-14T21:23:53.4015917Z Entering 'third_party/pybind11'
2025-08-14T21:23:53.4085071Z Entering 'third_party/python-peachpy'
2025-08-14T21:23:53.4153308Z Entering 'third_party/sleef'
2025-08-14T21:23:53.4220220Z Entering 'third_party/tensorpipe'
2025-08-14T21:23:53.4295288Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:23:53.4356934Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:23:53.4425681Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:23:53.4496258Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:23:53.4555608Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:23:53.4663599Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-08-14T21:23:53.4700928Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-08-14T21:23:53.5082225Z Entering 'android/libs/fbjni'
2025-08-14T21:23:53.5154784Z Entering 'third_party/FP16'
2025-08-14T21:23:53.5227832Z Entering 'third_party/FXdiv'
2025-08-14T21:23:53.5300187Z Entering 'third_party/NNPACK'
2025-08-14T21:23:53.5378670Z Entering 'third_party/NVTX'
2025-08-14T21:23:53.5448312Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:23:53.5517272Z Entering 'third_party/XNNPACK'
2025-08-14T21:23:53.5604829Z Entering 'third_party/aiter'
2025-08-14T21:23:53.5675724Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:23:53.5762260Z Entering 'third_party/benchmark'
2025-08-14T21:23:53.5834645Z Entering 'third_party/composable_kernel'
2025-08-14T21:23:53.5915853Z Entering 'third_party/cpp-httplib'
2025-08-14T21:23:53.5986112Z Entering 'third_party/cpuinfo'
2025-08-14T21:23:53.6059899Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:23:53.6132319Z Entering 'third_party/cutlass'
2025-08-14T21:23:53.6220756Z Entering 'third_party/fbgemm'
2025-08-14T21:23:53.6295550Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:23:53.6369428Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:23:53.6441530Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:23:53.6506210Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:23:53.6591576Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:23:53.6659750Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:23:53.6730189Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:23:53.6795838Z Entering 'third_party/flash-attention'
2025-08-14T21:23:53.6867384Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:23:53.6942740Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:23:53.7028634Z Entering 'third_party/flatbuffers'
2025-08-14T21:23:53.7104384Z Entering 'third_party/fmt'
2025-08-14T21:23:53.7173470Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:23:53.7252049Z Entering 'third_party/gloo'
2025-08-14T21:23:53.7321011Z Entering 'third_party/googletest'
2025-08-14T21:23:53.7389667Z Entering 'third_party/ideep'
2025-08-14T21:23:53.7456950Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:23:53.7533505Z Entering 'third_party/ittapi'
2025-08-14T21:23:53.7601394Z Entering 'third_party/kineto'
2025-08-14T21:23:53.7669012Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:23:53.7733359Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:23:53.7794756Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:23:53.7862277Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:23:53.7932786Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:23:53.7990500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:23:53.8061040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:23:53.8128424Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:23:53.8189899Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:23:53.8258178Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:23:53.8329659Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:23:53.8393820Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:23:53.8461641Z Entering 'third_party/kleidiai'
2025-08-14T21:23:53.8536655Z Entering 'third_party/mimalloc'
2025-08-14T21:23:53.8614934Z Entering 'third_party/nlohmann'
2025-08-14T21:23:53.8684995Z Entering 'third_party/onnx'
2025-08-14T21:23:53.8769114Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:23:53.8841167Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:23:53.8913479Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:23:53.8983870Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:23:53.9055502Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:23:53.9117371Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:23:53.9185719Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:23:53.9255975Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:23:53.9315617Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:23:53.9377700Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:23:53.9439481Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:23:53.9511465Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:23:53.9603397Z Entering 'third_party/pocketfft'
2025-08-14T21:23:53.9673371Z Entering 'third_party/protobuf'
2025-08-14T21:23:53.9747757Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:23:53.9816866Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:23:53.9883818Z Entering 'third_party/psimd'
2025-08-14T21:23:53.9953210Z Entering 'third_party/pthreadpool'
2025-08-14T21:23:54.0026496Z Entering 'third_party/pybind11'
2025-08-14T21:23:54.0097967Z Entering 'third_party/python-peachpy'
2025-08-14T21:23:54.0157717Z Entering 'third_party/sleef'
2025-08-14T21:23:54.0228108Z Entering 'third_party/tensorpipe'
2025-08-14T21:23:54.0298001Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:23:54.0359377Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:23:54.0426208Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:23:54.0497930Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:23:54.0555643Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:23:54.0660131Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-08-14T21:23:54.0706965Z ##[endgroup]
2025-08-14T21:23:54.0707316Z ##[group]Fetching the repository
2025-08-14T21:23:54.0714189Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-08-14T21:23:54.7979430Z From https://github.com/pytorch/pytorch
2025-08-14T21:23:54.7980162Z  - [deleted]                 (none)     -> ciflow/inductor/159530
2025-08-14T21:23:54.8776353Z  - [deleted]                 (none)     -> ciflow/trunk/159530
2025-08-14T21:23:56.0683096Z  + 81069eac44e...9e777070cd7 gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig  (forced update)
2025-08-14T21:23:56.0690475Z  + 83d85c0e02b...65bcf0cf22c gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig  (forced update)
2025-08-14T21:23:56.0692647Z    13d33dae859..d6b43a6cd4e  gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head
2025-08-14T21:23:56.0695352Z  + 7a68526b8bf...0768ffd59f3 gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig  (forced update)
2025-08-14T21:23:56.0697464Z    22ef8f43f77..87732b2c98e  gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base
2025-08-14T21:23:56.0699778Z    81ef0f46d54..5556c80b778  gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head
2025-08-14T21:23:56.0702192Z  + 9c9dea5ab2a...c6b3f286785 gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig  (forced update)
2025-08-14T21:23:56.0704567Z    8d6f73986c7..e4dcf56a427  gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base
2025-08-14T21:23:56.0707000Z    f6602a59ed3..2f44712cf54  gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head
2025-08-14T21:23:56.0709270Z  + 5ce995d9848...77fbab074fc gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig  (forced update)
2025-08-14T21:23:56.0711721Z    472b4f2b3f0..3d434480221  gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base
2025-08-14T21:23:56.0714413Z    6d088669e7c..347333b3995  gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head
2025-08-14T21:23:56.0716423Z  + d9e5647addb...9ee0a5f1e67 gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig  (forced update)
2025-08-14T21:23:56.0718604Z    f12b09aa2fc..72cea7eea1f  gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base
2025-08-14T21:23:56.0720899Z    6c4dfda1327..64b4dccb023  gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head
2025-08-14T21:23:56.0723500Z  + 965276ff7a3...03b5add0bd3 gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig  (forced update)
2025-08-14T21:23:56.0725298Z  + 9fbca79be0b...dbe36858fef gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig  (forced update)
2025-08-14T21:23:56.0751170Z    ea5b50f2a44..a16ac305acb  gh/xmfan/274/base           -> origin/gh/xmfan/274/base
2025-08-14T21:23:56.0753906Z    b87cd6dc372..a5ff158b3a4  gh/xmfan/274/head           -> origin/gh/xmfan/274/head
2025-08-14T21:23:56.0756585Z  + 16c7cc80222...49770cd58e4 gh/xmfan/274/orig           -> origin/gh/xmfan/274/orig  (forced update)
2025-08-14T21:23:56.0759514Z  * [new branch]              gh/xmfan/277/base           -> origin/gh/xmfan/277/base
2025-08-14T21:23:56.0761546Z  * [new branch]              gh/xmfan/277/head           -> origin/gh/xmfan/277/head
2025-08-14T21:23:56.0762930Z  * [new branch]              gh/xmfan/277/orig           -> origin/gh/xmfan/277/orig
2025-08-14T21:23:56.0771934Z    47a1db823df..3be70dc30e8  main                        -> origin/main
2025-08-14T21:23:56.0781362Z  t [tag update]              ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562
2025-08-14T21:23:56.0784173Z  t [tag update]              ciflow/inductor/159367      -> ciflow/inductor/159367
2025-08-14T21:23:56.0785798Z  t [tag update]              ciflow/inductor/159368      -> ciflow/inductor/159368
2025-08-14T21:23:56.0787360Z  t [tag update]              ciflow/inductor/159483      -> ciflow/inductor/159483
2025-08-14T21:23:56.0789124Z  t [tag update]              ciflow/inductor/159864      -> ciflow/inductor/159864
2025-08-14T21:23:56.0790677Z  t [tag update]              ciflow/inductor/159865      -> ciflow/inductor/159865
2025-08-14T21:23:56.0792244Z  t [tag update]              ciflow/inductor/159875      -> ciflow/inductor/159875
2025-08-14T21:23:56.0795233Z  * [new tag]                 ciflow/inductor/160679      -> ciflow/inductor/160679
2025-08-14T21:23:56.0797770Z  * [new tag]                 ciflow/trunk/157152         -> ciflow/trunk/157152
2025-08-14T21:23:56.0799533Z  t [tag update]              ciflow/trunk/159562         -> ciflow/trunk/159562
2025-08-14T21:23:56.0801704Z  t [tag update]              ciflow/win-arm64/159562     -> ciflow/win-arm64/159562
2025-08-14T21:23:56.0805928Z  * [new tag]                 trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 -> trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08
2025-08-14T21:23:56.1780847Z [command]/usr/bin/git rev-parse --verify --quiet 1fc683cf17c8c673044538d10266c00f92987be2^{object}
2025-08-14T21:23:56.1819054Z 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:23:56.1825172Z ##[endgroup]
2025-08-14T21:23:56.1825540Z ##[group]Determining the checkout info
2025-08-14T21:23:56.1827084Z ##[endgroup]
2025-08-14T21:23:56.1829359Z [command]/usr/bin/git sparse-checkout disable
2025-08-14T21:23:56.2040239Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-08-14T21:23:56.2080121Z ##[group]Checking out the ref
2025-08-14T21:23:56.2086093Z [command]/usr/bin/git checkout --progress --force 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:23:56.4875972Z Previous HEAD position was 884e299f170 Update on "[associative_scan] support gen_schema for associative_scan"
2025-08-14T21:23:56.4894407Z HEAD is now at 1fc683cf17c [Inductor] Allow indexing a flexible layout for extract_input_node_reduction_ranges (#160645)
2025-08-14T21:23:56.4966474Z ##[endgroup]
2025-08-14T21:23:56.4967086Z ##[group]Setting up auth for fetching submodules
2025-08-14T21:23:56.4974795Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-08-14T21:23:56.5021636Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-08-14T21:23:56.5060773Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-08-14T21:23:56.5098650Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-08-14T21:23:56.5133094Z ##[endgroup]
2025-08-14T21:23:56.5133666Z ##[group]Fetching submodules
2025-08-14T21:23:56.5136275Z [command]/usr/bin/git submodule sync --recursive
2025-08-14T21:23:56.5529126Z Synchronizing submodule url for 'android/libs/fbjni'
2025-08-14T21:23:56.5584174Z Synchronizing submodule url for 'third_party/FP16'
2025-08-14T21:23:56.5641001Z Synchronizing submodule url for 'third_party/FXdiv'
2025-08-14T21:23:56.5700636Z Synchronizing submodule url for 'third_party/NNPACK'
2025-08-14T21:23:56.5758585Z Synchronizing submodule url for 'third_party/NVTX'
2025-08-14T21:23:56.5820338Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-08-14T21:23:56.5881841Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-08-14T21:23:56.5959941Z Synchronizing submodule url for 'third_party/aiter'
2025-08-14T21:23:56.6018285Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:23:56.6086245Z Synchronizing submodule url for 'third_party/benchmark'
2025-08-14T21:23:56.6141836Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-08-14T21:23:56.6209592Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-08-14T21:23:56.6264855Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-08-14T21:23:56.6321234Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-08-14T21:23:56.6382112Z Synchronizing submodule url for 'third_party/cutlass'
2025-08-14T21:23:56.6451795Z Synchronizing submodule url for 'third_party/fbgemm'
2025-08-14T21:23:56.6505480Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-08-14T21:23:56.6556279Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:23:56.6617740Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:23:56.6672074Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-08-14T21:23:56.6738389Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-08-14T21:23:56.6792035Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:23:56.6843657Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-08-14T21:23:56.6905847Z Synchronizing submodule url for 'third_party/flash-attention'
2025-08-14T21:23:56.6957796Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:23:56.7018587Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:23:56.7088132Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-08-14T21:23:56.7148754Z Synchronizing submodule url for 'third_party/fmt'
2025-08-14T21:23:56.7206481Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:23:56.7264333Z Synchronizing submodule url for 'third_party/gloo'
2025-08-14T21:23:56.7322283Z Synchronizing submodule url for 'third_party/googletest'
2025-08-14T21:23:56.7383007Z Synchronizing submodule url for 'third_party/ideep'
2025-08-14T21:23:56.7434479Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-08-14T21:23:56.7498146Z Synchronizing submodule url for 'third_party/ittapi'
2025-08-14T21:23:56.7549707Z Synchronizing submodule url for 'third_party/kineto'
2025-08-14T21:23:56.7608857Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:23:56.7654062Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:23:56.7709173Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:23:56.7760963Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:23:56.7814554Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:23:56.7864015Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:23:56.7919624Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:23:56.7975093Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:23:56.8027954Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:23:56.8084796Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:23:56.8140204Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:23:56.8190600Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:23:56.8252660Z Synchronizing submodule url for 'third_party/kleidiai'
2025-08-14T21:23:56.8311702Z Synchronizing submodule url for 'third_party/mimalloc'
2025-08-14T21:23:56.8370291Z Synchronizing submodule url for 'third_party/nlohmann'
2025-08-14T21:23:56.8427478Z Synchronizing submodule url for 'third_party/onnx'
2025-08-14T21:23:56.8499965Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-08-14T21:23:56.8561135Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-08-14T21:23:56.8615674Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:23:56.8667752Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:23:56.8723617Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:23:56.8774298Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:23:56.8828908Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:23:56.8876670Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:23:56.8933246Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:23:56.8981330Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:23:56.9034873Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:23:56.9093043Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:23:56.9173117Z Synchronizing submodule url for 'third_party/pocketfft'
2025-08-14T21:23:56.9227138Z Synchronizing submodule url for 'third_party/protobuf'
2025-08-14T21:23:56.9285024Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:23:56.9335896Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-08-14T21:23:56.9396844Z Synchronizing submodule url for 'third_party/psimd'
2025-08-14T21:23:56.9457643Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-08-14T21:23:56.9515365Z Synchronizing submodule url for 'third_party/pybind11'
2025-08-14T21:23:56.9577930Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-08-14T21:23:56.9635184Z Synchronizing submodule url for 'third_party/sleef'
2025-08-14T21:23:56.9696752Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-08-14T21:23:56.9750076Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:23:56.9805987Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:23:56.9857329Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:23:56.9909816Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:23:56.9956283Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:23:57.0037697Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-08-14T21:23:57.0728787Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-08-14T21:23:57.1079965Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-08-14T21:23:57.1424352Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-08-14T21:23:57.1768844Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-08-14T21:23:57.2129181Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07'
2025-08-14T21:23:57.2487021Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-08-14T21:23:57.3039009Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-08-14T21:23:57.3511917Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-08-14T21:23:57.4021233Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-08-14T21:23:57.4411060Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-08-14T21:23:57.4964875Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-08-14T21:23:57.5349699Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c'
2025-08-14T21:23:57.5699966Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-08-14T21:23:57.6094933Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6'
2025-08-14T21:23:57.6552768Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491'
2025-08-14T21:23:57.7181290Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067'
2025-08-14T21:23:57.7498039Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea'
2025-08-14T21:23:57.7889705Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9'
2025-08-14T21:23:57.8232053Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-08-14T21:23:57.8648253Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad'
2025-08-14T21:23:57.8987134Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-08-14T21:23:57.9289048Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c'
2025-08-14T21:23:57.9659936Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-08-14T21:23:58.0058422Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-08-14T21:23:58.0568101Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-08-14T21:23:58.1007614Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-08-14T21:23:58.1438161Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-08-14T21:23:58.1803772Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-08-14T21:23:58.2156195Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-08-14T21:23:58.2500175Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2'
2025-08-14T21:23:58.2859472Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-08-14T21:23:58.3217926Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-08-14T21:23:58.3731587Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-08-14T21:23:58.4110004Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-08-14T21:23:58.4491096Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658'
2025-08-14T21:23:58.4836634Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-08-14T21:23:58.5179637Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-08-14T21:23:58.5510538Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-08-14T21:23:58.5847931Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-08-14T21:23:58.6164793Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-08-14T21:23:58.6485303Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-08-14T21:23:58.6820117Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-08-14T21:23:58.7148674Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-08-14T21:23:58.7524078Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-08-14T21:23:58.7838172Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-08-14T21:23:58.8170038Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-08-14T21:23:58.8492351Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-08-14T21:23:58.8858937Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7'
2025-08-14T21:23:58.9217510Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-08-14T21:23:58.9640614Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-08-14T21:23:59.0192186Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-08-14T21:23:59.0577137Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-08-14T21:23:59.1002306Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-08-14T21:23:59.1326756Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-08-14T21:23:59.1650540Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-08-14T21:23:59.1968729Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-08-14T21:23:59.2340807Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-08-14T21:23:59.2670621Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-08-14T21:23:59.2973428Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-08-14T21:23:59.3314191Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-08-14T21:23:59.3671101Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-08-14T21:23:59.4008281Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-08-14T21:23:59.4534426Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-08-14T21:23:59.4922516Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-08-14T21:23:59.5514676Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-08-14T21:23:59.5835753Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-08-14T21:23:59.6173708Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-08-14T21:23:59.6512994Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-08-14T21:23:59.6836066Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-08-14T21:23:59.7217249Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-08-14T21:23:59.7559341Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-08-14T21:23:59.7903306Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-08-14T21:23:59.8265083Z Submodule path 'third_party/tensorpipe': checked out 'dacda0567d9f23d4bc503e1c4f84aa65f33ac38a'
2025-08-14T21:23:59.8576360Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-08-14T21:23:59.8880290Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-08-14T21:23:59.9361330Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b'
2025-08-14T21:23:59.9693591Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-08-14T21:23:59.9994506Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-08-14T21:24:00.0131375Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-08-14T21:24:00.0501289Z Entering 'android/libs/fbjni'
2025-08-14T21:24:00.0561391Z Entering 'third_party/FP16'
2025-08-14T21:24:00.0625978Z Entering 'third_party/FXdiv'
2025-08-14T21:24:00.0693061Z Entering 'third_party/NNPACK'
2025-08-14T21:24:00.0755162Z Entering 'third_party/NVTX'
2025-08-14T21:24:00.0817293Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:24:00.0881804Z Entering 'third_party/XNNPACK'
2025-08-14T21:24:00.0962088Z Entering 'third_party/aiter'
2025-08-14T21:24:00.1027838Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:24:00.1107643Z Entering 'third_party/benchmark'
2025-08-14T21:24:00.1171358Z Entering 'third_party/composable_kernel'
2025-08-14T21:24:00.1240192Z Entering 'third_party/cpp-httplib'
2025-08-14T21:24:00.1303731Z Entering 'third_party/cpuinfo'
2025-08-14T21:24:00.1363828Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:24:00.1429746Z Entering 'third_party/cutlass'
2025-08-14T21:24:00.1505945Z Entering 'third_party/fbgemm'
2025-08-14T21:24:00.1575624Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:24:00.1633397Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:24:00.1702852Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:24:00.1756331Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:24:00.1827082Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:24:00.1893005Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:24:00.1948370Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:24:00.2018619Z Entering 'third_party/flash-attention'
2025-08-14T21:24:00.2082484Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:24:00.2151094Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:24:00.2229887Z Entering 'third_party/flatbuffers'
2025-08-14T21:24:00.2300413Z Entering 'third_party/fmt'
2025-08-14T21:24:00.2365008Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:24:00.2429185Z Entering 'third_party/gloo'
2025-08-14T21:24:00.2496170Z Entering 'third_party/googletest'
2025-08-14T21:24:00.2553759Z Entering 'third_party/ideep'
2025-08-14T21:24:00.2614743Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:24:00.2683575Z Entering 'third_party/ittapi'
2025-08-14T21:24:00.2746405Z Entering 'third_party/kineto'
2025-08-14T21:24:00.2813898Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:24:00.2869469Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:24:00.2936957Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:24:00.2990181Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:24:00.3048883Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:24:00.3104051Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:24:00.3160935Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:24:00.3219287Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:24:00.3278348Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:24:00.3337735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:24:00.3397701Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:24:00.3455668Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:24:00.3515478Z Entering 'third_party/kleidiai'
2025-08-14T21:24:00.3578920Z Entering 'third_party/mimalloc'
2025-08-14T21:24:00.3639698Z Entering 'third_party/nlohmann'
2025-08-14T21:24:00.3702949Z Entering 'third_party/onnx'
2025-08-14T21:24:00.3783619Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:24:00.3854126Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:24:00.3914801Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:24:00.3977106Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:24:00.4032672Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:24:00.4097080Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:24:00.4154323Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:24:00.4216427Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:24:00.4270187Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:24:00.4325496Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:24:00.4385145Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:24:00.4446406Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:24:00.4529797Z Entering 'third_party/pocketfft'
2025-08-14T21:24:00.4586516Z Entering 'third_party/protobuf'
2025-08-14T21:24:00.4652593Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:24:00.4710006Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:24:00.4768373Z Entering 'third_party/psimd'
2025-08-14T21:24:00.4829092Z Entering 'third_party/pthreadpool'
2025-08-14T21:24:00.4890585Z Entering 'third_party/pybind11'
2025-08-14T21:24:00.4946904Z Entering 'third_party/python-peachpy'
2025-08-14T21:24:00.5010215Z Entering 'third_party/sleef'
2025-08-14T21:24:00.5067083Z Entering 'third_party/tensorpipe'
2025-08-14T21:24:00.5128573Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:24:00.5185214Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:24:00.5249670Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:24:00.5305735Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:24:00.5361509Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:24:00.5453659Z ##[endgroup]
2025-08-14T21:24:00.5454109Z ##[group]Persisting credentials for submodules
2025-08-14T21:24:00.5459501Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-08-14T21:24:00.5831220Z Entering 'android/libs/fbjni'
2025-08-14T21:24:00.5866991Z url.https://github.com/.insteadof
2025-08-14T21:24:00.5867477Z url.https://github.com/.insteadof
2025-08-14T21:24:00.5921161Z Entering 'third_party/FP16'
2025-08-14T21:24:00.5959281Z url.https://github.com/.insteadof
2025-08-14T21:24:00.5959750Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6013948Z Entering 'third_party/FXdiv'
2025-08-14T21:24:00.6057512Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6057955Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6110056Z Entering 'third_party/NNPACK'
2025-08-14T21:24:00.6146674Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6147103Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6201249Z Entering 'third_party/NVTX'
2025-08-14T21:24:00.6239054Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6239529Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6295181Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:24:00.6338253Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6338565Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6391574Z Entering 'third_party/XNNPACK'
2025-08-14T21:24:00.6427897Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6428266Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6497170Z Entering 'third_party/aiter'
2025-08-14T21:24:00.6538818Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6539313Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6590443Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:24:00.6626735Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6627188Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6689814Z Entering 'third_party/benchmark'
2025-08-14T21:24:00.6729194Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6729685Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6783603Z Entering 'third_party/composable_kernel'
2025-08-14T21:24:00.6822364Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6822796Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6882781Z Entering 'third_party/cpp-httplib'
2025-08-14T21:24:00.6919103Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6919567Z url.https://github.com/.insteadof
2025-08-14T21:24:00.6975036Z Entering 'third_party/cpuinfo'
2025-08-14T21:24:00.7018953Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7073966Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7074419Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:24:00.7128061Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7128512Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7177080Z Entering 'third_party/cutlass'
2025-08-14T21:24:00.7218241Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7218696Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7282462Z Entering 'third_party/fbgemm'
2025-08-14T21:24:00.7321318Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7321785Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7378473Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:24:00.7419181Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7419629Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7471022Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:24:00.7505402Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7505859Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7563082Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:24:00.7597097Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7597531Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7654781Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:24:00.7691077Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7750021Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7750497Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:24:00.7788042Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7788478Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7834440Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:24:00.7867578Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7868012Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7916444Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:24:00.7949916Z url.https://github.com/.insteadof
2025-08-14T21:24:00.7950359Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8004269Z Entering 'third_party/flash-attention'
2025-08-14T21:24:00.8042308Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8042757Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8097493Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:24:00.8133137Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8133455Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8189297Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:24:00.8228004Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8228446Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8292756Z Entering 'third_party/flatbuffers'
2025-08-14T21:24:00.8332234Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8332759Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8385161Z Entering 'third_party/fmt'
2025-08-14T21:24:00.8425471Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8425907Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8475894Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:24:00.8510407Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8510830Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8565023Z Entering 'third_party/gloo'
2025-08-14T21:24:00.8603340Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8603780Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8659141Z Entering 'third_party/googletest'
2025-08-14T21:24:00.8694130Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8694683Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8747605Z Entering 'third_party/ideep'
2025-08-14T21:24:00.8789045Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8789474Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8837535Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:24:00.8871603Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8872038Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8932934Z Entering 'third_party/ittapi'
2025-08-14T21:24:00.8973150Z url.https://github.com/.insteadof
2025-08-14T21:24:00.8973614Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9022746Z Entering 'third_party/kineto'
2025-08-14T21:24:00.9063070Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9063518Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9114493Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:24:00.9151143Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9151606Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9195810Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:24:00.9232261Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9232770Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9289771Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:24:00.9326614Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9327056Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9380085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:24:00.9414403Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9416603Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9463332Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:24:00.9502437Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9502885Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9553123Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:24:00.9591251Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9591688Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9646103Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:24:00.9684355Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9684800Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9734059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:24:00.9766942Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9767521Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9819671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:24:00.9853687Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9854181Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9907603Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:24:00.9943647Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9944091Z url.https://github.com/.insteadof
2025-08-14T21:24:00.9996390Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:24:01.0033747Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0034163Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0089011Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:24:01.0118937Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0119353Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0174727Z Entering 'third_party/kleidiai'
2025-08-14T21:24:01.0216161Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0216567Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0269014Z Entering 'third_party/mimalloc'
2025-08-14T21:24:01.0301027Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0301482Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0349501Z Entering 'third_party/nlohmann'
2025-08-14T21:24:01.0384895Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0385225Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0435858Z Entering 'third_party/onnx'
2025-08-14T21:24:01.0475381Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0475820Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0547920Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:24:01.0584585Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0585023Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0647160Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:24:01.0686826Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0687344Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0740682Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:24:01.0776669Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0777126Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0828169Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:24:01.0864589Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0892111Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0913891Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:24:01.0943092Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0992569Z url.https://github.com/.insteadof
2025-08-14T21:24:01.0993176Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:24:01.1025850Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1026304Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1075389Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:24:01.1106046Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1106960Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1154052Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:24:01.1186357Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1186829Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1234208Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:24:01.1266728Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1267153Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1317579Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:24:01.1355881Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1356324Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1414338Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:24:01.1448014Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1448342Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1500155Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:24:01.1539338Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1539861Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1613362Z Entering 'third_party/pocketfft'
2025-08-14T21:24:01.1650329Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1650842Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1702580Z Entering 'third_party/protobuf'
2025-08-14T21:24:01.1740364Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1740875Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1793364Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:24:01.1827486Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1827928Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1879713Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:24:01.1914349Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1914818Z url.https://github.com/.insteadof
2025-08-14T21:24:01.1968768Z Entering 'third_party/psimd'
2025-08-14T21:24:01.2008654Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2009109Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2063095Z Entering 'third_party/pthreadpool'
2025-08-14T21:24:01.2099503Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2099953Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2152016Z Entering 'third_party/pybind11'
2025-08-14T21:24:01.2188420Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2188871Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2243755Z Entering 'third_party/python-peachpy'
2025-08-14T21:24:01.2280310Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2280642Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2336496Z Entering 'third_party/sleef'
2025-08-14T21:24:01.2379795Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2380324Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2432758Z Entering 'third_party/tensorpipe'
2025-08-14T21:24:01.2468985Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2469457Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2520662Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:24:01.2558426Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2559183Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2614775Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:24:01.2651416Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2651976Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2703295Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:24:01.2740413Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2740878Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2791382Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:24:01.2827945Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2828387Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2871739Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:24:01.2905978Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2906446Z url.https://github.com/.insteadof
2025-08-14T21:24:01.2993748Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-08-14T21:24:01.3374721Z Entering 'android/libs/fbjni'
2025-08-14T21:24:01.3428875Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-08-14T21:24:01.3461172Z Entering 'third_party/FP16'
2025-08-14T21:24:01.3523524Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-08-14T21:24:01.3555041Z Entering 'third_party/FXdiv'
2025-08-14T21:24:01.3612533Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-08-14T21:24:01.3645121Z Entering 'third_party/NNPACK'
2025-08-14T21:24:01.3704432Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-08-14T21:24:01.3735513Z Entering 'third_party/NVTX'
2025-08-14T21:24:01.3789015Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-08-14T21:24:01.3822571Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:24:01.3883327Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-08-14T21:24:01.3915797Z Entering 'third_party/XNNPACK'
2025-08-14T21:24:01.3973974Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-08-14T21:24:01.4022222Z Entering 'third_party/aiter'
2025-08-14T21:24:01.4085434Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-08-14T21:24:01.4115613Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:24:01.4173255Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-08-14T21:24:01.4213858Z Entering 'third_party/benchmark'
2025-08-14T21:24:01.4270409Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:24:01.4302003Z Entering 'third_party/composable_kernel'
2025-08-14T21:24:01.4364031Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-08-14T21:24:01.4405478Z Entering 'third_party/cpp-httplib'
2025-08-14T21:24:01.4464833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-08-14T21:24:01.4496659Z Entering 'third_party/cpuinfo'
2025-08-14T21:24:01.4550042Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-08-14T21:24:01.4583088Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:24:01.4645392Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-08-14T21:24:01.4677054Z Entering 'third_party/cutlass'
2025-08-14T21:24:01.4735083Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-08-14T21:24:01.4777252Z Entering 'third_party/fbgemm'
2025-08-14T21:24:01.4832828Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-08-14T21:24:01.4869929Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:24:01.4927405Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-08-14T21:24:01.4955844Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:24:01.5013264Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-08-14T21:24:01.5047298Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:24:01.5102124Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-08-14T21:24:01.5133502Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:24:01.5185672Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-08-14T21:24:01.5230282Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:24:01.5284809Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-08-14T21:24:01.5310966Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:24:01.5366516Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-08-14T21:24:01.5396297Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:24:01.5456626Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-08-14T21:24:01.5494827Z Entering 'third_party/flash-attention'
2025-08-14T21:24:01.5552324Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-08-14T21:24:01.5584626Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:24:01.5645470Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-08-14T21:24:01.5678632Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:24:01.5736975Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-08-14T21:24:01.5785160Z Entering 'third_party/flatbuffers'
2025-08-14T21:24:01.5846056Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-08-14T21:24:01.5876501Z Entering 'third_party/fmt'
2025-08-14T21:24:01.5937068Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-08-14T21:24:01.5972178Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:24:01.6036975Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-08-14T21:24:01.6068409Z Entering 'third_party/gloo'
2025-08-14T21:24:01.6126800Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-08-14T21:24:01.6155052Z Entering 'third_party/googletest'
2025-08-14T21:24:01.6215775Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:01.6251618Z Entering 'third_party/ideep'
2025-08-14T21:24:01.6305241Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-08-14T21:24:01.6335019Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:24:01.6397901Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-08-14T21:24:01.6440452Z Entering 'third_party/ittapi'
2025-08-14T21:24:01.6499725Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-08-14T21:24:01.6533864Z Entering 'third_party/kineto'
2025-08-14T21:24:01.6598400Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-08-14T21:24:01.6628834Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:24:01.6686593Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-08-14T21:24:01.6711511Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:24:01.6769158Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-08-14T21:24:01.6796471Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:24:01.6857759Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-08-14T21:24:01.6887666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:24:01.6943683Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-08-14T21:24:01.6971600Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:24:01.7025368Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-08-14T21:24:01.7052992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:24:01.7108291Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-08-14T21:24:01.7142019Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:24:01.7200931Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-08-14T21:24:01.7229275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:24:01.7287426Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:01.7313894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:24:01.7370808Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-08-14T21:24:01.7402916Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:24:01.7461214Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-08-14T21:24:01.7497570Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:24:01.7549895Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-08-14T21:24:01.7583961Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:24:01.7641874Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:01.7675669Z Entering 'third_party/kleidiai'
2025-08-14T21:24:01.7730614Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-08-14T21:24:01.7763721Z Entering 'third_party/mimalloc'
2025-08-14T21:24:01.7822504Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-08-14T21:24:01.7853985Z Entering 'third_party/nlohmann'
2025-08-14T21:24:01.7907750Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-08-14T21:24:01.7941329Z Entering 'third_party/onnx'
2025-08-14T21:24:01.8000803Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-08-14T21:24:01.8047649Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:24:01.8103581Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:24:01.8139082Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:24:01.8202755Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-08-14T21:24:01.8233692Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:24:01.8290936Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:24:01.8316057Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:24:01.8371625Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:01.8397804Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:24:01.8452021Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-08-14T21:24:01.8477820Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:24:01.8532395Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-08-14T21:24:01.8559926Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:24:01.8612826Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-08-14T21:24:01.8642783Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:24:01.8701871Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-08-14T21:24:01.8735121Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:24:01.8790236Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-08-14T21:24:01.8815206Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:24:01.8870808Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-08-14T21:24:01.8901590Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:24:01.8964082Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-08-14T21:24:01.8998266Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:24:01.9061051Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-08-14T21:24:01.9120569Z Entering 'third_party/pocketfft'
2025-08-14T21:24:01.9182725Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-08-14T21:24:01.9217541Z Entering 'third_party/protobuf'
2025-08-14T21:24:01.9267500Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-08-14T21:24:01.9299072Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:24:01.9352621Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:24:01.9386825Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:24:01.9445325Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:01.9479223Z Entering 'third_party/psimd'
2025-08-14T21:24:01.9542661Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-08-14T21:24:01.9577933Z Entering 'third_party/pthreadpool'
2025-08-14T21:24:01.9628284Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-08-14T21:24:01.9660579Z Entering 'third_party/pybind11'
2025-08-14T21:24:01.9725126Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:24:01.9756019Z Entering 'third_party/python-peachpy'
2025-08-14T21:24:01.9812663Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-08-14T21:24:01.9840370Z Entering 'third_party/sleef'
2025-08-14T21:24:01.9902675Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-08-14T21:24:01.9938348Z Entering 'third_party/tensorpipe'
2025-08-14T21:24:01.9992072Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-08-14T21:24:02.0025872Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:24:02.0086479Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:24:02.0115584Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:24:02.0174493Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-08-14T21:24:02.0200669Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:24:02.0255960Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-08-14T21:24:02.0282997Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:24:02.0342081Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:24:02.0368556Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:24:02.0423549Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-08-14T21:24:02.0733118Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-08-14T21:24:02.1111994Z Entering 'android/libs/fbjni'
2025-08-14T21:24:02.1172102Z Entering 'third_party/FP16'
2025-08-14T21:24:02.1234104Z Entering 'third_party/FXdiv'
2025-08-14T21:24:02.1295918Z Entering 'third_party/NNPACK'
2025-08-14T21:24:02.1353529Z Entering 'third_party/NVTX'
2025-08-14T21:24:02.1415905Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:24:02.1473692Z Entering 'third_party/XNNPACK'
2025-08-14T21:24:02.1551763Z Entering 'third_party/aiter'
2025-08-14T21:24:02.1614532Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:24:02.1684156Z Entering 'third_party/benchmark'
2025-08-14T21:24:02.1746594Z Entering 'third_party/composable_kernel'
2025-08-14T21:24:02.1816367Z Entering 'third_party/cpp-httplib'
2025-08-14T21:24:02.1880680Z Entering 'third_party/cpuinfo'
2025-08-14T21:24:02.1948900Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:24:02.2013723Z Entering 'third_party/cutlass'
2025-08-14T21:24:02.2085302Z Entering 'third_party/fbgemm'
2025-08-14T21:24:02.2149692Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:24:02.2212162Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:24:02.2274518Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:24:02.2333952Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:24:02.2398062Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:24:02.2456726Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:24:02.2513497Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:24:02.2578167Z Entering 'third_party/flash-attention'
2025-08-14T21:24:02.2635661Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:24:02.2701147Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:24:02.2774244Z Entering 'third_party/flatbuffers'
2025-08-14T21:24:02.2834819Z Entering 'third_party/fmt'
2025-08-14T21:24:02.2897928Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:24:02.2962044Z Entering 'third_party/gloo'
2025-08-14T21:24:02.3024988Z Entering 'third_party/googletest'
2025-08-14T21:24:02.3088963Z Entering 'third_party/ideep'
2025-08-14T21:24:02.3149464Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:24:02.3224840Z Entering 'third_party/ittapi'
2025-08-14T21:24:02.3294346Z Entering 'third_party/kineto'
2025-08-14T21:24:02.3350291Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:24:02.3412730Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:24:02.3470870Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:24:02.3534438Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:24:02.3587975Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:24:02.3651657Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:24:02.3710450Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:24:02.3774648Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:24:02.3828621Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:24:02.3887388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:24:02.3950917Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:24:02.4015139Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:24:02.4072552Z Entering 'third_party/kleidiai'
2025-08-14T21:24:02.4137813Z Entering 'third_party/mimalloc'
2025-08-14T21:24:02.4200276Z Entering 'third_party/nlohmann'
2025-08-14T21:24:02.4261445Z Entering 'third_party/onnx'
2025-08-14T21:24:02.4334892Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:24:02.4400533Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:24:02.4466403Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:24:02.4526117Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:24:02.4586543Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:24:02.4644656Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:24:02.4702988Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:24:02.4763924Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:24:02.4820490Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:24:02.4876278Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:24:02.4939266Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:24:02.4999044Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:24:02.5083669Z Entering 'third_party/pocketfft'
2025-08-14T21:24:02.5143786Z Entering 'third_party/protobuf'
2025-08-14T21:24:02.5210728Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:24:02.5264514Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:24:02.5324870Z Entering 'third_party/psimd'
2025-08-14T21:24:02.5388646Z Entering 'third_party/pthreadpool'
2025-08-14T21:24:02.5453280Z Entering 'third_party/pybind11'
2025-08-14T21:24:02.5515060Z Entering 'third_party/python-peachpy'
2025-08-14T21:24:02.5579969Z Entering 'third_party/sleef'
2025-08-14T21:24:02.5639539Z Entering 'third_party/tensorpipe'
2025-08-14T21:24:02.5701470Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:24:02.5751619Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:24:02.5810579Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:24:02.5865682Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:24:02.5921107Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:24:02.6013679Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-08-14T21:24:02.6391074Z Entering 'android/libs/fbjni'
2025-08-14T21:24:02.6452868Z Entering 'third_party/FP16'
2025-08-14T21:24:02.6518892Z Entering 'third_party/FXdiv'
2025-08-14T21:24:02.6583326Z Entering 'third_party/NNPACK'
2025-08-14T21:24:02.6651795Z Entering 'third_party/NVTX'
2025-08-14T21:24:02.6710257Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:24:02.6772329Z Entering 'third_party/XNNPACK'
2025-08-14T21:24:02.6850005Z Entering 'third_party/aiter'
2025-08-14T21:24:02.6908394Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:24:02.6985926Z Entering 'third_party/benchmark'
2025-08-14T21:24:02.7053007Z Entering 'third_party/composable_kernel'
2025-08-14T21:24:02.7119662Z Entering 'third_party/cpp-httplib'
2025-08-14T21:24:02.7182203Z Entering 'third_party/cpuinfo'
2025-08-14T21:24:02.7247322Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:24:02.7308631Z Entering 'third_party/cutlass'
2025-08-14T21:24:02.7384105Z Entering 'third_party/fbgemm'
2025-08-14T21:24:02.7454419Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:24:02.7509083Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:24:02.7575089Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:24:02.7632780Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:24:02.7703055Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:24:02.7755380Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:24:02.7813051Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:24:02.7875710Z Entering 'third_party/flash-attention'
2025-08-14T21:24:02.7938135Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:24:02.8002581Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:24:02.8072869Z Entering 'third_party/flatbuffers'
2025-08-14T21:24:02.8137509Z Entering 'third_party/fmt'
2025-08-14T21:24:02.8202991Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:24:02.8268271Z Entering 'third_party/gloo'
2025-08-14T21:24:02.8335531Z Entering 'third_party/googletest'
2025-08-14T21:24:02.8393854Z Entering 'third_party/ideep'
2025-08-14T21:24:02.8452862Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:24:02.8521601Z Entering 'third_party/ittapi'
2025-08-14T21:24:02.8584755Z Entering 'third_party/kineto'
2025-08-14T21:24:02.8647753Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:24:02.8702882Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:24:02.8764476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:24:02.8825100Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:24:02.8878077Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:24:02.8935163Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:24:02.8995191Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:24:02.9055265Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:24:02.9111779Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:24:02.9176212Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:24:02.9235888Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:24:02.9295940Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:24:02.9356085Z Entering 'third_party/kleidiai'
2025-08-14T21:24:02.9417378Z Entering 'third_party/mimalloc'
2025-08-14T21:24:02.9478906Z Entering 'third_party/nlohmann'
2025-08-14T21:24:02.9541698Z Entering 'third_party/onnx'
2025-08-14T21:24:02.9621838Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:24:02.9685779Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:24:02.9750571Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:24:02.9815888Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:24:02.9871816Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:24:02.9935881Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:24:02.9993772Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:24:03.0056701Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:24:03.0111786Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:24:03.0174304Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:24:03.0233747Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:24:03.0300419Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:24:03.0377514Z Entering 'third_party/pocketfft'
2025-08-14T21:24:03.0434536Z Entering 'third_party/protobuf'
2025-08-14T21:24:03.0500837Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:24:03.0558293Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:24:03.0621757Z Entering 'third_party/psimd'
2025-08-14T21:24:03.0687598Z Entering 'third_party/pthreadpool'
2025-08-14T21:24:03.0751916Z Entering 'third_party/pybind11'
2025-08-14T21:24:03.0820007Z Entering 'third_party/python-peachpy'
2025-08-14T21:24:03.0881189Z Entering 'third_party/sleef'
2025-08-14T21:24:03.0941477Z Entering 'third_party/tensorpipe'
2025-08-14T21:24:03.0997731Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:24:03.1052506Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:24:03.1109794Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:24:03.1169852Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:24:03.1220505Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:24:03.1301377Z ##[endgroup]
2025-08-14T21:24:03.1350871Z [command]/usr/bin/git log -1 --format=%H
2025-08-14T21:24:03.1385533Z 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:24:03.1589541Z Prepare all required actions
2025-08-14T21:24:03.1590000Z Getting action download info
2025-08-14T21:24:03.3212790Z ##[group]Run ./.github/actions/setup-rocm
2025-08-14T21:24:03.3213064Z env:
2025-08-14T21:24:03.3213239Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.3213462Z ##[endgroup]
2025-08-14T21:24:03.3234650Z ##[group]Run dpkg -l | grep -E "  rocm"
2025-08-14T21:24:03.3235197Z dpkg -l | grep -E "  rocm"
2025-08-14T21:24:03.3255952Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.3256267Z env:
2025-08-14T21:24:03.3256449Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.3256658Z ##[endgroup]
2025-08-14T21:24:03.3483247Z ii  rocm                                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) software stack meta package
2025-08-14T21:24:03.3483798Z ii  rocm-cmake                             0.14.0.60303-74~22.04                   amd64        rocm-cmake built using CMake
2025-08-14T21:24:03.3484381Z ii  rocm-core                              6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3484920Z ii  rocm-dbgapi                            0.77.0.60303-74~22.04                   amd64        Library to provide AMD GPU debugger API
2025-08-14T21:24:03.3485518Z ii  rocm-debug-agent                       2.0.3.60303-74~22.04                    amd64        Radeon Open Compute Debug Agent (ROCdebug-agent)
2025-08-14T21:24:03.3486132Z ii  rocm-developer-tools                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3486712Z ii  rocm-device-libs                       1.0.0.60303-74~22.04                    amd64        Radeon Open Compute - device libraries
2025-08-14T21:24:03.3487193Z ii  rocm-gdb                               15.2.60303-74~22.04                     amd64        ROCgdb
2025-08-14T21:24:03.3487717Z ii  rocm-hip-libraries                     6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3488308Z ii  rocm-hip-runtime                       6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3488905Z ii  rocm-hip-runtime-dev                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3489733Z ii  rocm-hip-sdk                           6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3490386Z ii  rocm-language-runtime                  6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3490976Z ii  rocm-llvm                              18.0.0.25012.60303-74~22.04             amd64        ROCm core compiler
2025-08-14T21:24:03.3491537Z ii  rocm-ml-libraries                      6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3492135Z ii  rocm-ml-sdk                            6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3492657Z ii  rocm-opencl                            2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-08-14T21:24:03.3493163Z ii  rocm-opencl-dev                        2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-08-14T21:24:03.3493743Z ii  rocm-opencl-runtime                    6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3494367Z ii  rocm-opencl-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3495320Z ii  rocm-openmp-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) OpenMP Software development Kit.
2025-08-14T21:24:03.3495972Z ii  rocm-smi-lib                           7.4.0.60303-74~22.04                    amd64        AMD System Management libraries
2025-08-14T21:24:03.3496546Z ii  rocm-utils                             6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:24:03.3497289Z ii  rocminfo                               1.0.0.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime rocminfo tool
2025-08-14T21:24:03.3561305Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T21:24:03.3561795Z # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T21:24:03.3562263Z # shellcheck disable=SC2046
2025-08-14T21:24:03.3562547Z docker stop $(docker ps -q) || true
2025-08-14T21:24:03.3562842Z # Prune all stopped containers.
2025-08-14T21:24:03.3563110Z docker container prune -f
2025-08-14T21:24:03.3581675Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.3581992Z env:
2025-08-14T21:24:03.3582171Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.3582387Z ##[endgroup]
2025-08-14T21:24:03.3989601Z "docker stop" requires at least 1 argument.
2025-08-14T21:24:03.3990169Z See 'docker stop --help'.
2025-08-14T21:24:03.3990441Z 
2025-08-14T21:24:03.3990684Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-08-14T21:24:03.3991079Z 
2025-08-14T21:24:03.3991248Z Stop one or more running containers
2025-08-14T21:24:03.4185272Z Total reclaimed space: 0B
2025-08-14T21:24:03.4238687Z ##[group]Run cat /etc/os-release || true
2025-08-14T21:24:03.4239045Z cat /etc/os-release || true
2025-08-14T21:24:03.4239350Z cat /etc/apt/sources.list.d/rocm.list || true
2025-08-14T21:24:03.4239677Z cat /opt/rocm/.info/version || true
2025-08-14T21:24:03.4239933Z whoami
2025-08-14T21:24:03.4262576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.4262915Z env:
2025-08-14T21:24:03.4263101Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.4263334Z ##[endgroup]
2025-08-14T21:24:03.4319910Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2025-08-14T21:24:03.4320345Z NAME="Ubuntu"
2025-08-14T21:24:03.4320701Z VERSION_ID="22.04"
2025-08-14T21:24:03.4321102Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2025-08-14T21:24:03.4321559Z VERSION_CODENAME=jammy
2025-08-14T21:24:03.4321894Z ID=ubuntu
2025-08-14T21:24:03.4322297Z ID_LIKE=debian
2025-08-14T21:24:03.4322663Z HOME_URL="https://www.ubuntu.com/"
2025-08-14T21:24:03.4323145Z SUPPORT_URL="https://help.ubuntu.com/"
2025-08-14T21:24:03.4323696Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-08-14T21:24:03.4324511Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-08-14T21:24:03.4325237Z UBUNTU_CODENAME=jammy
2025-08-14T21:24:03.4337829Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main
2025-08-14T21:24:03.4355306Z 6.3.3-74
2025-08-14T21:24:03.4380008Z pytorchci
2025-08-14T21:24:03.4406152Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-08-14T21:24:03.4406459Z dpkg -l | grep -E "  amdgpu"
2025-08-14T21:24:03.4425361Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.4425703Z env:
2025-08-14T21:24:03.4425900Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.4426124Z ##[endgroup]
2025-08-14T21:24:03.4643705Z ii  amdgpu-core                            1:6.3.60303-2119913.22.04               all          Core meta package for unified amdgpu driver.
2025-08-14T21:24:03.4644707Z ii  amdgpu-dkms                            1:6.10.5.60303-2119913.22.04            all          amdgpu driver in DKMS format.
2025-08-14T21:24:03.4645765Z ii  amdgpu-dkms-firmware                   1:6.10.5.60303-2119913.22.04            all          firmware blobs used by amdgpu driver in DKMS format
2025-08-14T21:24:03.4647293Z ii  amdgpu-install                         6.3.60303-2119913.22.04                 all          AMDGPU driver repository and installer
2025-08-14T21:24:03.4671928Z ##[group]Run rocm-smi
2025-08-14T21:24:03.4672184Z rocm-smi
2025-08-14T21:24:03.4693037Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.4693641Z env:
2025-08-14T21:24:03.4693838Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.4694063Z ##[endgroup]
2025-08-14T21:24:03.6204072Z 
2025-08-14T21:24:03.6204086Z 
2025-08-14T21:24:03.6204418Z ========================================= ROCm System Management Interface =========================================
2025-08-14T21:24:03.6204995Z =================================================== Concise Info ===================================================
2025-08-14T21:24:03.6205624Z Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2025-08-14T21:24:03.6206657Z               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
2025-08-14T21:24:03.6207157Z ====================================================================================================================
2025-08-14T21:24:03.6207890Z 0       4     0x740c,   57586  38.0°C  97.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:24:03.6208631Z 1       5     0x740c,   45873  36.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:24:03.6209293Z 2       2     0x740c,   51627  34.0°C  97.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:24:03.6209948Z 3       3     0x740c,   64489  30.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:24:03.6210605Z 4       8     0x740c,   30939  37.0°C  94.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:24:03.6211282Z 5       9     0x740c,   8466   35.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:24:03.6211935Z 6       6     0x740c,   41154  33.0°C  95.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:24:03.6212605Z 7       7     0x740c,   63755  33.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:24:03.6220486Z ====================================================================================================================
2025-08-14T21:24:03.6220847Z =============================================== End of ROCm SMI Log ================================================
2025-08-14T21:24:03.6366969Z ##[group]Run rocminfo
2025-08-14T21:24:03.6367258Z rocminfo
2025-08-14T21:24:03.6389232Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.6389557Z env:
2025-08-14T21:24:03.6389781Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.6390008Z ##[endgroup]
2025-08-14T21:24:03.7716801Z ROCk module version 6.10.5 is loaded
2025-08-14T21:24:03.7717234Z =====================    
2025-08-14T21:24:03.7717563Z HSA System Attributes    
2025-08-14T21:24:03.7717983Z =====================    
2025-08-14T21:24:03.7718292Z Runtime Version:         1.14
2025-08-14T21:24:03.7718715Z Runtime Ext Version:     1.6
2025-08-14T21:24:03.7719345Z System Timestamp Freq.:  1000.000000MHz
2025-08-14T21:24:03.7720244Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-08-14T21:24:03.7721020Z Machine Model:           LARGE                              
2025-08-14T21:24:03.7721673Z System Endianness:       LITTLE                             
2025-08-14T21:24:03.7722541Z Mwaitx:                  DISABLED
2025-08-14T21:24:03.7722989Z DMAbuf Support:          YES
2025-08-14T21:24:03.7723229Z 
2025-08-14T21:24:03.7723368Z ==========               
2025-08-14T21:24:03.7723839Z HSA Agents               
2025-08-14T21:24:03.7724191Z ==========               
2025-08-14T21:24:03.7724939Z *******                  
2025-08-14T21:24:03.7725328Z Agent 1                  
2025-08-14T21:24:03.7725678Z *******                  
2025-08-14T21:24:03.7726113Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:24:03.7726697Z   Uuid:                    CPU-XX                             
2025-08-14T21:24:03.7727291Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:24:03.7728212Z   Vendor Name:             CPU                                
2025-08-14T21:24:03.7728881Z   Feature:                 None specified                     
2025-08-14T21:24:03.7729321Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:24:03.7729743Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7730161Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:24:03.7730558Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:24:03.7730963Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:24:03.7731347Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7731725Z   Node:                    0                                  
2025-08-14T21:24:03.7732085Z   Device Type:             CPU                                
2025-08-14T21:24:03.7732442Z   Cache Info:              
2025-08-14T21:24:03.7732751Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:24:03.7733105Z   Chip ID:                 0(0x0)                             
2025-08-14T21:24:03.7733501Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:24:03.7733907Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:24:03.7734329Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:24:03.7734698Z   BDFID:                   0                                  
2025-08-14T21:24:03.7735102Z   Internal Node ID:        0                                  
2025-08-14T21:24:03.7735531Z   Compute Unit:            64                                 
2025-08-14T21:24:03.7735922Z   SIMDs per CU:            0                                  
2025-08-14T21:24:03.7736341Z   Shader Engines:          0                                  
2025-08-14T21:24:03.7736748Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:24:03.7737177Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:24:03.7737541Z   Memory Properties:       
2025-08-14T21:24:03.7737826Z   Features:                None
2025-08-14T21:24:03.7738097Z   Pool Info:               
2025-08-14T21:24:03.7738364Z     Pool 1                   
2025-08-14T21:24:03.7738694Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7739078Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:24:03.7739476Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7739860Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7740221Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7740572Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7740927Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7741233Z     Pool 2                   
2025-08-14T21:24:03.7741503Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7741824Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:24:03.7742130Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7742469Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7742812Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7743167Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7743647Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7743959Z     Pool 3                   
2025-08-14T21:24:03.7744237Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:24:03.7744553Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:24:03.7744869Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7745353Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7745699Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7746057Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7746394Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7746696Z     Pool 4                   
2025-08-14T21:24:03.7746976Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7747286Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:24:03.7747609Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7747938Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7748291Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7748638Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7748996Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7749296Z   ISA Info:                
2025-08-14T21:24:03.7749501Z *******                  
2025-08-14T21:24:03.7749712Z Agent 2                  
2025-08-14T21:24:03.7749905Z *******                  
2025-08-14T21:24:03.7750154Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:24:03.7750467Z   Uuid:                    CPU-XX                             
2025-08-14T21:24:03.7750799Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:24:03.7751136Z   Vendor Name:             CPU                                
2025-08-14T21:24:03.7751461Z   Feature:                 None specified                     
2025-08-14T21:24:03.7751787Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:24:03.7752111Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7752455Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:24:03.7752777Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:24:03.7753099Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:24:03.7753411Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7753713Z   Node:                    1                                  
2025-08-14T21:24:03.7754018Z   Device Type:             CPU                                
2025-08-14T21:24:03.7754298Z   Cache Info:              
2025-08-14T21:24:03.7754539Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:24:03.7754830Z   Chip ID:                 0(0x0)                             
2025-08-14T21:24:03.7755140Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:24:03.7755468Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:24:03.7755806Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:24:03.7756124Z   BDFID:                   0                                  
2025-08-14T21:24:03.7756428Z   Internal Node ID:        1                                  
2025-08-14T21:24:03.7756762Z   Compute Unit:            64                                 
2025-08-14T21:24:03.7757072Z   SIMDs per CU:            0                                  
2025-08-14T21:24:03.7757398Z   Shader Engines:          0                                  
2025-08-14T21:24:03.7757724Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:24:03.7758076Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:24:03.7758524Z   Memory Properties:       
2025-08-14T21:24:03.7758773Z   Features:                None
2025-08-14T21:24:03.7759008Z   Pool Info:               
2025-08-14T21:24:03.7759218Z     Pool 1                   
2025-08-14T21:24:03.7759498Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7759815Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:24:03.7760264Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7760597Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7760962Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7761317Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7761659Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7761964Z     Pool 2                   
2025-08-14T21:24:03.7762315Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7762641Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:24:03.7762951Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7763293Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7763635Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7764006Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7764356Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7764647Z     Pool 3                   
2025-08-14T21:24:03.7764951Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:24:03.7765275Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:24:03.7765579Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7765915Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7766263Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7766761Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7767121Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7767412Z     Pool 4                   
2025-08-14T21:24:03.7767695Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7768006Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:24:03.7768324Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7768657Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7769013Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:24:03.7769373Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7769712Z       Accessible by all:       TRUE                               
2025-08-14T21:24:03.7770021Z   ISA Info:                
2025-08-14T21:24:03.7770231Z *******                  
2025-08-14T21:24:03.7770445Z Agent 3                  
2025-08-14T21:24:03.7770641Z *******                  
2025-08-14T21:24:03.7770882Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7771203Z   Uuid:                    GPU-3c8d38f625d62637               
2025-08-14T21:24:03.7771532Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7771872Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7772188Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7772517Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7772839Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7773177Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7773677Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7774001Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7774340Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7774645Z   Node:                    2                                  
2025-08-14T21:24:03.7774964Z   Device Type:             GPU                                
2025-08-14T21:24:03.7775419Z   Cache Info:              
2025-08-14T21:24:03.7775676Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7775969Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7776250Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7776581Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7776916Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7777272Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7777582Z   BDFID:                   12800                              
2025-08-14T21:24:03.7777906Z   Internal Node ID:        2                                  
2025-08-14T21:24:03.7778238Z   Compute Unit:            104                                
2025-08-14T21:24:03.7778551Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7778898Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7779237Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7779586Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7779931Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7780245Z   Memory Properties:       
2025-08-14T21:24:03.7780487Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7780800Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7781144Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7781474Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7781791Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7782048Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7782325Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7782593Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7782897Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7783226Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7783550Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7783845Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7784077Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7784356Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7784626Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7784942Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7791755Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7792162Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7792540Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7792847Z   Pool Info:               
2025-08-14T21:24:03.7793090Z     Pool 1                   
2025-08-14T21:24:03.7793371Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7793719Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7794043Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7794373Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7794930Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7795287Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7795640Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7795935Z     Pool 2                   
2025-08-14T21:24:03.7796220Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7796664Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7796986Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7797325Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7797667Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7798019Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7798353Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7798656Z     Pool 3                   
2025-08-14T21:24:03.7798914Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7799238Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7799561Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7799879Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7800241Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7800588Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7800943Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7801243Z     Pool 4                   
2025-08-14T21:24:03.7801511Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7801832Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7802222Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7802563Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7802902Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7803259Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7803593Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7803894Z   ISA Info:                
2025-08-14T21:24:03.7804114Z     ISA 1                    
2025-08-14T21:24:03.7804397Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7804782Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7805138Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7805497Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7805846Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7806193Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7806538Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7806856Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7807148Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7807432Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7807732Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7808026Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7808334Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7808585Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7808863Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7809158Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7809678Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7809986Z *******                  
2025-08-14T21:24:03.7810192Z Agent 4                  
2025-08-14T21:24:03.7810399Z *******                  
2025-08-14T21:24:03.7810627Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7810941Z   Uuid:                    GPU-77e549644b650cda               
2025-08-14T21:24:03.7811442Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7811777Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7812114Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7812441Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7812779Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7813111Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7813453Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7813803Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7814125Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7814441Z   Node:                    3                                  
2025-08-14T21:24:03.7814745Z   Device Type:             GPU                                
2025-08-14T21:24:03.7815049Z   Cache Info:              
2025-08-14T21:24:03.7815289Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7815583Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7815864Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7816182Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7816523Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7816847Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7817170Z   BDFID:                   13568                              
2025-08-14T21:24:03.7817485Z   Internal Node ID:        3                                  
2025-08-14T21:24:03.7817819Z   Compute Unit:            104                                
2025-08-14T21:24:03.7818123Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7818448Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7818791Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7819128Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7819480Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7819777Z   Memory Properties:       
2025-08-14T21:24:03.7820020Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7820322Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7820661Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7821009Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7821309Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7821573Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7821835Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7822097Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7822390Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7822728Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7823062Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7823353Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7823594Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7823859Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7824134Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7824587Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7824969Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7825315Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7825651Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7826074Z   Pool Info:               
2025-08-14T21:24:03.7826289Z     Pool 1                   
2025-08-14T21:24:03.7826571Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7826890Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7827210Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7827535Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7827892Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7828258Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7828597Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7828901Z     Pool 2                   
2025-08-14T21:24:03.7829163Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7829488Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7829799Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7830142Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7830491Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7830842Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7831194Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7831487Z     Pool 3                   
2025-08-14T21:24:03.7831767Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7832086Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7832409Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7832752Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7833089Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7833445Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7833792Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7834091Z     Pool 4                   
2025-08-14T21:24:03.7834357Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7834679Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7834998Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7835330Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7835688Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7836037Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7836381Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7836672Z   ISA Info:                
2025-08-14T21:24:03.7836898Z     ISA 1                    
2025-08-14T21:24:03.7837166Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7837531Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7837884Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7838224Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7838585Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7838910Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7839426Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7839758Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7840052Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7840334Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7840595Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7841037Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7841330Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7841589Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7841857Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7842190Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7842503Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7842789Z *******                  
2025-08-14T21:24:03.7843005Z Agent 5                  
2025-08-14T21:24:03.7843202Z *******                  
2025-08-14T21:24:03.7843438Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7843740Z   Uuid:                    GPU-e37f2e040c07ff9c               
2025-08-14T21:24:03.7844080Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7844417Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7844747Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7845074Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7845399Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7845737Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7846054Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7846378Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7846698Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7847008Z   Node:                    4                                  
2025-08-14T21:24:03.7847317Z   Device Type:             GPU                                
2025-08-14T21:24:03.7847600Z   Cache Info:              
2025-08-14T21:24:03.7847848Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7848131Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7848424Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7848729Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7849070Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7849420Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7849721Z   BDFID:                   4352                               
2025-08-14T21:24:03.7850052Z   Internal Node ID:        4                                  
2025-08-14T21:24:03.7850376Z   Compute Unit:            104                                
2025-08-14T21:24:03.7850691Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7851007Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7851346Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7851698Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7852042Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7852356Z   Memory Properties:       
2025-08-14T21:24:03.7852594Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7852912Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7853290Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7853630Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7854100Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7854374Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7854644Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7854901Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7855200Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7855682Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7856015Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7856304Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7856549Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7856831Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7857100Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7857421Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7857784Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7858135Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7858465Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7858763Z   Pool Info:               
2025-08-14T21:24:03.7858995Z     Pool 1                   
2025-08-14T21:24:03.7859268Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7859619Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7859935Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7860277Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7860628Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7860992Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7861345Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7861638Z     Pool 2                   
2025-08-14T21:24:03.7861917Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7862232Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7862552Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7862884Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7863240Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7863597Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7863938Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7864240Z     Pool 3                   
2025-08-14T21:24:03.7864499Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7864826Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7865133Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7865471Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7865822Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7866167Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7866516Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7866807Z     Pool 4                   
2025-08-14T21:24:03.7867060Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7867359Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7867677Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7868016Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7868514Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7868887Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7869230Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7869542Z   ISA Info:                
2025-08-14T21:24:03.7869761Z     ISA 1                    
2025-08-14T21:24:03.7870046Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7870548Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7870912Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7871264Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7871628Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7871960Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7872290Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7872624Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7872900Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7873201Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7873488Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7873793Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7874110Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7874363Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7874660Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7874945Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7875272Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7875583Z *******                  
2025-08-14T21:24:03.7875798Z Agent 6                  
2025-08-14T21:24:03.7876022Z *******                  
2025-08-14T21:24:03.7876260Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7876582Z   Uuid:                    GPU-daa9355c85671e61               
2025-08-14T21:24:03.7876913Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7877265Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7877597Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7877941Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7902353Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7903051Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7903453Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7903819Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7904162Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7904486Z   Node:                    5                                  
2025-08-14T21:24:03.7904787Z   Device Type:             GPU                                
2025-08-14T21:24:03.7905093Z   Cache Info:              
2025-08-14T21:24:03.7905335Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7905642Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7905946Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7906267Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7906605Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7906934Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7907248Z   BDFID:                   5120                               
2025-08-14T21:24:03.7907557Z   Internal Node ID:        5                                  
2025-08-14T21:24:03.7908147Z   Compute Unit:            104                                
2025-08-14T21:24:03.7908510Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7908838Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7909189Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7909700Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7910074Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7910388Z   Memory Properties:       
2025-08-14T21:24:03.7910653Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7910964Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7911321Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7911677Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7911990Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7912273Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7912550Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7912830Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7913130Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7913480Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7913828Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7914132Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7914382Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7914658Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7914944Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7915258Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7915650Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7916015Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7916346Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7916651Z   Pool Info:               
2025-08-14T21:24:03.7916874Z     Pool 1                   
2025-08-14T21:24:03.7917171Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7917509Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7917844Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7918177Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7918541Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7918911Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7919267Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7919586Z     Pool 2                   
2025-08-14T21:24:03.7919868Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7920214Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7920527Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7920875Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7921241Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7921589Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7921944Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7922320Z     Pool 3                   
2025-08-14T21:24:03.7922597Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7922910Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7923396Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7923746Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7924089Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7924453Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7924801Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7925262Z     Pool 4                   
2025-08-14T21:24:03.7925522Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7925835Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7926150Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7926476Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7926829Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7927184Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7927531Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7927824Z   ISA Info:                
2025-08-14T21:24:03.7928051Z     ISA 1                    
2025-08-14T21:24:03.7928336Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7928690Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7929045Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7929381Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7929738Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7930059Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7930392Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7930718Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7930992Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7931275Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7931539Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7931838Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7932140Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7932399Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7932672Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7932953Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7933268Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7933549Z *******                  
2025-08-14T21:24:03.7933759Z Agent 7                  
2025-08-14T21:24:03.7933953Z *******                  
2025-08-14T21:24:03.7934195Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7934503Z   Uuid:                    GPU-fe05c03760243cdf               
2025-08-14T21:24:03.7934852Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7935197Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7935518Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7935844Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7936158Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7936499Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7936819Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7937150Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7937478Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7937911Z   Node:                    6                                  
2025-08-14T21:24:03.7938234Z   Device Type:             GPU                                
2025-08-14T21:24:03.7938519Z   Cache Info:              
2025-08-14T21:24:03.7938760Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7939043Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7939475Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7939789Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7940128Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7940460Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7940780Z   BDFID:                   44544                              
2025-08-14T21:24:03.7941114Z   Internal Node ID:        6                                  
2025-08-14T21:24:03.7941446Z   Compute Unit:            104                                
2025-08-14T21:24:03.7941779Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7942102Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7942446Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7942787Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7943144Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7943462Z   Memory Properties:       
2025-08-14T21:24:03.7943711Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7944027Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7944358Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7944699Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7945008Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7945279Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7945551Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7945826Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7946139Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7946464Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7946799Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7947089Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7947331Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7947602Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7947880Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7948197Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7948561Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7948927Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7949263Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7949561Z   Pool Info:               
2025-08-14T21:24:03.7949771Z     Pool 1                   
2025-08-14T21:24:03.7950056Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7950397Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7950715Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7951058Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7951403Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7951768Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7952104Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7952401Z     Pool 2                   
2025-08-14T21:24:03.7952805Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7953144Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7953461Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7953789Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7954292Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7954636Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7954983Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7955276Z     Pool 3                   
2025-08-14T21:24:03.7955545Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7955865Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7956173Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7956519Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7956859Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7957211Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7957535Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7957828Z     Pool 4                   
2025-08-14T21:24:03.7958089Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7958390Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7958714Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7959042Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7959406Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7959746Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7960092Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7960391Z   ISA Info:                
2025-08-14T21:24:03.7960595Z     ISA 1                    
2025-08-14T21:24:03.7960870Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7961222Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7961574Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7961906Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7962318Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7962651Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7962973Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7963298Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7963567Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7963858Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7964120Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7964423Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7964730Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7964979Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7965261Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7965528Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7965841Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7966125Z *******                  
2025-08-14T21:24:03.7966348Z Agent 8                  
2025-08-14T21:24:03.7966544Z *******                  
2025-08-14T21:24:03.7966785Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7967249Z   Uuid:                    GPU-e7591100b7dd4825               
2025-08-14T21:24:03.7967579Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.7967922Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.7968239Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.7968713Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.7969036Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.7969375Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.7969717Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.7970035Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.7970357Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.7970647Z   Node:                    7                                  
2025-08-14T21:24:03.7970957Z   Device Type:             GPU                                
2025-08-14T21:24:03.7971239Z   Cache Info:              
2025-08-14T21:24:03.7971490Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.7971766Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.7972061Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.7972380Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.7972708Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.7973038Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.7973338Z   BDFID:                   45824                              
2025-08-14T21:24:03.7973664Z   Internal Node ID:        7                                  
2025-08-14T21:24:03.7973981Z   Compute Unit:            104                                
2025-08-14T21:24:03.7974309Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.7974641Z   Shader Engines:          8                                  
2025-08-14T21:24:03.7974969Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.7975317Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.7975658Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.7975977Z   Memory Properties:       
2025-08-14T21:24:03.7976216Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.7976526Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.7976872Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.7977202Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7977512Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7977761Z     x                        1024(0x400)                        
2025-08-14T21:24:03.7978042Z     y                        1024(0x400)                        
2025-08-14T21:24:03.7978305Z     z                        1024(0x400)                        
2025-08-14T21:24:03.7978620Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.7978959Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.7979299Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7979630Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.7979864Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7980146Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7980415Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7980736Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.7981086Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.7981435Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.7981923Z   IOMMU Support::          None                               
2025-08-14T21:24:03.7982218Z   Pool Info:               
2025-08-14T21:24:03.7982441Z     Pool 1                   
2025-08-14T21:24:03.7982711Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.7983046Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7983485Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7983827Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7984182Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7984532Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7984878Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7985172Z     Pool 2                   
2025-08-14T21:24:03.7985451Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.7985774Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7986095Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7986438Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7986783Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7987148Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7987483Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7987795Z     Pool 3                   
2025-08-14T21:24:03.7988063Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.7988393Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.7988717Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.7989047Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.7989409Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.7989761Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.7990112Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7990403Z     Pool 4                   
2025-08-14T21:24:03.7990683Z       Segment:                 GROUP                              
2025-08-14T21:24:03.7990995Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.7991299Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.7991639Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.7991980Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.7992330Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.7992660Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.7992959Z   ISA Info:                
2025-08-14T21:24:03.7993168Z     ISA 1                    
2025-08-14T21:24:03.7993443Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.7993816Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.7994160Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.7994520Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7994867Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.7995198Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.7995520Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.7995841Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.7996115Z         x                        1024(0x400)                        
2025-08-14T21:24:03.7996519Z         y                        1024(0x400)                        
2025-08-14T21:24:03.7996807Z         z                        1024(0x400)                        
2025-08-14T21:24:03.7997097Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.7997417Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.7997669Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7998064Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7998327Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.7998629Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.7998917Z *******                  
2025-08-14T21:24:03.7999137Z Agent 9                  
2025-08-14T21:24:03.7999340Z *******                  
2025-08-14T21:24:03.7999575Z   Name:                    gfx90a                             
2025-08-14T21:24:03.7999878Z   Uuid:                    GPU-d5dc726d0c1f5d2d               
2025-08-14T21:24:03.8000218Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.8000556Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.8000886Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.8001215Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.8001539Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.8001882Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.8002267Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.8002601Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.8002913Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.8003223Z   Node:                    8                                  
2025-08-14T21:24:03.8003521Z   Device Type:             GPU                                
2025-08-14T21:24:03.8003820Z   Cache Info:              
2025-08-14T21:24:03.8004064Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.8004334Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.8004626Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.8004938Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.8005278Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.8005600Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.8005913Z   BDFID:                   36352                              
2025-08-14T21:24:03.8006221Z   Internal Node ID:        8                                  
2025-08-14T21:24:03.8006546Z   Compute Unit:            104                                
2025-08-14T21:24:03.8006860Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.8007178Z   Shader Engines:          8                                  
2025-08-14T21:24:03.8007515Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.8007851Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.8008202Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.8008514Z   Memory Properties:       
2025-08-14T21:24:03.8008760Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.8009071Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.8009398Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.8009742Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.8010043Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.8010301Z     x                        1024(0x400)                        
2025-08-14T21:24:03.8010559Z     y                        1024(0x400)                        
2025-08-14T21:24:03.8010988Z     z                        1024(0x400)                        
2025-08-14T21:24:03.8011305Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.8011634Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.8011965Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.8012257Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.8012673Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8012948Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8013222Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8013546Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.8013902Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.8014254Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.8014588Z   IOMMU Support::          None                               
2025-08-14T21:24:03.8014890Z   Pool Info:               
2025-08-14T21:24:03.8015102Z     Pool 1                   
2025-08-14T21:24:03.8015382Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.8015715Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8016025Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8016369Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8016706Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8017067Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8017406Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8017706Z     Pool 2                   
2025-08-14T21:24:03.8017982Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.8018299Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8018613Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8018941Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8019297Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8019642Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8019997Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8020294Z     Pool 3                   
2025-08-14T21:24:03.8020553Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.8020872Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8021180Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8021507Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8021847Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8022203Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8022544Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8022830Z     Pool 4                   
2025-08-14T21:24:03.8023079Z       Segment:                 GROUP                              
2025-08-14T21:24:03.8023376Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.8023689Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.8024019Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.8024351Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.8024692Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.8025020Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8025312Z   ISA Info:                
2025-08-14T21:24:03.8025649Z     ISA 1                    
2025-08-14T21:24:03.8025926Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.8026285Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.8026618Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.8026953Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.8027416Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.8027741Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.8028049Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.8028357Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.8028618Z         x                        1024(0x400)                        
2025-08-14T21:24:03.8028891Z         y                        1024(0x400)                        
2025-08-14T21:24:03.8029159Z         z                        1024(0x400)                        
2025-08-14T21:24:03.8029447Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.8029739Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.8029977Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8030255Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8030527Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8030831Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.8031114Z *******                  
2025-08-14T21:24:03.8031310Z Agent 10                 
2025-08-14T21:24:03.8031505Z *******                  
2025-08-14T21:24:03.8031723Z   Name:                    gfx90a                             
2025-08-14T21:24:03.8032019Z   Uuid:                    GPU-144bb695d5984246               
2025-08-14T21:24:03.8032333Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:24:03.8032672Z   Vendor Name:             AMD                                
2025-08-14T21:24:03.8032986Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:24:03.8033296Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:24:03.8033618Z   Float Round Mode:        NEAR                               
2025-08-14T21:24:03.8033942Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:24:03.8034261Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:24:03.8034566Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:24:03.8034876Z   Queue Type:              MULTI                              
2025-08-14T21:24:03.8035160Z   Node:                    9                                  
2025-08-14T21:24:03.8035453Z   Device Type:             GPU                                
2025-08-14T21:24:03.8035729Z   Cache Info:              
2025-08-14T21:24:03.8035955Z     L1:                      16(0x10) KB                        
2025-08-14T21:24:03.8036230Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:24:03.8036503Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:24:03.8036807Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:24:03.8037125Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:24:03.8037451Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:24:03.8037754Z   BDFID:                   37632                              
2025-08-14T21:24:03.8038053Z   Internal Node ID:        9                                  
2025-08-14T21:24:03.8038371Z   Compute Unit:            104                                
2025-08-14T21:24:03.8038677Z   SIMDs per CU:            4                                  
2025-08-14T21:24:03.8039000Z   Shader Engines:          8                                  
2025-08-14T21:24:03.8039454Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:24:03.8039802Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:24:03.8040142Z   Coherent Host Access:    FALSE                              
2025-08-14T21:24:03.8040491Z   Memory Properties:       
2025-08-14T21:24:03.8040728Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:24:03.8041140Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:24:03.8041476Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:24:03.8041800Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.8042155Z   Workgroup Max Size per Dimension:
2025-08-14T21:24:03.8042411Z     x                        1024(0x400)                        
2025-08-14T21:24:03.8042668Z     y                        1024(0x400)                        
2025-08-14T21:24:03.8042929Z     z                        1024(0x400)                        
2025-08-14T21:24:03.8043217Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:24:03.8043543Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:24:03.8043864Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.8044155Z   Grid Max Size per Dimension:
2025-08-14T21:24:03.8044383Z     x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8044656Z     y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8044920Z     z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8045222Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:24:03.8045574Z   Packet Processor uCode:: 92                                 
2025-08-14T21:24:03.8045906Z   SDMA engine uCode::      9                                  
2025-08-14T21:24:03.8046235Z   IOMMU Support::          None                               
2025-08-14T21:24:03.8046522Z   Pool Info:               
2025-08-14T21:24:03.8046728Z     Pool 1                   
2025-08-14T21:24:03.8046992Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:24:03.8047307Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8047623Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8047944Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8048295Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8048642Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8048982Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8049273Z     Pool 2                   
2025-08-14T21:24:03.8049530Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:24:03.8049847Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8050151Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8050476Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8050811Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8051154Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8051488Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8051778Z     Pool 3                   
2025-08-14T21:24:03.8052039Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:24:03.8052342Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:24:03.8052649Z       Allocatable:             TRUE                               
2025-08-14T21:24:03.8052968Z       Alloc Granule:           4KB                                
2025-08-14T21:24:03.8053311Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:24:03.8053818Z       Alloc Alignment:         4KB                                
2025-08-14T21:24:03.8054161Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8054452Z     Pool 4                   
2025-08-14T21:24:03.8054694Z       Segment:                 GROUP                              
2025-08-14T21:24:03.8054995Z       Size:                    64(0x40) KB                        
2025-08-14T21:24:03.8055437Z       Allocatable:             FALSE                              
2025-08-14T21:24:03.8055762Z       Alloc Granule:           0KB                                
2025-08-14T21:24:03.8056106Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:24:03.8056439Z       Alloc Alignment:         0KB                                
2025-08-14T21:24:03.8056773Z       Accessible by all:       FALSE                              
2025-08-14T21:24:03.8057055Z   ISA Info:                
2025-08-14T21:24:03.8057261Z     ISA 1                    
2025-08-14T21:24:03.8057528Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:24:03.8057882Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:24:03.8058214Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:24:03.8058549Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.8058902Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:24:03.8059228Z       Fast f16:                TRUE                               
2025-08-14T21:24:03.8059548Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:24:03.8059852Z       Workgroup Max Size per Dimension:
2025-08-14T21:24:03.8060124Z         x                        1024(0x400)                        
2025-08-14T21:24:03.8060386Z         y                        1024(0x400)                        
2025-08-14T21:24:03.8060649Z         z                        1024(0x400)                        
2025-08-14T21:24:03.8060944Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:24:03.8061227Z       Grid Max Size per Dimension:
2025-08-14T21:24:03.8061468Z         x                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8061734Z         y                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8062003Z         z                        4294967295(0xffffffff)             
2025-08-14T21:24:03.8062302Z       FBarrier Max Size:       32                                 
2025-08-14T21:24:03.8062590Z *** Done ***             
2025-08-14T21:24:03.8081402Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:24:03.8081792Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:24:03.8082481Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-14T21:24:03.8083047Z if [[ $ngpu -eq 0 ]]; then
2025-08-14T21:24:03.8083360Z     echo "Error: Failed to detect any GPUs on the runner"
2025-08-14T21:24:03.8083670Z     echo "$msg"
2025-08-14T21:24:03.8083867Z     exit 1
2025-08-14T21:24:03.8084053Z fi
2025-08-14T21:24:03.8102987Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.8103306Z env:
2025-08-14T21:24:03.8103493Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.8103698Z ##[endgroup]
2025-08-14T21:24:03.9715570Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-08-14T21:24:03.9715945Z with:
2025-08-14T21:24:03.9716147Z   diskspace-cutoff: 70
2025-08-14T21:24:03.9716356Z env:
2025-08-14T21:24:03.9716552Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.9716769Z ##[endgroup]
2025-08-14T21:24:03.9753223Z ##[group]Run set -ex
2025-08-14T21:24:03.9753505Z set -ex
2025-08-14T21:24:03.9753720Z diskspace_cutoff=70
2025-08-14T21:24:03.9754042Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-08-14T21:24:03.9754683Z if [ ! -d "$docker_root_dir" ]; then
2025-08-14T21:24:03.9755163Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-08-14T21:24:03.9755585Z     exit 0
2025-08-14T21:24:03.9755775Z fi
2025-08-14T21:24:03.9756145Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T21:24:03.9757162Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-14T21:24:03.9757811Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-08-14T21:24:03.9758141Z     docker system prune -af
2025-08-14T21:24:03.9758571Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T21:24:03.9759065Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-08-14T21:24:03.9759575Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-08-14T21:24:03.9760028Z         echo "$msg"
2025-08-14T21:24:03.9760248Z         exit 1
2025-08-14T21:24:03.9760459Z     else
2025-08-14T21:24:03.9760716Z         difference=$((diskspace - diskspace_new))
2025-08-14T21:24:03.9761063Z         echo "Diskspace saved: $difference percent"
2025-08-14T21:24:03.9761353Z     fi
2025-08-14T21:24:03.9761537Z fi
2025-08-14T21:24:03.9782453Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:03.9782785Z env:
2025-08-14T21:24:03.9782973Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:03.9783204Z ##[endgroup]
2025-08-14T21:24:03.9825891Z + diskspace_cutoff=70
2025-08-14T21:24:03.9831808Z ++ docker info -f '{{.DockerRootDir}}'
2025-08-14T21:24:04.0382400Z + docker_root_dir=/media/4TB/docker-rootless
2025-08-14T21:24:04.0382810Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-08-14T21:24:04.0392945Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-08-14T21:24:04.0396166Z ++ sed -n 2p
2025-08-14T21:24:04.0397978Z ++ sed s/%//
2025-08-14T21:24:04.0400089Z ++ sed 's/ //'
2025-08-14T21:24:04.0424064Z + diskspace=15
2025-08-14T21:24:04.0425014Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-08-14T21:24:04.0426003Z + [[ 15 -ge 70 ]]
2025-08-14T21:24:04.0460905Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-08-14T21:24:04.0461330Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-08-14T21:24:04.0461649Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-08-14T21:24:04.0461946Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-08-14T21:24:04.0462327Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:24:04.0462685Z 
2025-08-14T21:24:04.0462966Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-08-14T21:24:04.0463320Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-08-14T21:24:04.0463632Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-08-14T21:24:04.0464030Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:24:04.0464408Z 
2025-08-14T21:24:04.0464635Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-08-14T21:24:04.0464912Z rm -rf "${RUNNER_DOCS_DIR}"
2025-08-14T21:24:04.0465182Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-08-14T21:24:04.0465510Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:24:04.0486539Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:04.0486886Z env:
2025-08-14T21:24:04.0487073Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.0487303Z ##[endgroup]
2025-08-14T21:24:04.0693235Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:24:04.0693993Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:24:04.0694432Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:24:04.0714795Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:04.0715131Z env:
2025-08-14T21:24:04.0715313Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.0715933Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:04.0716491Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:04.0716988Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:04.0717339Z ##[endgroup]
2025-08-14T21:24:04.0836077Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-08-14T21:24:04.0836736Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-08-14T21:24:04.0837220Z # Add render group for container creation.
2025-08-14T21:24:04.0837606Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-08-14T21:24:04.0838050Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-08-14T21:24:04.0838506Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-08-14T21:24:04.0838892Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-08-14T21:24:04.0839208Z else
2025-08-14T21:24:04.0839430Z   DEVICE_FLAG="--device /dev/dri"
2025-08-14T21:24:04.0839681Z fi
2025-08-14T21:24:04.0840089Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-08-14T21:24:04.0840715Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-08-14T21:24:04.0841306Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-08-14T21:24:04.0841919Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-08-14T21:24:04.0843045Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-08-14T21:24:04.0863976Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:04.0864304Z env:
2025-08-14T21:24:04.0864501Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.0864874Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:04.0865403Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:04.0865902Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:04.0866248Z ##[endgroup]
2025-08-14T21:24:04.0981876Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:24:04.0982310Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:24:04.0982613Z if [[ $ngpu -lt 4 ]]; then
2025-08-14T21:24:04.0983037Z   echo "Error: only $ngpu GPU(s) detected, at least 4 GPUs are needed for distributed jobs"
2025-08-14T21:24:04.0983444Z   exit 1
2025-08-14T21:24:04.0983662Z fi
2025-08-14T21:24:04.1002656Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:04.1003000Z env:
2025-08-14T21:24:04.1003189Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.1003568Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:04.1004140Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:04.1004654Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:04.1005549Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:04.1006275Z ##[endgroup]
2025-08-14T21:24:04.2602307Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-08-14T21:24:04.2602807Z with:
2025-08-14T21:24:04.2603128Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-08-14T21:24:04.2603808Z   aws-region: us-east-1
2025-08-14T21:24:04.2604043Z   role-duration-seconds: 18000
2025-08-14T21:24:04.2604314Z   audience: sts.amazonaws.com
2025-08-14T21:24:04.2604541Z env:
2025-08-14T21:24:04.2604734Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.2605098Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:04.2605656Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:04.2606161Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:04.2607039Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:04.2607765Z ##[endgroup]
2025-08-14T21:24:04.5881605Z Assuming role with OIDC
2025-08-14T21:24:04.8022892Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-08-14T21:24:04.8706073Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076
2025-08-14T21:24:04.8706532Z with:
2025-08-14T21:24:04.8706752Z   mask-password: true
2025-08-14T21:24:04.8706992Z   registry-type: private
2025-08-14T21:24:04.8707233Z   skip-logout: false
2025-08-14T21:24:04.8707448Z env:
2025-08-14T21:24:04.8707644Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:04.8708021Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:04.8708569Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:04.8709080Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:04.8709940Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:04.8710734Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:04.8711028Z   AWS_REGION: us-east-1
2025-08-14T21:24:04.8711609Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:04.8711960Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:04.8716680Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:04.8716916Z ##[endgroup]
2025-08-14T21:24:05.2745113Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7234966Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-08-14T21:24:05.7235416Z with:
2025-08-14T21:24:05.7236040Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7236740Z   use-custom-docker-registry: true
2025-08-14T21:24:05.7237027Z   docker-build-dir: .ci/docker
2025-08-14T21:24:05.7237298Z   docker-build-script: ./build.sh
2025-08-14T21:24:05.7237566Z   working-directory: .
2025-08-14T21:24:05.7237862Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7238236Z   force-push: false
2025-08-14T21:24:05.7238448Z env:
2025-08-14T21:24:05.7238647Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:05.7239025Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:05.7239568Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:05.7240114Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:05.7240953Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:05.7241699Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:05.7241954Z   AWS_REGION: us-east-1
2025-08-14T21:24:05.7242410Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:05.7242813Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:05.7247516Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:05.7248036Z ##[endgroup]
2025-08-14T21:24:05.7267811Z ##[group]Run set -ex
2025-08-14T21:24:05.7268082Z set -ex
2025-08-14T21:24:05.7268292Z 
2025-08-14T21:24:05.7268636Z # If the docker build directory or the build script doesn't exist, the action will
2025-08-14T21:24:05.7269210Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-08-14T21:24:05.7269685Z # job could then download the pre-built image as usual
2025-08-14T21:24:05.7270265Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-08-14T21:24:05.7270814Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7271100Z else
2025-08-14T21:24:05.7271330Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7271706Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7272068Z 
2025-08-14T21:24:05.7272535Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-08-14T21:24:05.7273065Z   exit 0
2025-08-14T21:24:05.7273265Z fi
2025-08-14T21:24:05.7273457Z 
2025-08-14T21:24:05.7273761Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-08-14T21:24:05.7274273Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-08-14T21:24:05.7274726Z   # use it as it is, but first let's extract the tag
2025-08-14T21:24:05.7275141Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-08-14T21:24:05.7275576Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7275992Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7276326Z else
2025-08-14T21:24:05.7276578Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-08-14T21:24:05.7276912Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-08-14T21:24:05.7277249Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-08-14T21:24:05.7277559Z   fi
2025-08-14T21:24:05.7278208Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-08-14T21:24:05.7278767Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7279318Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7279914Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7280307Z fi
2025-08-14T21:24:05.7301821Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:05.7302170Z env:
2025-08-14T21:24:05.7302374Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:05.7302777Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:05.7303328Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:05.7303835Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:05.7304702Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:05.7305507Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:05.7305775Z   AWS_REGION: us-east-1
2025-08-14T21:24:05.7306115Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:05.7306461Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:05.7311151Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:05.7311387Z   REPO_NAME: pytorch
2025-08-14T21:24:05.7312017Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7312919Z   DOCKER_BUILD_DIR: .ci/docker
2025-08-14T21:24:05.7313182Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-08-14T21:24:05.7313521Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7313869Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-08-14T21:24:05.7314134Z   CUSTOM_TAG_PREFIX: 
2025-08-14T21:24:05.7314360Z ##[endgroup]
2025-08-14T21:24:05.7355269Z + [[ -d .ci/docker ]]
2025-08-14T21:24:05.7355677Z + [[ -f .ci/docker/./build.sh ]]
2025-08-14T21:24:05.7356079Z + [[ true == \t\r\u\e ]]
2025-08-14T21:24:05.7356594Z + echo skip=false
2025-08-14T21:24:05.7380768Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-08-14T21:24:05.7382119Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7382780Z ++ awk -F '[:,]' '{print $2}'
2025-08-14T21:24:05.7405466Z + DOCKER_TAG=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7406124Z + echo docker-tag=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7407252Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7440639Z ##[group]Run set +e
2025-08-14T21:24:05.7440961Z set +e
2025-08-14T21:24:05.7441174Z set -x
2025-08-14T21:24:05.7441380Z 
2025-08-14T21:24:05.7441583Z login() {
2025-08-14T21:24:05.7442017Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-08-14T21:24:05.7442562Z }
2025-08-14T21:24:05.7442761Z 
2025-08-14T21:24:05.7442952Z retry () {
2025-08-14T21:24:05.7443234Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-08-14T21:24:05.7443519Z }
2025-08-14T21:24:05.7443705Z 
2025-08-14T21:24:05.7443920Z retry login "${DOCKER_REGISTRY}"
2025-08-14T21:24:05.7444196Z 
2025-08-14T21:24:05.7444399Z START_TIME=$(date +%s)
2025-08-14T21:24:05.7444929Z # Wait up to 120 minutes
2025-08-14T21:24:05.7445277Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-08-14T21:24:05.7445700Z   # Check if image already exists, if it does then skip building it
2025-08-14T21:24:05.7446121Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-08-14T21:24:05.7446451Z     exit 0
2025-08-14T21:24:05.7446662Z   fi
2025-08-14T21:24:05.7446856Z 
2025-08-14T21:24:05.7447182Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-08-14T21:24:05.7447735Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-08-14T21:24:05.7448295Z   # latter, it will wait for the Docker images to become available before continuing
2025-08-14T21:24:05.7448736Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-08-14T21:24:05.7449088Z     # It's a Docker build job, let's build the image
2025-08-14T21:24:05.7449397Z     break
2025-08-14T21:24:05.7449603Z   else
2025-08-14T21:24:05.7449903Z     # It's a regular build job, wait for the image to become available
2025-08-14T21:24:05.7450262Z     sleep 300
2025-08-14T21:24:05.7450475Z   fi
2025-08-14T21:24:05.7450665Z done
2025-08-14T21:24:05.7450837Z 
2025-08-14T21:24:05.7451138Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-08-14T21:24:05.7451627Z # be empty.  The default action would be to continue rebuild the image
2025-08-14T21:24:05.7452291Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-08-14T21:24:05.7452678Z   # if we're on the base branch then use the parent commit
2025-08-14T21:24:05.7453024Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-08-14T21:24:05.7453292Z else
2025-08-14T21:24:05.7453566Z   # otherwise we're on a PR, so use the most recent base commit
2025-08-14T21:24:05.7453977Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-08-14T21:24:05.7454289Z fi
2025-08-14T21:24:05.7454476Z 
2025-08-14T21:24:05.7454680Z if [[ -z "${MERGE_BASE}" ]]; then
2025-08-14T21:24:05.7454977Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7455260Z 
2025-08-14T21:24:05.7455654Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-08-14T21:24:05.7456130Z   exit 0
2025-08-14T21:24:05.7456328Z fi
2025-08-14T21:24:05.7456516Z 
2025-08-14T21:24:05.7456779Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-08-14T21:24:05.7457357Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-08-14T21:24:05.7457849Z   exit 1
2025-08-14T21:24:05.7458042Z fi
2025-08-14T21:24:05.7458230Z 
2025-08-14T21:24:05.7458537Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-08-14T21:24:05.7459087Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-08-14T21:24:05.7459593Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-08-14T21:24:05.7460168Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-08-14T21:24:05.7460824Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-08-14T21:24:05.7461224Z fi
2025-08-14T21:24:05.7461412Z 
2025-08-14T21:24:05.7461653Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:05.7481915Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:05.7482342Z env:
2025-08-14T21:24:05.7482551Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:05.7483118Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:05.7483700Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:05.7484220Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:05.7485068Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:05.7485836Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:05.7486097Z   AWS_REGION: us-east-1
2025-08-14T21:24:05.7486519Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:05.7486886Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:05.7491588Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:05.7491829Z   DOCKER_BUILD_DIR: .ci/docker
2025-08-14T21:24:05.7492131Z   BASE_REVISION: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:24:05.7492850Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7493669Z   DOCKER_TAG: pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:05.7494192Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7494532Z   DOCKER_PUSH: 
2025-08-14T21:24:05.7494743Z ##[endgroup]
2025-08-14T21:24:05.7539451Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7539902Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:05.7544517Z + aws ecr get-login-password --region us-east-1
2025-08-14T21:24:05.7546868Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:06.9874491Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-08-14T21:24:06.9875694Z Configure a credential helper to remove this warning. See
2025-08-14T21:24:06.9876989Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-08-14T21:24:06.9877707Z 
2025-08-14T21:24:06.9881510Z Login Succeeded
2025-08-14T21:24:06.9921956Z ++ date +%s
2025-08-14T21:24:06.9939629Z + START_TIME=1755206646
2025-08-14T21:24:06.9946180Z ++ date +%s
2025-08-14T21:24:06.9961604Z + [[ 1755199446 -lt 1755206646 ]]
2025-08-14T21:24:06.9962684Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:07.8782738Z {
2025-08-14T21:24:07.8783447Z 	"schemaVersion": 2,
2025-08-14T21:24:07.8784434Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-08-14T21:24:07.8785386Z 	"config": {
2025-08-14T21:24:07.8786325Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-08-14T21:24:07.8787071Z 		"size": 28646,
2025-08-14T21:24:07.8787829Z 		"digest": "sha256:460acde6d8cbae0e961d7e3597d10525c881798e91edaf1ab0df7a07e0905a24"
2025-08-14T21:24:07.8788491Z 	},
2025-08-14T21:24:07.8788833Z 	"layers": [
2025-08-14T21:24:07.8789239Z 		{
2025-08-14T21:24:07.8789708Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8790189Z 			"size": 30448173,
2025-08-14T21:24:07.8790772Z 			"digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495"
2025-08-14T21:24:07.8791356Z 		},
2025-08-14T21:24:07.8791634Z 		{
2025-08-14T21:24:07.8792291Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8792889Z 			"size": 1554,
2025-08-14T21:24:07.8793472Z 			"digest": "sha256:c7b4a852a45516e27a9256df90878663d770f96d271d6155d43be78cc5225eef"
2025-08-14T21:24:07.8794027Z 		},
2025-08-14T21:24:07.8794305Z 		{
2025-08-14T21:24:07.8794847Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8795375Z 			"size": 313280717,
2025-08-14T21:24:07.8795830Z 			"digest": "sha256:c5628c7e7b33c6491651d9cce9ee596279d693cc7c3dcfa912aca94f75341cf7"
2025-08-14T21:24:07.8796854Z 		},
2025-08-14T21:24:07.8797159Z 		{
2025-08-14T21:24:07.8797653Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8798159Z 			"size": 704,
2025-08-14T21:24:07.8798740Z 			"digest": "sha256:836ab08052e8eb2bae68e69ae086fd23a5f04a8491c320718ab47f84f03aebb1"
2025-08-14T21:24:07.8799327Z 		},
2025-08-14T21:24:07.8799625Z 		{
2025-08-14T21:24:07.8800027Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8800573Z 			"size": 1218,
2025-08-14T21:24:07.8801003Z 			"digest": "sha256:0802c59adcde1d758d228271cc5f1bf6acbc597053ff8e8f3c1073d59a85bf9d"
2025-08-14T21:24:07.8801559Z 		},
2025-08-14T21:24:07.8801932Z 		{
2025-08-14T21:24:07.8802457Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8802998Z 			"size": 485,
2025-08-14T21:24:07.8803630Z 			"digest": "sha256:e97311a6a967664cbe10c5027a1ec60c514caa9a1160167d8363088fd1f9fe09"
2025-08-14T21:24:07.8804152Z 		},
2025-08-14T21:24:07.8804505Z 		{
2025-08-14T21:24:07.8804918Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8805355Z 			"size": 110343847,
2025-08-14T21:24:07.8805871Z 			"digest": "sha256:fe9f48a18b7ef6bdbd006ea4cb784b7e73e0daa2409a1a053c02bafed7993529"
2025-08-14T21:24:07.8806409Z 		},
2025-08-14T21:24:07.8806667Z 		{
2025-08-14T21:24:07.8807135Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8807609Z 			"size": 4242,
2025-08-14T21:24:07.8808115Z 			"digest": "sha256:4d3edeebe7da3872c026bd4e61d5f63aee7f760e1c8cf0eb226e1d95e528735b"
2025-08-14T21:24:07.8808999Z 		},
2025-08-14T21:24:07.8809283Z 		{
2025-08-14T21:24:07.8809779Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8810249Z 			"size": 1709,
2025-08-14T21:24:07.8810699Z 			"digest": "sha256:5a5cc76ada432cccf7d18e0eb79379afb95deaaa7afec482406267924d291ae4"
2025-08-14T21:24:07.8811279Z 		},
2025-08-14T21:24:07.8811565Z 		{
2025-08-14T21:24:07.8811951Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8812525Z 			"size": 724,
2025-08-14T21:24:07.8812985Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:24:07.8813557Z 		},
2025-08-14T21:24:07.8813943Z 		{
2025-08-14T21:24:07.8814337Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8814900Z 			"size": 3244135942,
2025-08-14T21:24:07.8815382Z 			"digest": "sha256:ee623dde94b52da337d3cc0e356115ccbc33b9b5ec03f9e06fce9f05ba3572ad"
2025-08-14T21:24:07.8815912Z 		},
2025-08-14T21:24:07.8816335Z 		{
2025-08-14T21:24:07.8816801Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8817348Z 			"size": 380,
2025-08-14T21:24:07.8817929Z 			"digest": "sha256:d6226eb61f823984003d5ac28f4d66fec9b27baf5d54a9513286483f5912cd88"
2025-08-14T21:24:07.8818447Z 		},
2025-08-14T21:24:07.8818782Z 		{
2025-08-14T21:24:07.8819292Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8819776Z 			"size": 234112,
2025-08-14T21:24:07.8820312Z 			"digest": "sha256:6de917b18d3a540adb3b7e446e9802f89a69d39f804f40d25751639b6cea7c33"
2025-08-14T21:24:07.8820840Z 		},
2025-08-14T21:24:07.8821078Z 		{
2025-08-14T21:24:07.8821533Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8822022Z 			"size": 793,
2025-08-14T21:24:07.8822455Z 			"digest": "sha256:76a69b57b6837bef07dbc1b481cf28a62dfd7c7063219d9f6e0d0d63067653c7"
2025-08-14T21:24:07.8823050Z 		},
2025-08-14T21:24:07.8823357Z 		{
2025-08-14T21:24:07.8823784Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8824392Z 			"size": 106,
2025-08-14T21:24:07.8824853Z 			"digest": "sha256:451d7a75db7137d6ca0ccb4e3635d4a16f3287db4cc2a3a4d3dbc2d3b0266f06"
2025-08-14T21:24:07.8825434Z 		},
2025-08-14T21:24:07.8825703Z 		{
2025-08-14T21:24:07.8826096Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8826826Z 			"size": 1496,
2025-08-14T21:24:07.8827292Z 			"digest": "sha256:acf5babd87f23aa905883eb434073e9a00ff41679134f2f4827dd86949f5a9d9"
2025-08-14T21:24:07.8828044Z 		},
2025-08-14T21:24:07.8828475Z 		{
2025-08-14T21:24:07.8828980Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8829632Z 			"size": 453553322,
2025-08-14T21:24:07.8830106Z 			"digest": "sha256:06e2fbb6cb2eeccd028bfaabb2e674bf25c9d84d3fa91c9e2f03bcf765f92599"
2025-08-14T21:24:07.8830732Z 		},
2025-08-14T21:24:07.8831007Z 		{
2025-08-14T21:24:07.8831383Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8831963Z 			"size": 163,
2025-08-14T21:24:07.8832422Z 			"digest": "sha256:87ddced081ff535d58b842d961844bf1b5979ddc8ad7f2d31f606cd6c73ec758"
2025-08-14T21:24:07.8832968Z 		},
2025-08-14T21:24:07.8833295Z 		{
2025-08-14T21:24:07.8833695Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8834231Z 			"size": 2484,
2025-08-14T21:24:07.8834792Z 			"digest": "sha256:1eb879bde5e5b9ce59b91eacb794a63e1180a55615094477886cba5016e993d7"
2025-08-14T21:24:07.8835282Z 		},
2025-08-14T21:24:07.8835673Z 		{
2025-08-14T21:24:07.8836019Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8836477Z 			"size": 8102124021,
2025-08-14T21:24:07.8837064Z 			"digest": "sha256:c3717fc4dc94ce7fdd93c87843ab97f72e05c0e87203ae15e48b87935b189a27"
2025-08-14T21:24:07.8837574Z 		},
2025-08-14T21:24:07.8837921Z 		{
2025-08-14T21:24:07.8838335Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8838972Z 			"size": 105,
2025-08-14T21:24:07.8839586Z 			"digest": "sha256:0c9ed178fd2ebc5a6630c12887a96b3f9884b85144d726435e77d33ab1ee11ab"
2025-08-14T21:24:07.8840095Z 		},
2025-08-14T21:24:07.8840376Z 		{
2025-08-14T21:24:07.8840799Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8841259Z 			"size": 612,
2025-08-14T21:24:07.8841718Z 			"digest": "sha256:b4626f6a341ca316ed5525f9c12e667b82f444b8cf673a0ad0bf8b78ccfd5bc9"
2025-08-14T21:24:07.8842383Z 		},
2025-08-14T21:24:07.8842684Z 		{
2025-08-14T21:24:07.8843006Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8843361Z 			"size": 677677354,
2025-08-14T21:24:07.8843720Z 			"digest": "sha256:4bf07717a6a2b5fd5a05d41b16dedebb413da5d50ae09457e112d5b42ecacbe1"
2025-08-14T21:24:07.8844130Z 		},
2025-08-14T21:24:07.8844306Z 		{
2025-08-14T21:24:07.8844582Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8844930Z 			"size": 111,
2025-08-14T21:24:07.8845270Z 			"digest": "sha256:f258db683e793ef4f283958f771de6376461ec4c2d2ac2947f2ad72a998f45ad"
2025-08-14T21:24:07.8845656Z 		},
2025-08-14T21:24:07.8845829Z 		{
2025-08-14T21:24:07.8846088Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8846434Z 			"size": 1555,
2025-08-14T21:24:07.8846793Z 			"digest": "sha256:2d70f6b6782f0d7c3342f35a89c20ca75e63acc6cffff00c2f5b0c4ae16aa20d"
2025-08-14T21:24:07.8847187Z 		},
2025-08-14T21:24:07.8847352Z 		{
2025-08-14T21:24:07.8847622Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8847963Z 			"size": 107,
2025-08-14T21:24:07.8848305Z 			"digest": "sha256:cfbe9750bae7f94f985be33742a90ac52adfdbe5870aa09b09ad6ce80db6541b"
2025-08-14T21:24:07.8848703Z 		},
2025-08-14T21:24:07.8848860Z 		{
2025-08-14T21:24:07.8849131Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8849465Z 			"size": 166,
2025-08-14T21:24:07.8849813Z 			"digest": "sha256:8d2ff9ef7f4c24e23debf7d02ee44f5acc6e3ed95168f970dfa061e270556036"
2025-08-14T21:24:07.8850219Z 		},
2025-08-14T21:24:07.8850384Z 		{
2025-08-14T21:24:07.8850652Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8850999Z 			"size": 2838566,
2025-08-14T21:24:07.8851341Z 			"digest": "sha256:59a73147541eccf1318db0147d0e40952ff983b3515bbc1b2586caa5d370c2e5"
2025-08-14T21:24:07.8851897Z 		},
2025-08-14T21:24:07.8852074Z 		{
2025-08-14T21:24:07.8852348Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8852691Z 			"size": 107,
2025-08-14T21:24:07.8853036Z 			"digest": "sha256:80e89acaf7bc105eb24090a7638ae51ec2ec270cf4e181e3002f3ef6cea7d2dc"
2025-08-14T21:24:07.8853427Z 		},
2025-08-14T21:24:07.8853594Z 		{
2025-08-14T21:24:07.8853860Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8854207Z 			"size": 802,
2025-08-14T21:24:07.8854536Z 			"digest": "sha256:c9c2b424b8e08d943dc259a3796d66eede3a1e93a6460df5db132c0036d3d6af"
2025-08-14T21:24:07.8854937Z 		},
2025-08-14T21:24:07.8855109Z 		{
2025-08-14T21:24:07.8855379Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8855724Z 			"size": 26113684,
2025-08-14T21:24:07.8856115Z 			"digest": "sha256:26060b8c841ee0f2c22485c0d51390219230b08ce81ce14c500e40be58488ac5"
2025-08-14T21:24:07.8856579Z 		},
2025-08-14T21:24:07.8856791Z 		{
2025-08-14T21:24:07.8857110Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8857517Z 			"size": 104,
2025-08-14T21:24:07.8857868Z 			"digest": "sha256:58b4bcac07b333b658450cb576e2bb832b11cd699d76f1f3846e739deb4a5738"
2025-08-14T21:24:07.8858260Z 		},
2025-08-14T21:24:07.8858439Z 		{
2025-08-14T21:24:07.8858711Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8859050Z 			"size": 425,
2025-08-14T21:24:07.8859392Z 			"digest": "sha256:18c4ed1ec491095788e352ae018afd84de0f251fbcfb8f74d5d893e1e9ab196d"
2025-08-14T21:24:07.8859960Z 		},
2025-08-14T21:24:07.8860126Z 		{
2025-08-14T21:24:07.8860394Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8860728Z 			"size": 19308715,
2025-08-14T21:24:07.8861072Z 			"digest": "sha256:4d7414e539c3ad1ab34e46032ca390dae2d25805222308bb02ecc1b35713b1a2"
2025-08-14T21:24:07.8861450Z 		},
2025-08-14T21:24:07.8861614Z 		{
2025-08-14T21:24:07.8861886Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8862235Z 			"size": 691,
2025-08-14T21:24:07.8862577Z 			"digest": "sha256:6738ba83282e002d92bff3d2b4951e3c1a67f5ec2c1bad2fd780c2f5d444748f"
2025-08-14T21:24:07.8862965Z 		},
2025-08-14T21:24:07.8863129Z 		{
2025-08-14T21:24:07.8863387Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8863734Z 			"size": 724,
2025-08-14T21:24:07.8864062Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:24:07.8864456Z 		},
2025-08-14T21:24:07.8864631Z 		{
2025-08-14T21:24:07.8864915Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8865258Z 			"size": 116,
2025-08-14T21:24:07.8865597Z 			"digest": "sha256:dfb0f24886393e1d394f1f433dc9346026679dafd7a60c3a93de17d94078c1ca"
2025-08-14T21:24:07.8865982Z 		},
2025-08-14T21:24:07.8866148Z 		{
2025-08-14T21:24:07.8866433Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8866783Z 			"size": 136,
2025-08-14T21:24:07.8867129Z 			"digest": "sha256:dc833b0762f2e144670a660f6b7ce62cec71a5fdd24df4e67b5c6173d5834451"
2025-08-14T21:24:07.8867527Z 		},
2025-08-14T21:24:07.8867704Z 		{
2025-08-14T21:24:07.8867987Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8868332Z 			"size": 139,
2025-08-14T21:24:07.8868672Z 			"digest": "sha256:8827df8ca2da347e0032d1bff3b0312437f711c5d0b5f2164f8a60c3368a9827"
2025-08-14T21:24:07.8869064Z 		},
2025-08-14T21:24:07.8869226Z 		{
2025-08-14T21:24:07.8869503Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8869863Z 			"size": 32,
2025-08-14T21:24:07.8870209Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-08-14T21:24:07.8870607Z 		},
2025-08-14T21:24:07.8870786Z 		{
2025-08-14T21:24:07.8871062Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8871552Z 			"size": 214,
2025-08-14T21:24:07.8871905Z 			"digest": "sha256:cd36d0dbfc741c82fee3a71ab009d02c22329a76463482a521b51f4e183248b3"
2025-08-14T21:24:07.8872292Z 		},
2025-08-14T21:24:07.8872465Z 		{
2025-08-14T21:24:07.8872744Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8873102Z 			"size": 347,
2025-08-14T21:24:07.8873440Z 			"digest": "sha256:4ba8e7a736c8199931fd7ff9931a5f17b7b931d0383a3e158f1b12b191a1d250"
2025-08-14T21:24:07.8873831Z 		},
2025-08-14T21:24:07.8874004Z 		{
2025-08-14T21:24:07.8874284Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8874641Z 			"size": 88301,
2025-08-14T21:24:07.8874977Z 			"digest": "sha256:d3f2353a1f26b127b1869d0eb92a134748c89b45f5defec4fcb2c356bc73c540"
2025-08-14T21:24:07.8875366Z 		},
2025-08-14T21:24:07.8875535Z 		{
2025-08-14T21:24:07.8875804Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8876149Z 			"size": 106,
2025-08-14T21:24:07.8876498Z 			"digest": "sha256:22c681dc84cd6e1f55b9874ead0fb9d31d471d5cb641391a3b3dd979c1502cc6"
2025-08-14T21:24:07.8876886Z 		},
2025-08-14T21:24:07.8877052Z 		{
2025-08-14T21:24:07.8877323Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8877656Z 			"size": 1666,
2025-08-14T21:24:07.8877986Z 			"digest": "sha256:519f3067ec6a99456e6fb280bd4b79e2478677684831299d601c9493523b67cc"
2025-08-14T21:24:07.8878363Z 		},
2025-08-14T21:24:07.8878529Z 		{
2025-08-14T21:24:07.8878797Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8879135Z 			"size": 724,
2025-08-14T21:24:07.8879624Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:24:07.8880012Z 		},
2025-08-14T21:24:07.8880176Z 		{
2025-08-14T21:24:07.8880444Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8880779Z 			"size": 139,
2025-08-14T21:24:07.8881120Z 			"digest": "sha256:6a8272f38d227d78a8f9b1ddf10bb73941fbd77e71ae3c582c27e08c0f6261c2"
2025-08-14T21:24:07.8881506Z 		},
2025-08-14T21:24:07.8881672Z 		{
2025-08-14T21:24:07.8881950Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8882356Z 			"size": 120,
2025-08-14T21:24:07.8882692Z 			"digest": "sha256:973e2484504fec2c1292e24122716507e8ca4717a35c142358174b60c0e401cf"
2025-08-14T21:24:07.8883075Z 		},
2025-08-14T21:24:07.8883249Z 		{
2025-08-14T21:24:07.8883517Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8883872Z 			"size": 5373253071,
2025-08-14T21:24:07.8884233Z 			"digest": "sha256:ac32a18ead25409bbcd67a8636c32d8af5852a347ea5503c7c9c6234cacf45a1"
2025-08-14T21:24:07.8884637Z 		},
2025-08-14T21:24:07.8884814Z 		{
2025-08-14T21:24:07.8885104Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8885458Z 			"size": 175,
2025-08-14T21:24:07.8885796Z 			"digest": "sha256:76846613d7067ca9de6a73db198902ca46b403bc78e2627f3f6fb805e1bbaa0a"
2025-08-14T21:24:07.8886249Z 		},
2025-08-14T21:24:07.8886450Z 		{
2025-08-14T21:24:07.8886800Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8887240Z 			"size": 1896,
2025-08-14T21:24:07.8887661Z 			"digest": "sha256:dcf1e01c98d6a6f72674d79a4e8e4047b54796576cd06ad682c225a92820a8f5"
2025-08-14T21:24:07.8888118Z 		},
2025-08-14T21:24:07.8888297Z 		{
2025-08-14T21:24:07.8888582Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8888942Z 			"size": 197509462,
2025-08-14T21:24:07.8889291Z 			"digest": "sha256:a88175fa027cfade948a7004100834f68828b13c0f21b314491c37084fbbe7e5"
2025-08-14T21:24:07.8889703Z 		},
2025-08-14T21:24:07.8889866Z 		{
2025-08-14T21:24:07.8890156Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8890510Z 			"size": 303,
2025-08-14T21:24:07.8890846Z 			"digest": "sha256:7f5277f691672469f431fd90a8c2bb702c6c68333f6be2cff868f00e416c5a1a"
2025-08-14T21:24:07.8891408Z 		},
2025-08-14T21:24:07.8891597Z 		{
2025-08-14T21:24:07.8891872Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8892215Z 			"size": 32,
2025-08-14T21:24:07.8892561Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-08-14T21:24:07.8892949Z 		},
2025-08-14T21:24:07.8893127Z 		{
2025-08-14T21:24:07.8893404Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8893750Z 			"size": 108,
2025-08-14T21:24:07.8894096Z 			"digest": "sha256:d0b078cebcc48c1f46a5d1286ae68a90c295cb6b81f033ba9a2f9c2498d6d7e5"
2025-08-14T21:24:07.8894500Z 		},
2025-08-14T21:24:07.8894679Z 		{
2025-08-14T21:24:07.8894958Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:24:07.8895306Z 			"size": 54145699,
2025-08-14T21:24:07.8895653Z 			"digest": "sha256:b01371e6ad0895119adb1d1dbfa20844d061be6aecadf9165128cebb28b9fd8e"
2025-08-14T21:24:07.8896051Z 		}
2025-08-14T21:24:07.8896233Z 	]
2025-08-14T21:24:07.8896417Z }
2025-08-14T21:24:07.8896624Z + exit 0
2025-08-14T21:24:07.8927440Z ##[group]Run set -eux
2025-08-14T21:24:07.8927751Z set -eux
2025-08-14T21:24:07.8928484Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-08-14T21:24:07.8949692Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:07.8950031Z env:
2025-08-14T21:24:07.8950223Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:07.8950610Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:07.8951420Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:07.8951943Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:07.8952800Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:07.8953563Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:07.8953817Z   AWS_REGION: us-east-1
2025-08-14T21:24:07.8954221Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:07.8954580Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:07.8959288Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:07.8959521Z ##[endgroup]
2025-08-14T21:24:07.9011578Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-08-14T21:24:07.9012408Z + jq --raw-output .SecretString
2025-08-14T21:24:07.9016603Z + jq -r .docker_hub_readonly_token
2025-08-14T21:24:07.9017309Z + docker login --username pytorchbot --password-stdin
2025-08-14T21:24:08.5691965Z 
2025-08-14T21:24:08.5694196Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2025-08-14T21:24:08.6531987Z Error: Cannot perform an interactive login from a non TTY device
2025-08-14T21:24:08.6584349Z ##[error]Process completed with exit code 1.
2025-08-14T21:24:08.6710416Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-08-14T21:24:08.6710826Z with:
2025-08-14T21:24:08.6711413Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:08.6712201Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:08.6712538Z env:
2025-08-14T21:24:08.6712742Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:08.6713116Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:08.6713649Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:08.6714157Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:08.6715055Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:08.6715815Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:08.6716071Z   AWS_REGION: us-east-1
2025-08-14T21:24:08.6716436Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:08.6716775Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:08.6721480Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:08.6721724Z ##[endgroup]
2025-08-14T21:24:08.6737325Z ##[group]Run set -x
2025-08-14T21:24:08.6737586Z set -x
2025-08-14T21:24:08.6737783Z set +e
2025-08-14T21:24:08.6737978Z 
2025-08-14T21:24:08.6738171Z login() {
2025-08-14T21:24:08.6738589Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-08-14T21:24:08.6739029Z }
2025-08-14T21:24:08.6739218Z 
2025-08-14T21:24:08.6739407Z retry () {
2025-08-14T21:24:08.6739646Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-08-14T21:24:08.6739915Z }
2025-08-14T21:24:08.6740089Z 
2025-08-14T21:24:08.6740305Z retry login "${DOCKER_REGISTRY}"
2025-08-14T21:24:08.6740575Z 
2025-08-14T21:24:08.6740984Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-08-14T21:24:08.6741561Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-08-14T21:24:08.6742161Z 
2025-08-14T21:24:08.6742363Z set -e
2025-08-14T21:24:08.6742666Z # ignore output since only exit code is used for conditional
2025-08-14T21:24:08.6743088Z # only pull docker image if it's not available locally
2025-08-14T21:24:08.6743547Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-08-14T21:24:08.6743977Z   retry docker pull "${DOCKER_IMAGE}"
2025-08-14T21:24:08.6744259Z fi
2025-08-14T21:24:08.6765801Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:08.6766142Z env:
2025-08-14T21:24:08.6766354Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:08.6766725Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:08.6767264Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:08.6767772Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:08.6768609Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:08.6769355Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:08.6769603Z   AWS_REGION: us-east-1
2025-08-14T21:24:08.6769897Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:08.6770241Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:08.6774930Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:08.6775781Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:08.6776516Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:08.6776854Z ##[endgroup]
2025-08-14T21:24:08.6821679Z + set +e
2025-08-14T21:24:08.6821985Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:08.6822380Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:08.6827684Z + aws ecr get-login-password --region us-east-1
2025-08-14T21:24:08.6830520Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:24:09.8347666Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-08-14T21:24:09.8348595Z Configure a credential helper to remove this warning. See
2025-08-14T21:24:09.8349446Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-08-14T21:24:09.8350014Z 
2025-08-14T21:24:09.8350177Z Login Succeeded
2025-08-14T21:24:09.8392989Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:09.8394313Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-08-14T21:24:10.7189418Z + IMAGE_SIZE=17743.216945648193
2025-08-14T21:24:10.7190087Z + echo 'Compressed size of image in MB: 17743.216945648193'
2025-08-14T21:24:10.7190573Z + set -e
2025-08-14T21:24:10.7190899Z Compressed size of image in MB: 17743.216945648193
2025-08-14T21:24:10.7191903Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:10.7477002Z Prepare all required actions
2025-08-14T21:24:10.7505910Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-08-14T21:24:10.7506220Z with:
2025-08-14T21:24:10.7506633Z   github-token: ***
2025-08-14T21:24:10.7506852Z env:
2025-08-14T21:24:10.7507052Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:10.7507430Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:10.7507984Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:10.7508506Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:10.7509635Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:10.7510395Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:10.7510646Z   AWS_REGION: us-east-1
2025-08-14T21:24:10.7510962Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:10.7511306Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:10.7516012Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:10.7516239Z ##[endgroup]
2025-08-14T21:24:10.7531101Z ##[group]Run set -eux
2025-08-14T21:24:10.7531348Z set -eux
2025-08-14T21:24:10.7531732Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-08-14T21:24:10.7553273Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:10.7553613Z env:
2025-08-14T21:24:10.7553821Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:10.7554202Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:10.7554784Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:10.7555288Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:10.7556126Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:10.7556879Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:10.7557130Z   AWS_REGION: us-east-1
2025-08-14T21:24:10.7557416Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:10.7557793Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:10.7562573Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:10.7562921Z   GITHUB_TOKEN: ***
2025-08-14T21:24:10.7563145Z ##[endgroup]
2025-08-14T21:24:10.7609556Z + python3 .github/scripts/get_workflow_job_id.py 16976370303 gpud515
2025-08-14T21:24:12.1301451Z Setting output job-id=48128182516
2025-08-14T21:24:12.1302199Z Setting output job-name=linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:24:12.1453316Z Prepare all required actions
2025-08-14T21:24:12.1453686Z Getting action download info
2025-08-14T21:24:12.3529397Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-08-14T21:24:13.1154113Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-08-14T21:24:13.8867864Z ##[group]Run ./.github/actions/download-build-artifacts
2025-08-14T21:24:13.8868183Z with:
2025-08-14T21:24:13.8868397Z   name: linux-jammy-rocm-py3.10
2025-08-14T21:24:13.8868655Z   s3-bucket: gha-artifacts
2025-08-14T21:24:13.8868870Z env:
2025-08-14T21:24:13.8869058Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:13.8869435Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:13.8869974Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:13.8870539Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:13.8871374Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:13.8872109Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:13.8872357Z   AWS_REGION: us-east-1
2025-08-14T21:24:13.8872656Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:13.8872988Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:13.8877688Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:13.8877915Z ##[endgroup]
2025-08-14T21:24:13.8906506Z ##[group]Run seemethere/download-artifact-s3@v4
2025-08-14T21:24:13.8906805Z with:
2025-08-14T21:24:13.8907012Z   name: linux-jammy-rocm-py3.10
2025-08-14T21:24:13.8907262Z   s3-bucket: gha-artifacts
2025-08-14T21:24:13.8907498Z   region: us-east-1
2025-08-14T21:24:13.8907695Z env:
2025-08-14T21:24:13.8908133Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:13.8908507Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:13.8909045Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:13.8909538Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:13.8910361Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:13.8911095Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:13.8911341Z   AWS_REGION: us-east-1
2025-08-14T21:24:13.8911617Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:13.8911952Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:13.8916641Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:13.8916872Z ##[endgroup]
2025-08-14T21:24:14.3772060Z (node:3703027) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-08-14T21:24:14.3772824Z 
2025-08-14T21:24:14.3773124Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-08-14T21:24:14.3773925Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-08-14T21:24:14.3774733Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-08-14T21:24:14.5446198Z Found 1 objects with prefix pytorch/pytorch/16976370303/linux-jammy-rocm-py3.10/
2025-08-14T21:24:14.5447253Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-08-14T21:24:37.1483619Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-08-14T21:24:37.1484228Z Artifact download has finished successfully
2025-08-14T21:24:37.1845254Z ##[group]Run unzip -o artifacts.zip
2025-08-14T21:24:37.1845568Z unzip -o artifacts.zip
2025-08-14T21:24:37.1866320Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:37.1866687Z env:
2025-08-14T21:24:37.1866889Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:37.1867554Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:37.1868136Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:37.1868653Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:37.1869487Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:37.1870235Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:37.1870489Z   AWS_REGION: us-east-1
2025-08-14T21:24:37.1870813Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:37.1871154Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:37.1875853Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:37.1876082Z ##[endgroup]
2025-08-14T21:24:37.1945338Z Archive:  artifacts.zip
2025-08-14T21:24:37.1946306Z    creating: dist/
2025-08-14T21:24:43.3892926Z   inflating: dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl  
2025-08-14T21:24:43.4044813Z   inflating: dist/.ninja_log         
2025-08-14T21:24:43.4045516Z    creating: build/custom_test_artifacts/
2025-08-14T21:24:43.4046137Z    creating: build/custom_test_artifacts/custom-op-build/
2025-08-14T21:24:43.4046858Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-08-14T21:24:43.4047815Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-08-14T21:24:43.4051888Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:24:43.4052554Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/
2025-08-14T21:24:43.4053121Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:24:43.4053723Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:24:43.4054934Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:24:43.4056114Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:24:43.4057920Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:24:43.4059096Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:24:43.4059810Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:24:43.4060427Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:24:43.4062383Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:24:43.4063735Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:24:43.4064757Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:24:43.4066457Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:24:43.4067851Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:24:43.4068535Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-08-14T21:24:43.4069060Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-08-14T21:24:43.4069602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:24:43.4070172Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-08-14T21:24:43.4070797Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-08-14T21:24:43.4071491Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-08-14T21:24:43.4072751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-08-14T21:24:43.4073434Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-08-14T21:24:43.4074097Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-08-14T21:24:43.4074751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-08-14T21:24:43.4075396Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-08-14T21:24:43.4076040Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-08-14T21:24:43.4076691Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-08-14T21:24:43.4099638Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-08-14T21:24:43.4312588Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-08-14T21:24:43.4313646Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-08-14T21:24:43.4314793Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-08-14T21:24:43.4316067Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-08-14T21:24:43.4317258Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-08-14T21:24:43.4318428Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-08-14T21:24:43.4319797Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-08-14T21:24:43.4321649Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-08-14T21:24:43.4322918Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-08-14T21:24:43.4323605Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-08-14T21:24:43.4324265Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-08-14T21:24:43.4342856Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-08-14T21:24:43.4431680Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-08-14T21:24:43.4432945Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:24:43.4434055Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:24:43.4435079Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-08-14T21:24:43.4436013Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-08-14T21:24:43.4436927Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:24:43.4437895Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:24:43.4438977Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:24:43.4440088Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:24:43.4441106Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-08-14T21:24:43.4442055Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-08-14T21:24:43.4442777Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-08-14T21:24:43.4618581Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-08-14T21:24:43.4679476Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-08-14T21:24:43.4680275Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-08-14T21:24:43.4680977Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-08-14T21:24:43.4681809Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-08-14T21:24:43.4685326Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:24:43.4686324Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/
2025-08-14T21:24:43.4687286Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:24:43.4688291Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:24:43.4689333Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:24:43.4690670Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:24:43.4692004Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:24:43.4693078Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:24:43.4693731Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:24:43.4694303Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:24:43.4695010Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:24:43.4696585Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:24:43.4697479Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:24:43.4699235Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:24:43.4700616Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:24:43.4701254Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-08-14T21:24:43.4701762Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-08-14T21:24:43.4702303Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:24:43.4702858Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-08-14T21:24:43.4703507Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-08-14T21:24:43.4704210Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-08-14T21:24:43.4704904Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-08-14T21:24:43.4705536Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-08-14T21:24:43.4706197Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-08-14T21:24:43.4706855Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-08-14T21:24:43.4707556Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-08-14T21:24:43.4708215Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-08-14T21:24:43.4708869Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-08-14T21:24:43.4731870Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-08-14T21:24:43.4802080Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-08-14T21:24:43.4803465Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:24:43.4804574Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:24:43.4805560Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-08-14T21:24:43.4806477Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-08-14T21:24:43.4807363Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:24:43.4808303Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:24:43.4809322Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:24:43.4810431Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:24:43.4811456Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-08-14T21:24:43.4812397Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-08-14T21:24:43.4813231Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-08-14T21:24:43.4852673Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-08-14T21:24:43.4853455Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-08-14T21:24:43.4854119Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-08-14T21:24:43.4854836Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-08-14T21:24:43.4858253Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:24:43.4859010Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/
2025-08-14T21:24:43.4859995Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:24:43.4860625Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:24:43.4861232Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:24:43.4862448Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:24:43.4864035Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:24:43.4864716Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:24:43.4865375Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:24:43.4866006Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:24:43.4868264Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:24:43.4869638Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:24:43.4870657Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:24:43.4872296Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:24:43.4873886Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:24:43.4874574Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-08-14T21:24:43.4875122Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-08-14T21:24:43.4875687Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:24:43.4876525Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-08-14T21:24:43.4877235Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-08-14T21:24:43.4878010Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-08-14T21:24:43.4878757Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-08-14T21:24:43.4879454Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-08-14T21:24:43.4880175Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-08-14T21:24:43.4880907Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-08-14T21:24:43.4881629Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-08-14T21:24:43.4882430Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-08-14T21:24:43.4883142Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-08-14T21:24:43.4886028Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-08-14T21:24:43.5018617Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-08-14T21:24:43.5019542Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-08-14T21:24:43.5020679Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-08-14T21:24:43.5022067Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-08-14T21:24:43.5023744Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-08-14T21:24:43.5025007Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-08-14T21:24:43.5026292Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-08-14T21:24:43.5027606Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-08-14T21:24:43.5028901Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-08-14T21:24:43.5030185Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-08-14T21:24:43.5031041Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-08-14T21:24:43.5048375Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-08-14T21:24:43.5108531Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-08-14T21:24:43.5109903Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:24:43.5111106Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:24:43.5112182Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-08-14T21:24:43.5113181Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-08-14T21:24:43.5114152Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:24:43.5115189Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:24:43.5116561Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:24:43.5117599Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:24:43.5118537Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-08-14T21:24:43.5119391Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-08-14T21:24:43.5120375Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-08-14T21:24:43.5227650Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-08-14T21:24:43.5270206Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-08-14T21:24:43.5270929Z    creating: build/lib/
2025-08-14T21:24:43.5360199Z   inflating: build/lib/libprotobuf-lite.a  
2025-08-14T21:24:43.5840161Z   inflating: build/lib/libprotobuf.a  
2025-08-14T21:24:43.6374744Z   inflating: build/lib/libprotoc.a   
2025-08-14T21:24:43.6385035Z   inflating: build/lib/libpthreadpool.a  
2025-08-14T21:24:43.6393648Z   inflating: build/lib/libcpuinfo.a  
2025-08-14T21:24:43.6402071Z   inflating: build/lib/libcpuinfo_internals.a  
2025-08-14T21:24:43.6403110Z   inflating: build/lib/libclog.a     
2025-08-14T21:24:43.6423662Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-08-14T21:24:43.6425927Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-08-14T21:24:43.6445191Z   inflating: build/lib/libnnpack.a   
2025-08-14T21:24:43.6645514Z   inflating: build/lib/libmicrokernels-prod.a  
2025-08-14T21:24:43.7585927Z   inflating: build/lib/libmicrokernels-all.a  
2025-08-14T21:24:43.7661692Z   inflating: build/lib/libgtest.a    
2025-08-14T21:24:43.7680036Z   inflating: build/lib/libgmock.a    
2025-08-14T21:24:43.7680735Z   inflating: build/lib/libgmock_main.a  
2025-08-14T21:24:43.7681472Z   inflating: build/lib/libgtest_main.a  
2025-08-14T21:24:43.7779037Z   inflating: build/lib/libXNNPACK.a  
2025-08-14T21:24:43.7860109Z   inflating: build/lib/libbenchmark.a  
2025-08-14T21:24:43.7860764Z   inflating: build/lib/libbenchmark_main.a  
2025-08-14T21:24:43.7869445Z   inflating: build/lib/libittnotify.a  
2025-08-14T21:24:43.7870193Z   inflating: build/lib/libjitprofiling.a  
2025-08-14T21:24:43.7940023Z   inflating: build/lib/libasmjit.a   
2025-08-14T21:24:43.9246721Z   inflating: build/lib/libfbgemm.a   
2025-08-14T21:24:43.9279198Z   inflating: build/lib/libtensorpipe_uv.a  
2025-08-14T21:24:43.9882558Z   inflating: build/lib/libtensorpipe.a  
2025-08-14T21:24:44.0012700Z   inflating: build/lib/libgloo.a     
2025-08-14T21:24:44.0064116Z   inflating: build/lib/libonnx_proto.a  
2025-08-14T21:24:44.0539771Z   inflating: build/lib/libgloo_hip.a  
2025-08-14T21:24:44.1321073Z   inflating: build/lib/libonnx.a     
2025-08-14T21:24:45.2548696Z   inflating: build/lib/libdnnl.a     
2025-08-14T21:24:45.2568257Z   inflating: build/lib/libfmt.a      
2025-08-14T21:24:45.2880341Z   inflating: build/lib/libkineto.a   
2025-08-14T21:24:45.2997267Z   inflating: build/lib/libc10.so     
2025-08-14T21:24:45.2998449Z   inflating: build/lib/libtorch_global_deps.so  
2025-08-14T21:24:45.3000072Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-08-14T21:24:45.3041227Z   inflating: build/lib/libc10_hip.so  
2025-08-14T21:24:48.5403839Z   inflating: build/lib/libtorch_cpu.so  
2025-08-14T21:24:48.5407892Z   inflating: build/lib/libshm.so     
2025-08-14T21:24:49.4852713Z   inflating: build/lib/libtorch_hip.so  
2025-08-14T21:24:49.4853984Z   inflating: build/lib/libtorch.so   
2025-08-14T21:24:49.4873934Z   inflating: build/lib/libjitbackend_test.so  
2025-08-14T21:24:49.4950135Z   inflating: build/lib/libtorchbind_test.so  
2025-08-14T21:24:49.4975972Z   inflating: build/lib/libbackend_with_compiler.so  
2025-08-14T21:24:49.5004760Z   inflating: build/lib/libaoti_custom_ops.so  
2025-08-14T21:24:49.7208292Z   inflating: build/lib/libtorch_python.so  
2025-08-14T21:24:49.7243913Z   inflating: build/lib/libnnapi_backend.so  
2025-08-14T21:24:49.7244406Z    creating: build/bin/
2025-08-14T21:24:49.7245476Z    creating: build/bin/CMakeFiles/
2025-08-14T21:24:49.7246011Z   inflating: build/bin/cmake_install.cmake  
2025-08-14T21:24:49.7246543Z   inflating: build/bin/CTestTestfile.cmake  
2025-08-14T21:24:49.7730390Z   inflating: build/bin/protoc-3.13.0.0  
2025-08-14T21:24:49.8214067Z   inflating: build/bin/protoc        
2025-08-14T21:24:49.8276321Z   inflating: build/bin/c10_AllocatorConfig_test  
2025-08-14T21:24:49.8335136Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-08-14T21:24:49.8395585Z   inflating: build/bin/c10_DeviceGuard_test  
2025-08-14T21:24:49.8456338Z   inflating: build/bin/c10_Device_test  
2025-08-14T21:24:49.8525917Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-08-14T21:24:49.8589070Z   inflating: build/bin/c10_Scalar_test  
2025-08-14T21:24:49.8647003Z   inflating: build/bin/c10_StreamGuard_test  
2025-08-14T21:24:49.8706993Z   inflating: build/bin/c10_SymInt_test  
2025-08-14T21:24:49.8771387Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-08-14T21:24:49.8836623Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-08-14T21:24:49.8902475Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-08-14T21:24:49.8983606Z   inflating: build/bin/c10_cow_test  
2025-08-14T21:24:49.9041343Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-08-14T21:24:49.9099798Z   inflating: build/bin/c10_ArrayRef_test  
2025-08-14T21:24:49.9162099Z   inflating: build/bin/c10_Bitset_test  
2025-08-14T21:24:49.9220609Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-08-14T21:24:49.9287541Z   inflating: build/bin/c10_Enumerate_test  
2025-08-14T21:24:49.9346927Z   inflating: build/bin/c10_Half_test  
2025-08-14T21:24:49.9408956Z   inflating: build/bin/c10_IntrusiveList_test  
2025-08-14T21:24:49.9474537Z   inflating: build/bin/c10_LeftRight_test  
2025-08-14T21:24:49.9539636Z   inflating: build/bin/c10_Metaprogramming_test  
2025-08-14T21:24:49.9602301Z   inflating: build/bin/c10_NetworkFlow_test  
2025-08-14T21:24:49.9660751Z   inflating: build/bin/c10_Semaphore_test  
2025-08-14T21:24:49.9726019Z   inflating: build/bin/c10_ThreadLocal_test  
2025-08-14T21:24:49.9784938Z   inflating: build/bin/c10_Synchronized_test  
2025-08-14T21:24:49.9846391Z   inflating: build/bin/c10_TypeIndex_test  
2025-08-14T21:24:49.9906361Z   inflating: build/bin/c10_TypeList_test  
2025-08-14T21:24:49.9964268Z   inflating: build/bin/c10_TypeTraits_test  
2025-08-14T21:24:50.0025063Z   inflating: build/bin/c10_accumulate_test  
2025-08-14T21:24:50.0090529Z   inflating: build/bin/c10_bfloat16_test  
2025-08-14T21:24:50.0157048Z   inflating: build/bin/c10_complex_math_test  
2025-08-14T21:24:50.0217298Z   inflating: build/bin/c10_bit_cast_test  
2025-08-14T21:24:50.0281996Z   inflating: build/bin/c10_complex_test  
2025-08-14T21:24:50.0340427Z   inflating: build/bin/c10_error_test  
2025-08-14T21:24:50.0399607Z   inflating: build/bin/c10_flags_test  
2025-08-14T21:24:50.0461329Z   inflating: build/bin/c10_exception_test  
2025-08-14T21:24:50.0521408Z   inflating: build/bin/c10_irange_test  
2025-08-14T21:24:50.0580549Z   inflating: build/bin/c10_generic_math_test  
2025-08-14T21:24:50.0768794Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-08-14T21:24:50.0831471Z   inflating: build/bin/c10_lazy_test  
2025-08-14T21:24:50.0898313Z   inflating: build/bin/c10_logging_test  
2025-08-14T21:24:50.0985283Z   inflating: build/bin/c10_optional_test  
2025-08-14T21:24:50.1047794Z   inflating: build/bin/c10_registry_test  
2025-08-14T21:24:50.1120013Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-08-14T21:24:50.1180517Z   inflating: build/bin/c10_ssize_test  
2025-08-14T21:24:50.1358109Z   inflating: build/bin/c10_small_vector_test  
2025-08-14T21:24:50.1424376Z   inflating: build/bin/c10_string_util_test  
2025-08-14T21:24:50.1483491Z   inflating: build/bin/c10_tempfile_test  
2025-08-14T21:24:50.1541984Z   inflating: build/bin/c10_string_view_test  
2025-08-14T21:24:50.1608007Z   inflating: build/bin/c10_typeid_test  
2025-08-14T21:24:50.1659291Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-08-14T21:24:50.1717044Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-08-14T21:24:50.1775337Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-08-14T21:24:50.1833323Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-08-14T21:24:50.1890988Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-08-14T21:24:50.1949561Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-08-14T21:24:50.2007126Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-08-14T21:24:50.2064539Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-08-14T21:24:50.2122578Z   inflating: build/bin/c10_hip_HIPTest  
2025-08-14T21:24:50.2781748Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-08-14T21:24:50.3460506Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-08-14T21:24:50.4145612Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-08-14T21:24:50.4207775Z   inflating: build/bin/BackoffTest   
2025-08-14T21:24:50.4270008Z   inflating: build/bin/FileStoreTest  
2025-08-14T21:24:50.4336527Z   inflating: build/bin/TCPStoreTest  
2025-08-14T21:24:50.4399898Z   inflating: build/bin/HashStoreTest  
2025-08-14T21:24:50.4476748Z   inflating: build/bin/ProcessGroupGlooTest  
2025-08-14T21:24:50.4479795Z   inflating: build/bin/example_allreduce  
2025-08-14T21:24:50.4484274Z   inflating: build/bin/torch_shm_manager  
2025-08-14T21:24:50.4546540Z   inflating: build/bin/static_runtime_bench  
2025-08-14T21:24:50.4829410Z   inflating: build/bin/static_runtime_test  
2025-08-14T21:24:50.4915237Z   inflating: build/bin/Dict_test     
2025-08-14T21:24:50.4976879Z   inflating: build/bin/Dimname_test  
2025-08-14T21:24:50.5053276Z   inflating: build/bin/MaybeOwned_test  
2025-08-14T21:24:50.5119984Z   inflating: build/bin/NamedTensor_test  
2025-08-14T21:24:50.5188463Z   inflating: build/bin/apply_utils_test  
2025-08-14T21:24:50.5257097Z   inflating: build/bin/atest         
2025-08-14T21:24:50.5331830Z   inflating: build/bin/basic         
2025-08-14T21:24:50.5397435Z   inflating: build/bin/broadcast_test  
2025-08-14T21:24:50.5457434Z   inflating: build/bin/cpu_allocator_test  
2025-08-14T21:24:50.5525559Z   inflating: build/bin/cpu_generator_test  
2025-08-14T21:24:50.5587767Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-08-14T21:24:50.5693939Z   inflating: build/bin/cpu_rng_test  
2025-08-14T21:24:50.5753558Z   inflating: build/bin/dlconvertor_test  
2025-08-14T21:24:50.5820462Z   inflating: build/bin/extension_backend_test  
2025-08-14T21:24:50.5885706Z   inflating: build/bin/half_test     
2025-08-14T21:24:50.5995324Z   inflating: build/bin/ivalue_test   
2025-08-14T21:24:50.6054198Z   inflating: build/bin/lazy_tensor_test  
2025-08-14T21:24:50.6116780Z   inflating: build/bin/math_kernel_test  
2025-08-14T21:24:50.6179633Z   inflating: build/bin/memory_format_test  
2025-08-14T21:24:50.6242683Z   inflating: build/bin/memory_overlapping_test  
2025-08-14T21:24:50.6304908Z   inflating: build/bin/mobile_memory_cleanup  
2025-08-14T21:24:50.6370634Z   inflating: build/bin/native_test   
2025-08-14T21:24:50.6430729Z   inflating: build/bin/operator_name_test  
2025-08-14T21:24:50.6490602Z   inflating: build/bin/operators_test  
2025-08-14T21:24:50.6551711Z   inflating: build/bin/packedtensoraccessor_test  
2025-08-14T21:24:50.6618910Z   inflating: build/bin/quantized_test  
2025-08-14T21:24:50.6697075Z   inflating: build/bin/pow_test      
2025-08-14T21:24:50.6755998Z   inflating: build/bin/reduce_ops_test  
2025-08-14T21:24:50.6816545Z   inflating: build/bin/reportMemoryUsage_test  
2025-08-14T21:24:50.6882700Z   inflating: build/bin/scalar_tensor_test  
2025-08-14T21:24:50.6943041Z   inflating: build/bin/StorageUtils_test  
2025-08-14T21:24:50.7011602Z   inflating: build/bin/scalar_test   
2025-08-14T21:24:50.7072837Z   inflating: build/bin/stride_properties_test  
2025-08-14T21:24:50.7164966Z   inflating: build/bin/tensor_iterator_test  
2025-08-14T21:24:50.7229266Z   inflating: build/bin/type_ptr_test  
2025-08-14T21:24:50.7293161Z   inflating: build/bin/test_parallel  
2025-08-14T21:24:50.7352316Z   inflating: build/bin/thread_init_test  
2025-08-14T21:24:50.7421716Z   inflating: build/bin/type_test     
2025-08-14T21:24:50.7483584Z   inflating: build/bin/undefined_tensor_test  
2025-08-14T21:24:50.7541990Z   inflating: build/bin/verify_api_visibility  
2025-08-14T21:24:50.7601903Z   inflating: build/bin/weakref_test  
2025-08-14T21:24:50.7682969Z   inflating: build/bin/legacy_vmap_test  
2025-08-14T21:24:50.7743523Z   inflating: build/bin/wrapdim_test  
2025-08-14T21:24:50.7804359Z   inflating: build/bin/xla_tensor_test  
2025-08-14T21:24:50.7873697Z   inflating: build/bin/IListRef_test  
2025-08-14T21:24:50.7996235Z   inflating: build/bin/List_test     
2025-08-14T21:24:50.8132387Z   inflating: build/bin/kernel_function_legacy_test  
2025-08-14T21:24:50.8242782Z   inflating: build/bin/kernel_function_test  
2025-08-14T21:24:50.8386331Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-08-14T21:24:50.8463327Z   inflating: build/bin/KernelFunction_test  
2025-08-14T21:24:50.8580137Z   inflating: build/bin/kernel_lambda_test  
2025-08-14T21:24:50.8651043Z   inflating: build/bin/kernel_stackbased_test  
2025-08-14T21:24:50.8759854Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-08-14T21:24:50.8820040Z   inflating: build/bin/CppSignature_test  
2025-08-14T21:24:50.8877666Z   inflating: build/bin/op_allowlist_test  
2025-08-14T21:24:50.8955321Z   inflating: build/bin/inline_container_test  
2025-08-14T21:24:50.9307217Z   inflating: build/bin/op_registration_test  
2025-08-14T21:24:50.9371972Z   inflating: build/bin/backend_fallback_test  
2025-08-14T21:24:50.9429672Z   inflating: build/bin/hip_complex_math_test  
2025-08-14T21:24:50.9492151Z   inflating: build/bin/hip_apply_test  
2025-08-14T21:24:50.9550401Z   inflating: build/bin/hip_complex_test  
2025-08-14T21:24:50.9608418Z   inflating: build/bin/hip_distributions_test  
2025-08-14T21:24:50.9666150Z   inflating: build/bin/hip_generator_test  
2025-08-14T21:24:50.9724689Z   inflating: build/bin/hip_half_test  
2025-08-14T21:24:50.9782476Z   inflating: build/bin/hip_integer_divider_test  
2025-08-14T21:24:50.9840096Z   inflating: build/bin/hip_optional_test  
2025-08-14T21:24:50.9897917Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-08-14T21:24:50.9955632Z   inflating: build/bin/hip_vectorized_test  
2025-08-14T21:24:51.0016371Z   inflating: build/bin/hip_dlconvertor_test  
2025-08-14T21:24:51.1241971Z   inflating: build/bin/test_jit      
2025-08-14T21:24:51.1592451Z   inflating: build/bin/test_nativert  
2025-08-14T21:24:51.1656841Z   inflating: build/bin/test_dist_autograd  
2025-08-14T21:24:51.1735375Z   inflating: build/bin/test_cpp_rpc  
2025-08-14T21:24:51.3003271Z   inflating: build/bin/test_api      
2025-08-14T21:24:51.3005572Z   inflating: build/bin/parallel_benchmark  
2025-08-14T21:24:51.3393472Z   inflating: build/bin/test_lazy     
2025-08-14T21:24:51.3394002Z    creating: .additional_ci_files/
2025-08-14T21:24:51.3479380Z   inflating: .additional_ci_files/test-times.json  
2025-08-14T21:24:51.3803036Z   inflating: .additional_ci_files/test-class-times.json  
2025-08-14T21:24:51.3845386Z ##[group]Run rm artifacts.zip
2025-08-14T21:24:51.3845694Z rm artifacts.zip
2025-08-14T21:24:51.3867219Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:51.3867563Z env:
2025-08-14T21:24:51.3867805Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:51.3868199Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:51.3868769Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:51.3869297Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:51.3870455Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:51.3871236Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:51.3871502Z   AWS_REGION: us-east-1
2025-08-14T21:24:51.3871862Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:51.3872298Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:51.3876992Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:51.3877230Z ##[endgroup]
2025-08-14T21:24:51.7461184Z ##[group]Run df -H
2025-08-14T21:24:51.7461438Z df -H
2025-08-14T21:24:51.7483623Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:51.7483967Z env:
2025-08-14T21:24:51.7484169Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:51.7484547Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:51.7485107Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:51.7485645Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:51.7486490Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:51.7487236Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:51.7487484Z   AWS_REGION: us-east-1
2025-08-14T21:24:51.7487793Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:51.7488119Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:51.7492792Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:51.7493016Z ##[endgroup]
2025-08-14T21:24:51.7567527Z Filesystem                                       Size  Used Avail Use% Mounted on
2025-08-14T21:24:51.7567945Z tmpfs                                            109G   33M  109G   1% /run
2025-08-14T21:24:51.7568325Z /dev/nvme0n1p2                                   944G   74G  823G   9% /
2025-08-14T21:24:51.7568973Z tmpfs                                            542G   33k  542G   1% /dev/shm
2025-08-14T21:24:51.7569339Z tmpfs                                            5.3M     0  5.3M   0% /run/lock
2025-08-14T21:24:51.7569717Z /dev/nvme0n1p1                                   536M  6.4M  530M   2% /boot/efi
2025-08-14T21:24:51.7570092Z /dev/nvme1n1p1                                   3.8T  530G  3.1T  15% /media/4TB
2025-08-14T21:24:51.7570479Z tmpfs                                            109G   33k  109G   1% /run/user/1307800118
2025-08-14T21:24:51.7570886Z 172.18.148.8:/export/amd2                        5.5T  278G  5.3T   6% /mnt
2025-08-14T21:24:51.7571375Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch  108G  348M  108G   1% /home/amd-pytorch
2025-08-14T21:24:51.7572158Z 172.18.148.15:/GroupStorage                      165T  137T   29T  83% /groups
2025-08-14T21:24:51.7572603Z 172.18.148.15:/GroupStorage/Scratch              5.5T  1.8T  3.8T  33% /scratch
2025-08-14T21:24:51.7601887Z Prepare all required actions
2025-08-14T21:24:51.7602411Z Getting action download info
2025-08-14T21:24:51.9498758Z ##[group]Run ./.github/actions/download-td-artifacts
2025-08-14T21:24:51.9499080Z with:
2025-08-14T21:24:51.9499254Z env:
2025-08-14T21:24:51.9499443Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:51.9499816Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:51.9500354Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:51.9500850Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:51.9501695Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:51.9502441Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:51.9502688Z   AWS_REGION: us-east-1
2025-08-14T21:24:51.9503008Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:51.9503363Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:51.9508047Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:51.9508279Z ##[endgroup]
2025-08-14T21:24:51.9535681Z ##[group]Run seemethere/download-artifact-s3@v4
2025-08-14T21:24:51.9535983Z with:
2025-08-14T21:24:51.9536173Z   name: td_results
2025-08-14T21:24:51.9536399Z   s3-bucket: gha-artifacts
2025-08-14T21:24:51.9536642Z   region: us-east-1
2025-08-14T21:24:51.9536842Z env:
2025-08-14T21:24:51.9537029Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:51.9537396Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:51.9537945Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:51.9538450Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:51.9539306Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:51.9540069Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:51.9540316Z   AWS_REGION: us-east-1
2025-08-14T21:24:51.9540586Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:51.9540914Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:51.9545621Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:51.9545846Z ##[endgroup]
2025-08-14T21:24:52.4404493Z (node:3703086) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-08-14T21:24:52.4405229Z 
2025-08-14T21:24:52.4405530Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-08-14T21:24:52.4406317Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-08-14T21:24:52.4407134Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-08-14T21:24:52.6158613Z Found 1 objects with prefix pytorch/pytorch/16976370303/td_results/
2025-08-14T21:24:52.6159646Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-08-14T21:24:52.9099047Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-08-14T21:24:52.9105005Z Artifact download has finished successfully
2025-08-14T21:24:52.9499346Z ##[group]Run mkdir -p .additional_ci_files
2025-08-14T21:24:52.9499689Z mkdir -p .additional_ci_files
2025-08-14T21:24:52.9500061Z mv td_results.json .additional_ci_files/td_results.json || true
2025-08-14T21:24:52.9520576Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:52.9520919Z env:
2025-08-14T21:24:52.9521119Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:52.9521504Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:52.9522061Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:52.9522651Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:52.9523779Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:52.9524569Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:52.9524823Z   AWS_REGION: us-east-1
2025-08-14T21:24:52.9525122Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:52.9525454Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:52.9530148Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:52.9530372Z ##[endgroup]
2025-08-14T21:24:52.9646470Z ##[group]Run .github/scripts/parse_ref.py
2025-08-14T21:24:52.9646833Z .github/scripts/parse_ref.py
2025-08-14T21:24:52.9665929Z shell: /usr/bin/bash -e {0}
2025-08-14T21:24:52.9666180Z env:
2025-08-14T21:24:52.9666377Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:52.9666753Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:52.9667300Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:52.9667850Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:52.9668700Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:52.9669456Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:52.9669704Z   AWS_REGION: us-east-1
2025-08-14T21:24:52.9669986Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:52.9670332Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:52.9675017Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:52.9675237Z ##[endgroup]
2025-08-14T21:24:52.9908542Z Setting output branch=main
2025-08-14T21:24:53.0025507Z Prepare all required actions
2025-08-14T21:24:53.0025860Z Getting action download info
2025-08-14T21:24:53.1668234Z ##[group]Run ./.github/actions/filter-test-configs
2025-08-14T21:24:53.1668526Z with:
2025-08-14T21:24:53.1668887Z   github-token: ***
2025-08-14T21:24:53.1670085Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:24:53.1671539Z   job-name: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:24:53.1672010Z env:
2025-08-14T21:24:53.1672202Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:53.1672604Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:53.1673143Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:53.1673646Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:53.1674487Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:53.1675471Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:53.1675723Z   AWS_REGION: us-east-1
2025-08-14T21:24:53.1675995Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:53.1676362Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:53.1681109Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:53.1681358Z ##[endgroup]
2025-08-14T21:24:53.1712475Z ##[group]Run nick-fields/retry@v3.0.0
2025-08-14T21:24:53.1712746Z with:
2025-08-14T21:24:53.1712920Z   shell: bash
2025-08-14T21:24:53.1713135Z   timeout_minutes: 10
2025-08-14T21:24:53.1713359Z   max_attempts: 5
2025-08-14T21:24:53.1713570Z   retry_wait_seconds: 30
2025-08-14T21:24:53.1714237Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.2

2025-08-14T21:24:53.1714929Z   polling_interval_seconds: 1
2025-08-14T21:24:53.1715175Z   warning_on_retry: true
2025-08-14T21:24:53.1715398Z   continue_on_error: false
2025-08-14T21:24:53.1715615Z env:
2025-08-14T21:24:53.1715799Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:53.1716154Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:53.1716715Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:53.1717221Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:53.1718065Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:53.1718812Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:53.1719057Z   AWS_REGION: us-east-1
2025-08-14T21:24:53.1719330Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:53.1719658Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:53.1724475Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:53.1724843Z   GITHUB_TOKEN: ***
2025-08-14T21:24:53.1725053Z ##[endgroup]
2025-08-14T21:24:53.2513708Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2
2025-08-14T21:24:53.5475551Z Defaulting to user installation because normal site-packages is not writeable
2025-08-14T21:24:53.6574443Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2025-08-14T21:24:53.6579498Z Requirement already satisfied: pyyaml==6.0.2 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.2)
2025-08-14T21:24:53.6681477Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2025-08-14T21:24:53.6691021Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2025-08-14T21:24:53.6701684Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2025-08-14T21:24:53.6706585Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2025-08-14T21:24:54.2515015Z Command completed after 1 attempt(s).
2025-08-14T21:24:54.2589411Z ##[group]Run set -x
2025-08-14T21:24:54.2589650Z set -x
2025-08-14T21:24:54.2589847Z 
2025-08-14T21:24:54.2590192Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-08-14T21:24:54.2590619Z # in runner workspace
2025-08-14T21:24:54.2605344Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-08-14T21:24:54.2626889Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:54.2627220Z env:
2025-08-14T21:24:54.2627410Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:54.2627774Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:54.2628623Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:54.2629146Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:54.2629981Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:54.2630725Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:54.2630967Z   AWS_REGION: us-east-1
2025-08-14T21:24:54.2631261Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:54.2631586Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:54.2636262Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:54.2636485Z ##[endgroup]
2025-08-14T21:24:54.2683079Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-08-14T21:24:54.2859710Z Setting output branch=main
2025-08-14T21:24:54.2921347Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-08-14T21:24:54.2921727Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-08-14T21:24:54.2922015Z echo "Job name: ${JOB_NAME}"
2025-08-14T21:24:54.2922417Z 
2025-08-14T21:24:54.2922748Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-08-14T21:24:54.2923174Z # in runner workspace
2025-08-14T21:24:54.2923547Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-08-14T21:24:54.2923961Z   --workflow "${GITHUB_WORKFLOW}" \
2025-08-14T21:24:54.2924261Z   --job-name "${JOB_NAME}" \
2025-08-14T21:24:54.2925563Z   --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}" \
2025-08-14T21:24:54.2926841Z   --selected-test-configs "" \
2025-08-14T21:24:54.2927132Z   --pr-number "${PR_NUMBER}" \
2025-08-14T21:24:54.2927401Z   --tag "${TAG}" \
2025-08-14T21:24:54.2927653Z   --event-name "${EVENT_NAME}" \
2025-08-14T21:24:54.2927927Z   --schedule "${SCHEDULE}" \
2025-08-14T21:24:54.2928199Z   --branch "${HEAD_BRANCH}"
2025-08-14T21:24:54.2948671Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:54.2949011Z env:
2025-08-14T21:24:54.2949211Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:54.2949797Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:54.2950369Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:54.2950872Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:54.2951713Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:54.2952456Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:54.2952706Z   AWS_REGION: us-east-1
2025-08-14T21:24:54.2952996Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:54.2953328Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:54.2958021Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:54.2958382Z   GITHUB_TOKEN: ***
2025-08-14T21:24:54.2958833Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:24:54.2959310Z   PR_NUMBER: 
2025-08-14T21:24:54.2959504Z   TAG: 
2025-08-14T21:24:54.2959687Z   EVENT_NAME: schedule
2025-08-14T21:24:54.2959909Z   SCHEDULE: 45 4,12,20 * * 1-5
2025-08-14T21:24:54.2960142Z   HEAD_BRANCH: main
2025-08-14T21:24:54.2960341Z ##[endgroup]
2025-08-14T21:24:54.3005092Z Workflow: periodic
2025-08-14T21:24:54.3005910Z Job name: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:24:54.7179457Z Setting output keep-going=True
2025-08-14T21:24:54.7179994Z Setting output ci-verbose-test-logs=False
2025-08-14T21:24:54.7180528Z Setting output ci-test-showlocals=False
2025-08-14T21:24:54.7181047Z Setting output ci-no-test-timeout=False
2025-08-14T21:24:54.7181528Z Setting output ci-no-td=False
2025-08-14T21:24:54.7182004Z Setting output ci-td-distributed=False
2025-08-14T21:24:54.7182496Z Setting output is-unstable=False
2025-08-14T21:24:54.7182946Z Setting output reenabled-issues=
2025-08-14T21:24:54.7185182Z Setting output test-matrix={"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:24:54.7187422Z Setting output is-test-matrix-empty=False
2025-08-14T21:24:54.7335484Z ##[group]Run echo "Filtered matrix:"
2025-08-14T21:24:54.7335791Z echo "Filtered matrix:"
2025-08-14T21:24:54.7337042Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}"
2025-08-14T21:24:54.7338271Z 
2025-08-14T21:24:54.7338459Z echo
2025-08-14T21:24:54.7338705Z echo "Is the current job unstable? False"
2025-08-14T21:24:54.7338985Z 
2025-08-14T21:24:54.7339186Z echo
2025-08-14T21:24:54.7339401Z echo "Is keep-going label set? True"
2025-08-14T21:24:54.7339675Z 
2025-08-14T21:24:54.7339858Z echo
2025-08-14T21:24:54.7340061Z echo "Reenabled issues? "
2025-08-14T21:24:54.7360401Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:54.7360748Z env:
2025-08-14T21:24:54.7360945Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:54.7361310Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:54.7361847Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:54.7362665Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:54.7363521Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:54.7364269Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:54.7364525Z   AWS_REGION: us-east-1
2025-08-14T21:24:54.7364821Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:54.7365157Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:54.7369839Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:54.7370066Z ##[endgroup]
2025-08-14T21:24:54.7412976Z Filtered matrix:
2025-08-14T21:24:54.7414396Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}]}
2025-08-14T21:24:54.7415686Z 
2025-08-14T21:24:54.7415832Z Is the current job unstable? False
2025-08-14T21:24:54.7416026Z 
2025-08-14T21:24:54.7416135Z Is keep-going label set? True
2025-08-14T21:24:54.7416309Z 
2025-08-14T21:24:54.7416404Z Reenabled issues? 
2025-08-14T21:24:54.7453699Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:54.7454414Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-08-14T21:24:54.7473089Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:24:54.7473423Z env:
2025-08-14T21:24:54.7473622Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:54.7473990Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:54.7474525Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:54.7475032Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:54.7475875Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:54.7476633Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:54.7476925Z   AWS_REGION: us-east-1
2025-08-14T21:24:54.7477200Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:54.7477533Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:54.7482333Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:54.7482559Z   JOB_TIMEOUT: 300
2025-08-14T21:24:54.7482760Z ##[endgroup]
2025-08-14T21:24:54.7568144Z ##[group]Run set -x
2025-08-14T21:24:54.7568441Z set -x
2025-08-14T21:24:54.7568643Z 
2025-08-14T21:24:54.7568870Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-08-14T21:24:54.7569212Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-08-14T21:24:54.7569560Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-08-14T21:24:54.7569875Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-08-14T21:24:54.7570141Z else
2025-08-14T21:24:54.7570373Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-08-14T21:24:54.7570646Z fi
2025-08-14T21:24:54.7570819Z 
2025-08-14T21:24:54.7571101Z # detached container should get cleaned up by teardown_ec2_linux
2025-08-14T21:24:54.7571569Z # TODO: Stop building test binaries as part of the build phase
2025-08-14T21:24:54.7571967Z # Used for GPU_FLAG since that doesn't play nice
2025-08-14T21:24:54.7572323Z # shellcheck disable=SC2086,SC2090
2025-08-14T21:24:54.7572611Z container_name=$(docker run \
2025-08-14T21:24:54.7572880Z   ${GPU_FLAG:-} \
2025-08-14T21:24:54.7573117Z   -e BUILD_ENVIRONMENT \
2025-08-14T21:24:54.7573386Z   -e PR_NUMBER \
2025-08-14T21:24:54.7573621Z   -e GITHUB_ACTIONS \
2025-08-14T21:24:54.7573870Z   -e GITHUB_REPOSITORY \
2025-08-14T21:24:54.7574127Z   -e GITHUB_WORKFLOW \
2025-08-14T21:24:54.7574368Z   -e GITHUB_JOB \
2025-08-14T21:24:54.7574600Z   -e GITHUB_RUN_ID \
2025-08-14T21:24:54.7574840Z   -e GITHUB_RUN_NUMBER \
2025-08-14T21:24:54.7575089Z   -e GITHUB_RUN_ATTEMPT \
2025-08-14T21:24:54.7575334Z   -e JOB_ID \
2025-08-14T21:24:54.7575553Z   -e JOB_NAME \
2025-08-14T21:24:54.7575770Z   -e BRANCH \
2025-08-14T21:24:54.7575975Z   -e SHA1 \
2025-08-14T21:24:54.7576193Z   -e AWS_DEFAULT_REGION \
2025-08-14T21:24:54.7576446Z   -e IN_WHEEL_TEST \
2025-08-14T21:24:54.7576682Z   -e SHARD_NUMBER \
2025-08-14T21:24:54.7576914Z   -e TEST_CONFIG \
2025-08-14T21:24:54.7577152Z   -e NUM_TEST_SHARDS \
2025-08-14T21:24:54.7577404Z   -e REENABLED_ISSUES \
2025-08-14T21:24:54.7577658Z   -e CONTINUE_THROUGH_ERROR \
2025-08-14T21:24:54.7577926Z   -e VERBOSE_TEST_LOGS \
2025-08-14T21:24:54.7578175Z   -e TEST_SHOWLOCALS \
2025-08-14T21:24:54.7578419Z   -e NO_TEST_TIMEOUT \
2025-08-14T21:24:54.7578655Z   -e NO_TD \
2025-08-14T21:24:54.7578897Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-08-14T21:24:54.7579220Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-08-14T21:24:54.7579538Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-08-14T21:24:54.7580054Z   -e TESTS_TO_INCLUDE \
2025-08-14T21:24:54.7580304Z   -e DASHBOARD_TAG \
2025-08-14T21:24:54.7580615Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-08-14T21:24:54.7580961Z   --ulimit stack=10485760:83886080 \
2025-08-14T21:24:54.7581234Z   --ulimit core=0 \
2025-08-14T21:24:54.7581500Z   --security-opt seccomp=unconfined \
2025-08-14T21:24:54.7581788Z   --cap-add=SYS_PTRACE \
2025-08-14T21:24:54.7582036Z   --shm-size="8g" \
2025-08-14T21:24:54.7582258Z   --tty \
2025-08-14T21:24:54.7582463Z   --detach \
2025-08-14T21:24:54.7582695Z   --name="${container_name}" \
2025-08-14T21:24:54.7582970Z   --user jenkins \
2025-08-14T21:24:54.7583266Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-08-14T21:24:54.7583599Z   -w /var/lib/jenkins/workspace \
2025-08-14T21:24:54.7583872Z   "${DOCKER_IMAGE}"
2025-08-14T21:24:54.7584092Z )
2025-08-14T21:24:54.7584304Z # save container name for later step
2025-08-14T21:24:54.7584853Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-08-14T21:24:54.7585479Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-08-14T21:24:54.7586271Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-08-14T21:24:54.7605300Z shell: /usr/bin/bash -e {0}
2025-08-14T21:24:54.7605543Z env:
2025-08-14T21:24:54.7605737Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:24:54.7606108Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:24:54.7606650Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:24:54.7607157Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:24:54.7607990Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:24:54.7608741Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:24:54.7608989Z   AWS_REGION: us-east-1
2025-08-14T21:24:54.7609264Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:24:54.7609594Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:24:54.7614297Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:24:54.7614564Z   BUILD_ENVIRONMENT: linux-jammy-rocm-py3.10
2025-08-14T21:24:54.7614846Z   PR_NUMBER: 
2025-08-14T21:24:54.7615058Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-08-14T21:24:54.7615329Z   GITHUB_WORKFLOW: periodic
2025-08-14T21:24:54.7615560Z   GITHUB_JOB: test
2025-08-14T21:24:54.7615765Z   GITHUB_RUN_ID: 16976370303
2025-08-14T21:24:54.7615995Z   GITHUB_RUN_NUMBER: 17921
2025-08-14T21:24:54.7616222Z   GITHUB_RUN_ATTEMPT: 1
2025-08-14T21:24:54.7616442Z   JOB_ID: 48128182516
2025-08-14T21:24:54.7616879Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:24:54.7617351Z   BRANCH: main
2025-08-14T21:24:54.7617576Z   SHA1: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:24:54.7617866Z   CONTINUE_THROUGH_ERROR: True
2025-08-14T21:24:54.7618108Z   VERBOSE_TEST_LOGS: False
2025-08-14T21:24:54.7618337Z   TEST_SHOWLOCALS: False
2025-08-14T21:24:54.7618563Z   NO_TEST_TIMEOUT: False
2025-08-14T21:24:54.7618766Z   NO_TD: False
2025-08-14T21:24:54.7618972Z   TEST_CONFIG: distributed
2025-08-14T21:24:54.7619194Z   SHARD_NUMBER: 1
2025-08-14T21:24:54.7619395Z   NUM_TEST_SHARDS: 3
2025-08-14T21:24:54.7619604Z   REENABLED_ISSUES: 
2025-08-14T21:24:54.7620208Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:54.7620865Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-08-14T21:24:54.7621339Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-08-14T21:24:54.7621599Z   TESTS_TO_INCLUDE: 
2025-08-14T21:24:54.7621808Z   DASHBOARD_TAG: 
2025-08-14T21:24:54.7622003Z ##[endgroup]
2025-08-14T21:24:54.7661835Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2025-08-14T21:24:54.7662205Z + [[ linux-jammy-rocm-py3.10 == *onnx* ]]
2025-08-14T21:24:54.7662551Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-08-14T21:24:54.7673828Z +++ nproc --ignore=2
2025-08-14T21:24:54.7695493Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_16976370303 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:24:54.8996612Z + container_name=2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-14T21:24:54.8997660Z + echo CONTAINER_NAME=2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-14T21:24:54.8999193Z + docker exec -t 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-08-14T21:25:12.2470507Z Processing ./dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl
2025-08-14T21:25:13.1436179Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.18.0)
2025-08-14T21:25:13.1440535Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (4.14.1)
2025-08-14T21:25:13.1444619Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (1.13.3)
2025-08-14T21:25:13.1448405Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2.8.8)
2025-08-14T21:25:13.1451295Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.1.6)
2025-08-14T21:25:13.1455774Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2025.5.1)
2025-08-14T21:25:13.1814549Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git1fc683c) (1.3.0)
2025-08-14T21:25:13.1866910Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.9.0a0+git1fc683c) (3.0.2)
2025-08-14T21:25:13.6092023Z Installing collected packages: torch
2025-08-14T21:25:26.3303879Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-08-14T21:25:26.3305424Z helion 0.1.0 requires filecheck, which is not installed.
2025-08-14T21:25:26.3306184Z Successfully installed torch-2.9.0a0+git1fc683c
2025-08-14T21:25:26.3825018Z + export TERM=vt100
2025-08-14T21:25:26.3825395Z + TERM=vt100
2025-08-14T21:25:26.3832571Z ++ dirname .ci/pytorch/test.sh
2025-08-14T21:25:26.3847723Z + source .ci/pytorch/common.sh
2025-08-14T21:25:26.3856660Z +++ dirname .ci/pytorch/common.sh
2025-08-14T21:25:26.3874178Z ++ source .ci/pytorch/common_utils.sh
2025-08-14T21:25:26.3876913Z +++ declare -f -t trap_add
2025-08-14T21:25:26.3881431Z ++ set -ex -o pipefail
2025-08-14T21:25:26.3881724Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:25:26.3882067Z ++ unset HIP_PLATFORM
2025-08-14T21:25:26.3882526Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:25:26.3882823Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:25:26.3883091Z ++ BUILD_TEST_LIBTORCH=0
2025-08-14T21:25:26.3890158Z ++ dirname .ci/pytorch/test.sh
2025-08-14T21:25:26.3905866Z + source .ci/pytorch/common-build.sh
2025-08-14T21:25:26.3908178Z ++ [[ linux-jammy-rocm-py3.10 != *win-* ]]
2025-08-14T21:25:26.3921540Z ++++ dirname .ci/pytorch/common-build.sh
2025-08-14T21:25:26.3937577Z +++ cd .ci/pytorch
2025-08-14T21:25:26.3938163Z +++ pwd -P
2025-08-14T21:25:26.3941886Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch
2025-08-14T21:25:26.3942589Z ++ [[ linux-jammy-rocm-py3.10 == *-pch* ]]
2025-08-14T21:25:26.3942898Z ++ which sccache
2025-08-14T21:25:26.3964058Z ++ [[ -z '' ]]
2025-08-14T21:25:26.3964426Z ++ unset SCCACHE_BUCKET
2025-08-14T21:25:26.3964809Z ++ unset SCCACHE_REGION
2025-08-14T21:25:26.3965196Z ++ sccache --stop-server
2025-08-14T21:25:26.4004237Z ++ true
2025-08-14T21:25:26.4004611Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-08-14T21:25:26.4023543Z ++ trap_add sccache_epilogue EXIT
2025-08-14T21:25:26.4023999Z ++ trap_add_cmd=sccache_epilogue
2025-08-14T21:25:26.4024404Z ++ shift
2025-08-14T21:25:26.4024721Z ++ for trap_add_name in "$@"
2025-08-14T21:25:26.4037513Z ++++ trap -p EXIT
2025-08-14T21:25:26.4043169Z +++ eval 'extract_trap_cmd '
2025-08-14T21:25:26.4043418Z ++++ extract_trap_cmd
2025-08-14T21:25:26.4044318Z ++++ printf '%s\n' ''
2025-08-14T21:25:26.4044556Z +++ printf '%s\n' sccache_epilogue
2025-08-14T21:25:26.4047542Z ++ trap -- '
2025-08-14T21:25:26.4047745Z sccache_epilogue' EXIT
2025-08-14T21:25:26.4047968Z ++ [[ -n '' ]]
2025-08-14T21:25:26.4048196Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:25:26.4048535Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-08-14T21:25:26.4048854Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-08-14T21:25:26.4049095Z ++ sccache --start-server
2025-08-14T21:25:26.4078719Z sccache: Starting the server...
2025-08-14T21:25:26.4448403Z sccache: Listening on address 127.0.0.1:4226
2025-08-14T21:25:26.4464381Z ++ sccache --zero-stats
2025-08-14T21:25:26.4493731Z Statistics zeroed.
2025-08-14T21:25:26.4498854Z ++ which ccache
2025-08-14T21:25:26.4513505Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-08-14T21:25:26.4513823Z + echo 'Environment variables:'
2025-08-14T21:25:26.4514095Z Environment variables:
2025-08-14T21:25:26.4514318Z + env
2025-08-14T21:25:26.4527487Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:25:26.4527962Z CONTINUE_THROUGH_ERROR=True
2025-08-14T21:25:26.4528250Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-08-14T21:25:26.4528593Z HOSTNAME=gpud515.jax.cs.cpe.ice.amd.com
2025-08-14T21:25:26.4529240Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4529868Z GITHUB_ACTION=__self
2025-08-14T21:25:26.4530119Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-08-14T21:25:26.4530403Z GITHUB_RUN_NUMBER=17921
2025-08-14T21:25:26.4530649Z TEST_CONFIG=distributed
2025-08-14T21:25:26.4530909Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-08-14T21:25:26.4531198Z AWS_DEFAULT_REGION=us-east-1
2025-08-14T21:25:26.4531463Z GITHUB_TRIGGERING_ACTOR=huydhn
2025-08-14T21:25:26.4531729Z GITHUB_REF_TYPE=branch
2025-08-14T21:25:26.4532165Z ***
2025-08-14T21:25:26.4532398Z GITHUB_REPOSITORY_ID=65600975
2025-08-14T21:25:26.4532691Z GITHUB_ACTIONS=true
2025-08-14T21:25:26.4533215Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4533591Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4534127Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main
2025-08-14T21:25:26.4534632Z UCC_HOME=/usr
2025-08-14T21:25:26.4534863Z VERBOSE_TEST_LOGS=False
2025-08-14T21:25:26.4535122Z GITHUB_REF=refs/heads/main
2025-08-14T21:25:26.4535391Z SHARD_NUMBER=1
2025-08-14T21:25:26.4535628Z GITHUB_REF_PROTECTED=true
2025-08-14T21:25:26.4535891Z HOME=/var/lib/jenkins
2025-08-14T21:25:26.4536180Z GITHUB_API_URL=https://api.github.com
2025-08-14T21:25:26.4536536Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-08-14T21:25:26.4536839Z LANG=C.UTF-8
2025-08-14T21:25:26.4537123Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-08-14T21:25:26.4537489Z PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:25:26.4537747Z NUM_TEST_SHARDS=3
2025-08-14T21:25:26.4537982Z UCX_HOME=/usr
2025-08-14T21:25:26.4538633Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4539608Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:25:26.4540377Z MAGMA_HOME=/opt/rocm/magma
2025-08-14T21:25:26.4541015Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4541907Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-08-14T21:25:26.4542465Z GITHUB_EVENT_NAME=schedule
2025-08-14T21:25:26.4542737Z DASHBOARD_TAG=
2025-08-14T21:25:26.4542975Z GITHUB_RUN_ID=16976370303
2025-08-14T21:25:26.4543562Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4544150Z GITHUB_ACTOR=huydhn
2025-08-14T21:25:26.4544356Z PR_NUMBER=
2025-08-14T21:25:26.4544546Z GITHUB_RUN_ATTEMPT=1
2025-08-14T21:25:26.4544783Z ANACONDA_PYTHON_VERSION=3.10
2025-08-14T21:25:26.4545082Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-08-14T21:25:26.4545386Z TERM=vt100
2025-08-14T21:25:26.4545575Z INSTALLED_VISION=yes
2025-08-14T21:25:26.4545786Z BRANCH=main
2025-08-14T21:25:26.4545989Z OPENSSL_ROOT_DIR=/opt/openssl
2025-08-14T21:25:26.4546225Z TESTS_TO_INCLUDE=
2025-08-14T21:25:26.4546662Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-08-14T21:25:26.4547156Z GITHUB_SERVER_URL=https://github.com
2025-08-14T21:25:26.4547433Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-08-14T21:25:26.4547714Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-08-14T21:25:26.4548000Z REENABLED_ISSUES=
2025-08-14T21:25:26.4548200Z SHLVL=1
2025-08-14T21:25:26.4548386Z MAX_JOBS=126
2025-08-14T21:25:26.4548588Z GITHUB_ACTOR_ID=475357
2025-08-14T21:25:26.4548870Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4549192Z GITHUB_REF_NAME=main
2025-08-14T21:25:26.4549400Z ROCM_PATH=/opt/rocm
2025-08-14T21:25:26.4549600Z GITHUB_JOB=test
2025-08-14T21:25:26.4549794Z NO_TEST_TIMEOUT=False
2025-08-14T21:25:26.4550023Z GITHUB_REPOSITORY=pytorch/pytorch
2025-08-14T21:25:26.4550270Z LC_ALL=C.UTF-8
2025-08-14T21:25:26.4550472Z GITHUB_RETENTION_DAYS=90
2025-08-14T21:25:26.4550699Z OPENSSL_DIR=/opt/openssl
2025-08-14T21:25:26.4550920Z GITHUB_ACTION_REPOSITORY=
2025-08-14T21:25:26.4551727Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:25:26.4552532Z GITHUB_BASE_REF=
2025-08-14T21:25:26.4552721Z CI=true
2025-08-14T21:25:26.4552916Z GITHUB_REPOSITORY_OWNER=pytorch
2025-08-14T21:25:26.4553150Z JOB_ID=48128182516
2025-08-14T21:25:26.4553342Z GITHUB_HEAD_REF=
2025-08-14T21:25:26.4553537Z GITHUB_ACTION_REF=
2025-08-14T21:25:26.4553737Z TEST_SHOWLOCALS=False
2025-08-14T21:25:26.4554119Z GITHUB_WORKFLOW=periodic
2025-08-14T21:25:26.4554357Z DEBIAN_FRONTEND=noninteractive
2025-08-14T21:25:26.4554921Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4555482Z NO_TD=False
2025-08-14T21:25:26.4555672Z OLDPWD=/var/lib/jenkins
2025-08-14T21:25:26.4555886Z _=/usr/bin/env
2025-08-14T21:25:26.4556154Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-08-14T21:25:26.4670835Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-08-14T21:25:26.4671741Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-14T21:25:26.4672568Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-08-14T21:25:26.4673412Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-08-14T21:25:26.4674073Z + BUILD_DIR=build
2025-08-14T21:25:26.4674442Z + BUILD_RENAMED_DIR=build_renamed
2025-08-14T21:25:26.4674876Z + BUILD_BIN_DIR=build/bin
2025-08-14T21:25:26.4675249Z + SHARD_NUMBER=1
2025-08-14T21:25:26.4675584Z + NUM_TEST_SHARDS=3
2025-08-14T21:25:26.4675961Z + export TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:25:26.4676735Z + TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:25:26.4677160Z + export VALGRIND=ON
2025-08-14T21:25:26.4677511Z + VALGRIND=ON
2025-08-14T21:25:26.4677913Z + [[ linux-jammy-rocm-py3.10 == *clang9* ]]
2025-08-14T21:25:26.4678422Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-08-14T21:25:26.4678903Z + [[ linux-jammy-rocm-py3.10 == *s390x* ]]
2025-08-14T21:25:26.4679326Z + [[ 0 == \1 ]]
2025-08-14T21:25:26.4679645Z + [[ True == \1 ]]
2025-08-14T21:25:26.4680011Z + [[ linux-jammy-rocm-py3.10 != *bazel* ]]
2025-08-14T21:25:26.4693646Z ++ realpath build/custom_test_artifacts
2025-08-14T21:25:26.4696691Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-08-14T21:25:26.4697138Z + [[ -n '' ]]
2025-08-14T21:25:26.4697361Z + echo 'Environment variables'
2025-08-14T21:25:26.4697635Z Environment variables
2025-08-14T21:25:26.4697851Z + env
2025-08-14T21:25:26.4709255Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:25:26.4709680Z CONTINUE_THROUGH_ERROR=True
2025-08-14T21:25:26.4709983Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-08-14T21:25:26.4710294Z HOSTNAME=gpud515.jax.cs.cpe.ice.amd.com
2025-08-14T21:25:26.4710874Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4711432Z GITHUB_ACTION=__self
2025-08-14T21:25:26.4711673Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-08-14T21:25:26.4711935Z GITHUB_RUN_NUMBER=17921
2025-08-14T21:25:26.4712159Z TEST_CONFIG=distributed
2025-08-14T21:25:26.4712393Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-08-14T21:25:26.4712665Z AWS_DEFAULT_REGION=us-east-1
2025-08-14T21:25:26.4712910Z GITHUB_TRIGGERING_ACTOR=huydhn
2025-08-14T21:25:26.4713148Z GITHUB_REF_TYPE=branch
2025-08-14T21:25:26.4713417Z ***
2025-08-14T21:25:26.4713607Z GITHUB_REPOSITORY_ID=65600975
2025-08-14T21:25:26.4713855Z GITHUB_ACTIONS=true
2025-08-14T21:25:26.4714091Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4714418Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4714865Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main
2025-08-14T21:25:26.4715267Z UCC_HOME=/usr
2025-08-14T21:25:26.4715475Z TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:25:26.4715716Z VERBOSE_TEST_LOGS=False
2025-08-14T21:25:26.4715940Z GITHUB_REF=refs/heads/main
2025-08-14T21:25:26.4716159Z SHARD_NUMBER=1
2025-08-14T21:25:26.4716366Z GITHUB_REF_PROTECTED=true
2025-08-14T21:25:26.4716590Z HOME=/var/lib/jenkins
2025-08-14T21:25:26.4716833Z GITHUB_API_URL=https://api.github.com
2025-08-14T21:25:26.4717127Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-08-14T21:25:26.4717377Z LANG=C.UTF-8
2025-08-14T21:25:26.4717618Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-08-14T21:25:26.4718154Z PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:25:26.4718382Z NUM_TEST_SHARDS=3
2025-08-14T21:25:26.4718581Z UCX_HOME=/usr
2025-08-14T21:25:26.4719098Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4719887Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:25:26.4720370Z MAGMA_HOME=/opt/rocm/magma
2025-08-14T21:25:26.4720895Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4721614Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-08-14T21:25:26.4722061Z GITHUB_EVENT_NAME=schedule
2025-08-14T21:25:26.4722395Z DASHBOARD_TAG=
2025-08-14T21:25:26.4722600Z GITHUB_RUN_ID=16976370303
2025-08-14T21:25:26.4723156Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4723751Z GITHUB_ACTOR=huydhn
2025-08-14T21:25:26.4723947Z PR_NUMBER=
2025-08-14T21:25:26.4724141Z GITHUB_RUN_ATTEMPT=1
2025-08-14T21:25:26.4724512Z VALGRIND=ON
2025-08-14T21:25:26.4724722Z ANACONDA_PYTHON_VERSION=3.10
2025-08-14T21:25:26.4725012Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-08-14T21:25:26.4725326Z TERM=vt100
2025-08-14T21:25:26.4725518Z INSTALLED_VISION=yes
2025-08-14T21:25:26.4725720Z BRANCH=main
2025-08-14T21:25:26.4725917Z OPENSSL_ROOT_DIR=/opt/openssl
2025-08-14T21:25:26.4726151Z TESTS_TO_INCLUDE=
2025-08-14T21:25:26.4726577Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-08-14T21:25:26.4727081Z GITHUB_SERVER_URL=https://github.com
2025-08-14T21:25:26.4727357Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-08-14T21:25:26.4727631Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-08-14T21:25:26.4727915Z REENABLED_ISSUES=
2025-08-14T21:25:26.4728116Z SHLVL=1
2025-08-14T21:25:26.4728293Z MAX_JOBS=126
2025-08-14T21:25:26.4728485Z GITHUB_ACTOR_ID=475357
2025-08-14T21:25:26.4728765Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:25:26.4729083Z GITHUB_REF_NAME=main
2025-08-14T21:25:26.4729291Z ROCM_PATH=/opt/rocm
2025-08-14T21:25:26.4729498Z GITHUB_JOB=test
2025-08-14T21:25:26.4729695Z NO_TEST_TIMEOUT=False
2025-08-14T21:25:26.4729927Z GITHUB_REPOSITORY=pytorch/pytorch
2025-08-14T21:25:26.4730179Z LC_ALL=C.UTF-8
2025-08-14T21:25:26.4730379Z GITHUB_RETENTION_DAYS=90
2025-08-14T21:25:26.4730604Z OPENSSL_DIR=/opt/openssl
2025-08-14T21:25:26.4730830Z GITHUB_ACTION_REPOSITORY=
2025-08-14T21:25:26.4731627Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:25:26.4732433Z GITHUB_BASE_REF=
2025-08-14T21:25:26.4732625Z CI=true
2025-08-14T21:25:26.4732824Z GITHUB_REPOSITORY_OWNER=pytorch
2025-08-14T21:25:26.4733060Z JOB_ID=48128182516
2025-08-14T21:25:26.4733257Z GITHUB_HEAD_REF=
2025-08-14T21:25:26.4733457Z GITHUB_ACTION_REF=
2025-08-14T21:25:26.4733659Z TEST_SHOWLOCALS=False
2025-08-14T21:25:26.4733886Z GITHUB_WORKFLOW=periodic
2025-08-14T21:25:26.4734115Z DEBIAN_FRONTEND=noninteractive
2025-08-14T21:25:26.4734672Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_d737a0ab-4db0-473a-a499-b7ae72cae254
2025-08-14T21:25:26.4735229Z NO_TD=False
2025-08-14T21:25:26.4735422Z OLDPWD=/var/lib/jenkins
2025-08-14T21:25:26.4735631Z _=/usr/bin/env
2025-08-14T21:25:26.4735829Z + echo 'Testing pytorch'
2025-08-14T21:25:26.4736042Z Testing pytorch
2025-08-14T21:25:26.4736243Z + export LANG=C.UTF-8
2025-08-14T21:25:26.4736448Z + LANG=C.UTF-8
2025-08-14T21:25:26.4736629Z + PR_NUMBER=
2025-08-14T21:25:26.4736836Z + [[ distributed == \d\e\f\a\u\l\t ]]
2025-08-14T21:25:26.4737117Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-08-14T21:25:26.4737589Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:25:26.4737878Z + export HIP_VISIBLE_DEVICES=0,1,2,3
2025-08-14T21:25:26.4738144Z + HIP_VISIBLE_DEVICES=0,1,2,3
2025-08-14T21:25:26.4738390Z + [[ distributed == \s\l\o\w ]]
2025-08-14T21:25:26.4738673Z + [[ linux-jammy-rocm-py3.10 == *slow-gradcheck* ]]
2025-08-14T21:25:26.4738987Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-08-14T21:25:26.4739269Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:25:26.4739552Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-08-14T21:25:26.4739843Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-08-14T21:25:26.4740114Z + [[ distributed == *crossref* ]]
2025-08-14T21:25:26.4740386Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:25:26.4740652Z + export VALGRIND=OFF
2025-08-14T21:25:26.4740860Z + VALGRIND=OFF
2025-08-14T21:25:26.4741050Z + rocminfo
2025-08-14T21:25:26.4854063Z ROCk module version 6.10.5 is loaded
2025-08-14T21:25:26.7002270Z =====================    
2025-08-14T21:25:26.7002884Z HSA System Attributes    
2025-08-14T21:25:26.7003311Z =====================    
2025-08-14T21:25:26.7003726Z Runtime Version:         1.15
2025-08-14T21:25:26.7004153Z Runtime Ext Version:     1.7
2025-08-14T21:25:26.7005226Z System Timestamp Freq.:  1000.000000MHz
2025-08-14T21:25:26.7006015Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-08-14T21:25:26.7006915Z Machine Model:           LARGE                              
2025-08-14T21:25:26.7007890Z System Endianness:       LITTLE                             
2025-08-14T21:25:26.7008725Z Mwaitx:                  DISABLED
2025-08-14T21:25:26.7009371Z XNACK enabled:           NO
2025-08-14T21:25:26.7009989Z DMAbuf Support:          YES
2025-08-14T21:25:26.7010602Z VMM Support:             YES
2025-08-14T21:25:26.7010848Z 
2025-08-14T21:25:26.7010973Z ==========               
2025-08-14T21:25:26.7011299Z HSA Agents               
2025-08-14T21:25:26.7011579Z ==========               
2025-08-14T21:25:26.7011847Z *******                  
2025-08-14T21:25:26.7012126Z Agent 1                  
2025-08-14T21:25:26.7012397Z *******                  
2025-08-14T21:25:26.7012736Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.7013164Z   Uuid:                    CPU-XX                             
2025-08-14T21:25:26.7013613Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.7014081Z   Vendor Name:             CPU                                
2025-08-14T21:25:26.7014530Z   Feature:                 None specified                     
2025-08-14T21:25:26.7014974Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:25:26.7015422Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7015957Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:25:26.7016390Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:25:26.7016858Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:25:26.7017312Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7017669Z   Node:                    0                                  
2025-08-14T21:25:26.7018031Z   Device Type:             CPU                                
2025-08-14T21:25:26.7018368Z   Cache Info:              
2025-08-14T21:25:26.7018644Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:25:26.7018998Z   Chip ID:                 0(0x0)                             
2025-08-14T21:25:26.7019361Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:25:26.7019747Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:25:26.7020137Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:25:26.7020493Z   BDFID:                   0                                  
2025-08-14T21:25:26.7020845Z   Internal Node ID:        0                                  
2025-08-14T21:25:26.7021473Z   Compute Unit:            64                                 
2025-08-14T21:25:26.7021841Z   SIMDs per CU:            0                                  
2025-08-14T21:25:26.7022221Z   Shader Engines:          0                                  
2025-08-14T21:25:26.7022612Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:25:26.7023011Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:25:26.7023362Z   Memory Properties:       
2025-08-14T21:25:26.7023621Z   Features:                None
2025-08-14T21:25:26.7023885Z   Pool Info:               
2025-08-14T21:25:26.7024130Z     Pool 1                   
2025-08-14T21:25:26.7024431Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7024817Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:25:26.7025190Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7025579Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7025994Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7026560Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7026968Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7027307Z     Pool 2                   
2025-08-14T21:25:26.7027615Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7027962Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:25:26.7028306Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7028650Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7029012Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7029381Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7029731Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7030039Z     Pool 3                   
2025-08-14T21:25:26.7030311Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:25:26.7030644Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:25:26.7030970Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7031307Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7031655Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7032007Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7032347Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7032646Z     Pool 4                   
2025-08-14T21:25:26.7032915Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7033233Z       Size:                    528249792(0x1f7c73c0) KB           
2025-08-14T21:25:26.7033556Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7033894Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7034249Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7034604Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7034948Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7035248Z   ISA Info:                
2025-08-14T21:25:26.7035470Z *******                  
2025-08-14T21:25:26.7035684Z Agent 2                  
2025-08-14T21:25:26.7035893Z *******                  
2025-08-14T21:25:26.7036150Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.7036473Z   Uuid:                    CPU-XX                             
2025-08-14T21:25:26.7036814Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.7037156Z   Vendor Name:             CPU                                
2025-08-14T21:25:26.7037631Z   Feature:                 None specified                     
2025-08-14T21:25:26.7037978Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:25:26.7038313Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7038654Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:25:26.7038986Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:25:26.7039305Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:25:26.7039644Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7039965Z   Node:                    1                                  
2025-08-14T21:25:26.7040278Z   Device Type:             CPU                                
2025-08-14T21:25:26.7040578Z   Cache Info:              
2025-08-14T21:25:26.7040821Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:25:26.7041125Z   Chip ID:                 0(0x0)                             
2025-08-14T21:25:26.7041443Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:25:26.7041914Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:25:26.7042334Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:25:26.7042659Z   BDFID:                   0                                  
2025-08-14T21:25:26.7042977Z   Internal Node ID:        1                                  
2025-08-14T21:25:26.7043306Z   Compute Unit:            64                                 
2025-08-14T21:25:26.7043622Z   SIMDs per CU:            0                                  
2025-08-14T21:25:26.7043949Z   Shader Engines:          0                                  
2025-08-14T21:25:26.7044285Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:25:26.7044633Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:25:26.7044954Z   Memory Properties:       
2025-08-14T21:25:26.7045192Z   Features:                None
2025-08-14T21:25:26.7045425Z   Pool Info:               
2025-08-14T21:25:26.7045650Z     Pool 1                   
2025-08-14T21:25:26.7045925Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7046266Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:25:26.7046588Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7046926Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7047291Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7047654Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7048002Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7048303Z     Pool 2                   
2025-08-14T21:25:26.7048585Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7048915Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:25:26.7049236Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7049572Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7049923Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7050281Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7050627Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7050923Z     Pool 3                   
2025-08-14T21:25:26.7051194Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:25:26.7051511Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:25:26.7051828Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7052163Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7052687Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7053038Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7053393Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7053691Z     Pool 4                   
2025-08-14T21:25:26.7053959Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7054282Z       Size:                    528402468(0x1f7ec824) KB           
2025-08-14T21:25:26.7054608Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7054945Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7055299Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:25:26.7055659Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7056007Z       Accessible by all:       TRUE                               
2025-08-14T21:25:26.7056310Z   ISA Info:                
2025-08-14T21:25:26.7056536Z *******                  
2025-08-14T21:25:26.7056745Z Agent 3                  
2025-08-14T21:25:26.7057142Z *******                  
2025-08-14T21:25:26.7057390Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7057705Z   Uuid:                    GPU-3c8d38f625d62637               
2025-08-14T21:25:26.7058043Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7058389Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7058717Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7059040Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7059372Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7059708Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7060047Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7060363Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7060699Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7061003Z   Node:                    2                                  
2025-08-14T21:25:26.7061315Z   Device Type:             GPU                                
2025-08-14T21:25:26.7061611Z   Cache Info:              
2025-08-14T21:25:26.7061852Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7062143Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7062437Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7062755Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7063088Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7063422Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7063735Z   BDFID:                   12800                              
2025-08-14T21:25:26.7064054Z   Internal Node ID:        2                                  
2025-08-14T21:25:26.7064387Z   Compute Unit:            104                                
2025-08-14T21:25:26.7064709Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7065042Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7065384Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7065730Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7066095Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7066411Z   Memory Properties:       
2025-08-14T21:25:26.7066669Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7066982Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7067319Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7067813Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7068136Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7068418Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7068706Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7068985Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7069295Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7069643Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7069984Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7070287Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7070530Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7070806Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7071101Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7071429Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7071958Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7072317Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7072673Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7072983Z   Pool Info:               
2025-08-14T21:25:26.7073206Z     Pool 1                   
2025-08-14T21:25:26.7073485Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7073825Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7074152Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7074488Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7074844Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7075213Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7075565Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7075870Z     Pool 2                   
2025-08-14T21:25:26.7076148Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7076476Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7076797Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7077146Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7077494Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7077856Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7078201Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7078498Z     Pool 3                   
2025-08-14T21:25:26.7078778Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7079104Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7079432Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7079776Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7080139Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7080502Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7080851Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7081153Z     Pool 4                   
2025-08-14T21:25:26.7081417Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7081721Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7082032Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7082616Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7082985Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7083358Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7083721Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7084037Z   ISA Info:                
2025-08-14T21:25:26.7084264Z     ISA 1                    
2025-08-14T21:25:26.7084566Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7084946Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7085310Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7085667Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7086037Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7086371Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7086717Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7087039Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7090010Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7090329Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7090611Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7090921Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7091229Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7091497Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7091792Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7092084Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7092401Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7092709Z *******                  
2025-08-14T21:25:26.7092933Z Agent 4                  
2025-08-14T21:25:26.7093147Z *******                  
2025-08-14T21:25:26.7093397Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7093716Z   Uuid:                    GPU-77e549644b650cda               
2025-08-14T21:25:26.7094060Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7094412Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7094745Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7095081Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7095413Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7095759Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7096091Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7096423Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7096749Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7097065Z   Node:                    3                                  
2025-08-14T21:25:26.7097394Z   Device Type:             GPU                                
2025-08-14T21:25:26.7097693Z   Cache Info:              
2025-08-14T21:25:26.7097942Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7098232Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7098537Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7098855Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7099201Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7099560Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7099888Z   BDFID:                   13568                              
2025-08-14T21:25:26.7100379Z   Internal Node ID:        3                                  
2025-08-14T21:25:26.7100719Z   Compute Unit:            104                                
2025-08-14T21:25:26.7101059Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7101394Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7101738Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7102088Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7102451Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7102762Z   Memory Properties:       
2025-08-14T21:25:26.7103004Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7103334Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7103681Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7104032Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7104344Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7104608Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7105014Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7105297Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7105606Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7105946Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7106273Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7106577Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7106824Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7107118Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7107402Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7107729Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7108098Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7108461Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7108806Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7109109Z   Pool Info:               
2025-08-14T21:25:26.7109336Z     Pool 1                   
2025-08-14T21:25:26.7109618Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7109952Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7110279Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7110621Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7110979Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7111353Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7111717Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7112027Z     Pool 2                   
2025-08-14T21:25:26.7112310Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7112643Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7112970Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7113319Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7113669Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7114040Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7114395Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7114708Z     Pool 3                   
2025-08-14T21:25:26.7114984Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7115455Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7115780Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7116131Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7116488Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7116851Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7117214Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7117514Z     Pool 4                   
2025-08-14T21:25:26.7117778Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7118103Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7118427Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7118769Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7119135Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7119502Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7120008Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7120325Z   ISA Info:                
2025-08-14T21:25:26.7120554Z     ISA 1                    
2025-08-14T21:25:26.7120848Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7121221Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7121584Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7121942Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7122374Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7122715Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7123060Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7123401Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7123697Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7124009Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7124311Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7124629Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7124950Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7125206Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7125498Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7125787Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7126110Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7126412Z *******                  
2025-08-14T21:25:26.7126639Z Agent 5                  
2025-08-14T21:25:26.7126883Z *******                  
2025-08-14T21:25:26.7127186Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7127574Z   Uuid:                    GPU-e37f2e040c07ff9c               
2025-08-14T21:25:26.7128000Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7128407Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7128806Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7129168Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7129514Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7129857Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7130193Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7130522Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7131025Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7131342Z   Node:                    4                                  
2025-08-14T21:25:26.7131657Z   Device Type:             GPU                                
2025-08-14T21:25:26.7131949Z   Cache Info:              
2025-08-14T21:25:26.7132185Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7132470Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7132760Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7133077Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7133408Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7133744Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7134062Z   BDFID:                   4352                               
2025-08-14T21:25:26.7134386Z   Internal Node ID:        4                                  
2025-08-14T21:25:26.7134725Z   Compute Unit:            104                                
2025-08-14T21:25:26.7135204Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7135538Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7135881Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7136226Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7136577Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7136918Z   Memory Properties:       
2025-08-14T21:25:26.7137331Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7137711Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7138123Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7138534Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7138862Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7139132Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7139423Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7139703Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7140002Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7140351Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7140699Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7141014Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7141275Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7141563Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7141849Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7142178Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7142554Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7142916Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7143262Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7143562Z   Pool Info:               
2025-08-14T21:25:26.7143776Z     Pool 1                   
2025-08-14T21:25:26.7144053Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7144386Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7144712Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7145050Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7145408Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7145776Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7146124Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7146578Z     Pool 2                   
2025-08-14T21:25:26.7146851Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7147178Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7147494Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7147825Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7148174Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7148528Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7148871Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7149167Z     Pool 3                   
2025-08-14T21:25:26.7149441Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7149754Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7150071Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7150403Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7150881Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7151237Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7151583Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7151883Z     Pool 4                   
2025-08-14T21:25:26.7152140Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7152448Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7152764Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7153101Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7153449Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7153812Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7154154Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7154461Z   ISA Info:                
2025-08-14T21:25:26.7154670Z     ISA 1                    
2025-08-14T21:25:26.7154956Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7155323Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7155674Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7156021Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7156379Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7156711Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7157040Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7157364Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7157651Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7157943Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7158218Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7158527Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7158835Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7159089Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7159369Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7159651Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7159966Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7160258Z *******                  
2025-08-14T21:25:26.7160469Z Agent 6                  
2025-08-14T21:25:26.7160816Z *******                  
2025-08-14T21:25:26.7161053Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7161366Z   Uuid:                    GPU-daa9355c85671e61               
2025-08-14T21:25:26.7161712Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7162049Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7162437Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7162763Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7163098Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7163433Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7163759Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7164086Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7164409Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7164720Z   Node:                    5                                  
2025-08-14T21:25:26.7165019Z   Device Type:             GPU                                
2025-08-14T21:25:26.7165469Z   Cache Info:              
2025-08-14T21:25:26.7165724Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7166009Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7166302Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7166625Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7166958Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7167293Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7167607Z   BDFID:                   5120                               
2025-08-14T21:25:26.7167928Z   Internal Node ID:        5                                  
2025-08-14T21:25:26.7168263Z   Compute Unit:            104                                
2025-08-14T21:25:26.7168577Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7168907Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7169245Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7169594Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7169944Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7170254Z   Memory Properties:       
2025-08-14T21:25:26.7170498Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7170809Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7171155Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7171498Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7171812Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7172075Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7172355Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7172630Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7172932Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7173274Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7173614Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7173914Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7174161Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7174442Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7174722Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7175042Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7175401Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7175946Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7176289Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7176588Z   Pool Info:               
2025-08-14T21:25:26.7176813Z     Pool 1                   
2025-08-14T21:25:26.7177090Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7177424Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7177750Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7178084Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7178436Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7178799Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7179155Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7179454Z     Pool 2                   
2025-08-14T21:25:26.7179730Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7180182Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7180508Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7180840Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7181193Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7181552Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7181896Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7182190Z     Pool 3                   
2025-08-14T21:25:26.7182452Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7182770Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7183081Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7183420Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7183774Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7184129Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7184477Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7184776Z     Pool 4                   
2025-08-14T21:25:26.7185034Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7185344Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7185663Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7186002Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7186353Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7186699Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7187085Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7187448Z   ISA Info:                
2025-08-14T21:25:26.7187722Z     ISA 1                    
2025-08-14T21:25:26.7188061Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7188500Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7188884Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7189232Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7189588Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7189916Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7190241Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7190549Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7190970Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7191254Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7191536Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7191837Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7192136Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7192396Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7192680Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7192958Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7193269Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7193566Z *******                  
2025-08-14T21:25:26.7193777Z Agent 7                  
2025-08-14T21:25:26.7193985Z *******                  
2025-08-14T21:25:26.7194224Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7194546Z   Uuid:                    GPU-fe05c03760243cdf               
2025-08-14T21:25:26.7195024Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7195372Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7195699Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7196028Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7196358Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7196699Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7197032Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7197354Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7197678Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7197983Z   Node:                    6                                  
2025-08-14T21:25:26.7198301Z   Device Type:             GPU                                
2025-08-14T21:25:26.7198596Z   Cache Info:              
2025-08-14T21:25:26.7198838Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7209931Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7210354Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7210726Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7211087Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7211442Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7211761Z   BDFID:                   44544                              
2025-08-14T21:25:26.7212084Z   Internal Node ID:        6                                  
2025-08-14T21:25:26.7212434Z   Compute Unit:            104                                
2025-08-14T21:25:26.7212780Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7213118Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7213471Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7213827Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7214194Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7214518Z   Memory Properties:       
2025-08-14T21:25:26.7214790Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7215129Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7215485Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7215842Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7216170Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7216449Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7216995Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7217275Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7217601Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7217956Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7218305Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7218616Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7218873Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7219152Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7219440Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7219770Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7220152Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7220516Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7220870Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7221183Z   Pool Info:               
2025-08-14T21:25:26.7221581Z     Pool 1                   
2025-08-14T21:25:26.7221886Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7222243Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7222587Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7222933Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7223309Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7223691Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7224057Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7224372Z     Pool 2                   
2025-08-14T21:25:26.7224677Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7225017Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7225350Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7225703Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7226063Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7226428Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7226781Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7227111Z     Pool 3                   
2025-08-14T21:25:26.7227438Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7227828Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7228227Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7228642Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7229060Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7229428Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7229784Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7230096Z     Pool 4                   
2025-08-14T21:25:26.7230358Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7230670Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7230982Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7231324Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7231679Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7232041Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7232453Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7232932Z   ISA Info:                
2025-08-14T21:25:26.7233169Z     ISA 1                    
2025-08-14T21:25:26.7233467Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7233851Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7234221Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7234586Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7234944Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7235286Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7235625Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7235956Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7236246Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7236546Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7236832Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7237276Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7237610Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7237885Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7238181Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7238464Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7238791Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7239098Z *******                  
2025-08-14T21:25:26.7239326Z Agent 8                  
2025-08-14T21:25:26.7239548Z *******                  
2025-08-14T21:25:26.7239803Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7240133Z   Uuid:                    GPU-e7591100b7dd4825               
2025-08-14T21:25:26.7240489Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7240852Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7241188Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7241528Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7241867Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7242297Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7242639Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7242971Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7243303Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7243627Z   Node:                    7                                  
2025-08-14T21:25:26.7243947Z   Device Type:             GPU                                
2025-08-14T21:25:26.7244260Z   Cache Info:              
2025-08-14T21:25:26.7244517Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7244831Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7245144Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7245458Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7245790Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7246117Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7246427Z   BDFID:                   45824                              
2025-08-14T21:25:26.7246737Z   Internal Node ID:        7                                  
2025-08-14T21:25:26.7247065Z   Compute Unit:            104                                
2025-08-14T21:25:26.7247393Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7247892Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7248228Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7248587Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7248941Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7249248Z   Memory Properties:       
2025-08-14T21:25:26.7249506Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7249810Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7250147Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7250482Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7250806Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7251068Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7251338Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7251611Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7251908Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7252406Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7252749Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7253043Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7253296Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7253583Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7253869Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7254184Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7254544Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7254903Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7255249Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7255541Z   Pool Info:               
2025-08-14T21:25:26.7255767Z     Pool 1                   
2025-08-14T21:25:26.7256064Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7256394Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7256718Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7257052Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7257409Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7257779Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7258121Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7258413Z     Pool 2                   
2025-08-14T21:25:26.7258687Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7259016Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7259325Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7259664Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7260017Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7260382Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7260720Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7261010Z     Pool 3                   
2025-08-14T21:25:26.7261283Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7261607Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7261922Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7262252Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7262742Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7263093Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7263439Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7263743Z     Pool 4                   
2025-08-14T21:25:26.7264001Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7264303Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7264615Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7264955Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7265306Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7265665Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7266012Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7266321Z   ISA Info:                
2025-08-14T21:25:26.7266548Z     ISA 1                    
2025-08-14T21:25:26.7266838Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7267366Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7267737Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7268090Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7268454Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7268787Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7268922Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7269031Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7269136Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7269232Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7269341Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7269466Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7269573Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7269671Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7269772Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7269867Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7270003Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7270081Z *******                  
2025-08-14T21:25:26.7270166Z Agent 9                  
2025-08-14T21:25:26.7270242Z *******                  
2025-08-14T21:25:26.7270365Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7270482Z   Uuid:                    GPU-d5dc726d0c1f5d2d               
2025-08-14T21:25:26.7270635Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7270762Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7270894Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7271012Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7271139Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7271268Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7271386Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7271508Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7271624Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7271737Z   Node:                    8                                  
2025-08-14T21:25:26.7271862Z   Device Type:             GPU                                
2025-08-14T21:25:26.7272088Z   Cache Info:              
2025-08-14T21:25:26.7272191Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7272294Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7272407Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7272537Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7272661Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7272797Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7272905Z   BDFID:                   36352                              
2025-08-14T21:25:26.7273040Z   Internal Node ID:        8                                  
2025-08-14T21:25:26.7273163Z   Compute Unit:            104                                
2025-08-14T21:25:26.7273292Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7273421Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7273562Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7273716Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7273971Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7274080Z   Memory Properties:       
2025-08-14T21:25:26.7274186Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7274332Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7274462Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7274609Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7274713Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7274823Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7274916Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7275018Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7275155Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7275294Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7275428Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7275535Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7275636Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7275733Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7275835Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7275972Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7276121Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7276251Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7276389Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7276481Z   Pool Info:               
2025-08-14T21:25:26.7276575Z     Pool 1                   
2025-08-14T21:25:26.7276715Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7276842Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7276977Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7277114Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7277266Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7277409Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7277547Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7277644Z     Pool 2                   
2025-08-14T21:25:26.7277781Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7277902Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7278172Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7278317Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7278466Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7278607Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7278753Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7278841Z     Pool 3                   
2025-08-14T21:25:26.7278981Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7279088Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7279217Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7279342Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7279487Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7279618Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7279753Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7279952Z     Pool 4                   
2025-08-14T21:25:26.7280081Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7280183Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7280313Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7280439Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7280583Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7280709Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7280844Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7280927Z   ISA Info:                
2025-08-14T21:25:26.7281011Z     ISA 1                    
2025-08-14T21:25:26.7281157Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7281295Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7281431Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7281567Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7281705Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7281815Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7281951Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7282051Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7282218Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7282317Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7282413Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7282541Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7282639Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7282742Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7282848Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7282947Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7283084Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7283163Z *******                  
2025-08-14T21:25:26.7283243Z Agent 10                 
2025-08-14T21:25:26.7283328Z *******                  
2025-08-14T21:25:26.7283446Z   Name:                    gfx90a                             
2025-08-14T21:25:26.7283576Z   Uuid:                    GPU-144bb695d5984246               
2025-08-14T21:25:26.7283720Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.7284035Z   Vendor Name:             AMD                                
2025-08-14T21:25:26.7284156Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:25:26.7284286Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:25:26.7284414Z   Float Round Mode:        NEAR                               
2025-08-14T21:25:26.7284545Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:25:26.7284663Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:25:26.7284787Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:25:26.7284903Z   Queue Type:              MULTI                              
2025-08-14T21:25:26.7285007Z   Node:                    9                                  
2025-08-14T21:25:26.7285127Z   Device Type:             GPU                                
2025-08-14T21:25:26.7285215Z   Cache Info:              
2025-08-14T21:25:26.7285315Z     L1:                      16(0x10) KB                        
2025-08-14T21:25:26.7285416Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:25:26.7285687Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:25:26.7285817Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:25:26.7285947Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:25:26.7286072Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:25:26.7286183Z   BDFID:                   37632                              
2025-08-14T21:25:26.7286307Z   Internal Node ID:        9                                  
2025-08-14T21:25:26.7286432Z   Compute Unit:            104                                
2025-08-14T21:25:26.7286546Z   SIMDs per CU:            4                                  
2025-08-14T21:25:26.7286676Z   Shader Engines:          8                                  
2025-08-14T21:25:26.7286802Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:25:26.7286944Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:25:26.7287077Z   Coherent Host Access:    FALSE                              
2025-08-14T21:25:26.7287176Z   Memory Properties:       
2025-08-14T21:25:26.7287278Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:25:26.7287411Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:25:26.7287536Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:25:26.7287665Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7287767Z   Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7287861Z     x                        1024(0x400)                        
2025-08-14T21:25:26.7287959Z     y                        1024(0x400)                        
2025-08-14T21:25:26.7288050Z     z                        1024(0x400)                        
2025-08-14T21:25:26.7288181Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:25:26.7288314Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:25:26.7288447Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7288540Z   Grid Max Size per Dimension:
2025-08-14T21:25:26.7288646Z     x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7288746Z     y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7288850Z     z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7288987Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:25:26.7289134Z   Packet Processor uCode:: 92                                 
2025-08-14T21:25:26.7289260Z   SDMA engine uCode::      9                                  
2025-08-14T21:25:26.7289394Z   IOMMU Support::          None                               
2025-08-14T21:25:26.7289479Z   Pool Info:               
2025-08-14T21:25:26.7289705Z     Pool 1                   
2025-08-14T21:25:26.7289839Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:25:26.7289948Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7290090Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7290218Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7290371Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7290499Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7290638Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7290724Z     Pool 2                   
2025-08-14T21:25:26.7290868Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:25:26.7290974Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7291109Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7291239Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7291386Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7291655Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7291805Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7291887Z     Pool 3                   
2025-08-14T21:25:26.7292020Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:25:26.7292127Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:25:26.7292261Z       Allocatable:             TRUE                               
2025-08-14T21:25:26.7292388Z       Alloc Granule:           4KB                                
2025-08-14T21:25:26.7292528Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:25:26.7292662Z       Alloc Alignment:         4KB                                
2025-08-14T21:25:26.7292802Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7292888Z     Pool 4                   
2025-08-14T21:25:26.7293010Z       Segment:                 GROUP                              
2025-08-14T21:25:26.7293118Z       Size:                    64(0x40) KB                        
2025-08-14T21:25:26.7293244Z       Allocatable:             FALSE                              
2025-08-14T21:25:26.7293373Z       Alloc Granule:           0KB                                
2025-08-14T21:25:26.7293515Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:25:26.7293649Z       Alloc Alignment:         0KB                                
2025-08-14T21:25:26.7293782Z       Accessible by all:       FALSE                              
2025-08-14T21:25:26.7293871Z   ISA Info:                
2025-08-14T21:25:26.7293951Z     ISA 1                    
2025-08-14T21:25:26.7294098Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:25:26.7294241Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:25:26.7294380Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:25:26.7294517Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7294659Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:25:26.7294769Z       Fast f16:                TRUE                               
2025-08-14T21:25:26.7294905Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:25:26.7295011Z       Workgroup Max Size per Dimension:
2025-08-14T21:25:26.7295112Z         x                        1024(0x400)                        
2025-08-14T21:25:26.7295212Z         y                        1024(0x400)                        
2025-08-14T21:25:26.7295305Z         z                        1024(0x400)                        
2025-08-14T21:25:26.7295568Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:25:26.7295664Z       Grid Max Size per Dimension:
2025-08-14T21:25:26.7295770Z         x                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7295873Z         y                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7295977Z         z                        4294967295(0xffffffff)             
2025-08-14T21:25:26.7296106Z       FBarrier Max Size:       32                                 
2025-08-14T21:25:26.7296192Z *** Done ***             
2025-08-14T21:25:26.7296280Z + rocminfo
2025-08-14T21:25:26.7296384Z + grep -E 'Name:.*\sgfx|Marketing'
2025-08-14T21:25:26.9480297Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.9481024Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:25:26.9481661Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9482381Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9482950Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9483494Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9484384Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9484956Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9485507Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9486053Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9486607Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9487170Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9487721Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9488263Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9488802Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9489346Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9489879Z   Name:                    gfx90a                             
2025-08-14T21:25:26.9490424Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:25:26.9663519Z + MAYBE_ROCM=rocm/
2025-08-14T21:25:26.9664490Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-08-14T21:25:26.9665111Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-08-14T21:25:26.9665626Z + pip_install ninja==1.10.2
2025-08-14T21:25:26.9666196Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-08-14T21:25:26.9666880Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-08-14T21:25:27.3537175Z Collecting ninja==1.10.2
2025-08-14T21:25:27.3756371Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-08-14T21:25:27.3841902Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-08-14T21:25:27.7481470Z Installing collected packages: ninja
2025-08-14T21:25:27.7482455Z   Attempting uninstall: ninja
2025-08-14T21:25:27.7489254Z     Found existing installation: ninja 1.11.1.3
2025-08-14T21:25:27.7511744Z     Uninstalling ninja-1.11.1.3:
2025-08-14T21:25:27.7600322Z       Successfully uninstalled ninja-1.11.1.3
2025-08-14T21:25:27.7932212Z Successfully installed ninja-1.10.2
2025-08-14T21:25:27.8540009Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:25:27.8542799Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:25:27.8544457Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-08-14T21:25:27.8545514Z + [[ linux-jammy-rocm-py3.10 == *asan* ]]
2025-08-14T21:25:27.8546020Z + [[ linux-jammy-rocm-py3.10 == *-debug* ]]
2025-08-14T21:25:27.8546531Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-08-14T21:25:27.8547258Z + echo 'We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass'
2025-08-14T21:25:27.8548182Z We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass
2025-08-14T21:25:27.8548939Z + cd test
2025-08-14T21:25:27.8549680Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-08-14T21:25:29.3807424Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-08-14T21:25:29.3808329Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-08-14T21:25:29.3809241Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-08-14T21:25:29.3812719Z + DYNAMO_BENCHMARK_FLAGS=()
2025-08-14T21:25:29.3813152Z + [[ distributed == *pr_time_benchmarks* ]]
2025-08-14T21:25:29.3813605Z + [[ distributed == *dynamo_eager* ]]
2025-08-14T21:25:29.3814056Z + [[ distributed == *aot_eager* ]]
2025-08-14T21:25:29.3814468Z + [[ distributed == *aot_inductor* ]]
2025-08-14T21:25:29.3814917Z + [[ distributed == *max_autotune_inductor* ]]
2025-08-14T21:25:29.3815827Z + [[ distributed == *inductor* ]]
2025-08-14T21:25:29.3816265Z + [[ distributed == *dynamic* ]]
2025-08-14T21:25:29.3816655Z + [[ distributed == *cpu* ]]
2025-08-14T21:25:29.3817079Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-08-14T21:25:29.3833930Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-08-14T21:25:29.3834272Z + [[ linux-jammy-rocm-py3.10 == *-bazel-* ]]
2025-08-14T21:25:29.3840302Z + cd test
2025-08-14T21:25:29.3840801Z + python -c 'import torch; print(torch.__config__.show())'
2025-08-14T21:25:30.8741989Z PyTorch built with:
2025-08-14T21:25:30.8742455Z   - GCC 11.4
2025-08-14T21:25:30.8742827Z   - C++ Version: 201703
2025-08-14T21:25:30.8743688Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-08-14T21:25:30.8744782Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-08-14T21:25:30.8745439Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-08-14T21:25:30.8745981Z   - LAPACK is enabled (usually provided by MKL)
2025-08-14T21:25:30.8746497Z   - NNPACK is enabled
2025-08-14T21:25:30.8746794Z   - CPU capability usage: AVX2
2025-08-14T21:25:30.8747049Z   - HIP Runtime 6.4.43484
2025-08-14T21:25:30.8747280Z   - MIOpen 3.4.0
2025-08-14T21:25:30.8747483Z   - Magma 2.7.2
2025-08-14T21:25:30.8751503Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=1fc683cf17c8c673044538d10266c00f92987be2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-08-14T21:25:30.8755723Z 
2025-08-14T21:25:31.1605269Z + cd test
2025-08-14T21:25:31.1606209Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-08-14T21:25:32.4110705Z ATen/Parallel:
2025-08-14T21:25:32.4111161Z 	at::get_num_threads() : 128
2025-08-14T21:25:32.4111611Z 	at::get_num_interop_threads() : 128
2025-08-14T21:25:32.4116269Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-08-14T21:25:32.4116814Z 	omp_get_max_threads() : 128
2025-08-14T21:25:32.4117849Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-08-14T21:25:32.4118834Z 	mkl_get_max_threads() : 128
2025-08-14T21:25:32.4119246Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-08-14T21:25:32.4119665Z std::thread::hardware_concurrency() : 128
2025-08-14T21:25:32.4119954Z Environment variables:
2025-08-14T21:25:32.4120187Z 	OMP_NUM_THREADS : [not set]
2025-08-14T21:25:32.4120430Z 	MKL_NUM_THREADS : [not set]
2025-08-14T21:25:32.4120670Z ATen parallel backend: OpenMP
2025-08-14T21:25:32.4120839Z 
2025-08-14T21:25:32.6672338Z + [[ distributed == *numpy_2* ]]
2025-08-14T21:25:32.6672845Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-08-14T21:25:32.6673358Z + [[ distributed == *backward* ]]
2025-08-14T21:25:32.6673792Z + [[ distributed == *xla* ]]
2025-08-14T21:25:32.6674208Z + [[ distributed == *executorch* ]]
2025-08-14T21:25:32.6674679Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2025-08-14T21:25:32.6675199Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-08-14T21:25:32.6675687Z + [[ distributed == distributed ]]
2025-08-14T21:25:32.6676476Z + test_distributed
2025-08-14T21:25:32.6676890Z + echo 'Testing distributed python tests'
2025-08-14T21:25:32.6677381Z Testing distributed python tests
2025-08-14T21:25:32.6677983Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2025-08-14T21:25:35.6420902Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T21:25:35.6423243Z   import pkg_resources
2025-08-14T21:25:36.4118197Z Excluding distributed/rpc/test_faulty_agent on ROCm
2025-08-14T21:25:36.4118953Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm
2025-08-14T21:25:36.4119599Z Excluding distributed/rpc/test_share_memory on ROCm
2025-08-14T21:25:36.4120251Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm
2025-08-14T21:25:37.9402576Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-14T21:25:38.1941650Z Ignoring disabled issues:  ['']
2025-08-14T21:25:38.2069870Z Found test times from artifacts
2025-08-14T21:25:38.2562922Z Found test times from artifacts
2025-08-14T21:25:38.2576987Z Running all tests
2025-08-14T21:25:38.2688127Z Running parallel tests on 8 processes
2025-08-14T21:25:38.2692378Z Name: tests to run (est. time: 125.52min)
2025-08-14T21:25:38.2692880Z   Serial tests (81):
2025-08-14T21:25:38.2693347Z     distributed/fsdp/test_fsdp_apply 1/1
2025-08-14T21:25:38.2693868Z     distributed/test_dynamo_distributed 2/2
2025-08-14T21:25:38.2694455Z     distributed/tensor/parallel/test_micro_pipeline_tp 1/1
2025-08-14T21:25:38.2695106Z     distributed/tensor/debug/test_op_coverage 1/1
2025-08-14T21:25:38.2695654Z     distributed/_tools/test_mod_tracker 1/1
2025-08-14T21:25:38.2696224Z     distributed/checkpoint/test_nested_dict 1/1
2025-08-14T21:25:38.2696603Z     distributed/flight_recorder/test_fr_analysis 1/1
2025-08-14T21:25:38.2696963Z     distributed/test_compute_comm_reordering 1/1
2025-08-14T21:25:38.2697369Z     distributed/checkpoint/test_consolidate_hf_safetensors 1/1
2025-08-14T21:25:38.2697775Z     distributed/_tools/test_runtime_estimator 1/1
2025-08-14T21:25:38.2698103Z     distributed/_tools/test_sac_ilp 1/1
2025-08-14T21:25:38.2698454Z     distributed/pipelining/test_schedule_multiproc 1/1
2025-08-14T21:25:38.2698814Z     distributed/pipelining/test_pipe 1/1
2025-08-14T21:25:38.2699140Z     distributed/_tools/test_fake_collectives 1/1
2025-08-14T21:25:38.2699479Z     distributed/pipelining/test_backward 1/1
2025-08-14T21:25:38.2699854Z     distributed/checkpoint/_experimental/test_barriers 1/1
2025-08-14T21:25:38.2700767Z     distributed/checkpoint/_experimental/test_types 1/1
2025-08-14T21:25:38.2701134Z     distributed/_tools/test_sac_estimator 1/1
2025-08-14T21:25:38.2701485Z     distributed/tensor/test_xla_integration 1/1
2025-08-14T21:25:38.2701835Z     distributed/checkpoint/test_dedup_tensors 1/1
2025-08-14T21:25:38.2702180Z     distributed/pipelining/test_unflatten 1/1
2025-08-14T21:25:38.2702519Z     distributed/checkpoint/test_format_utils 1/1
2025-08-14T21:25:38.2702857Z     distributed/tensor/test_dtensor_ops 1/1
2025-08-14T21:25:38.2703186Z     distributed/pipelining/test_microbatch 1/1
2025-08-14T21:25:38.2703511Z     distributed/tensor/test_api 1/1
2025-08-14T21:25:38.2703821Z     distributed/tensor/test_convolution_ops 1/1
2025-08-14T21:25:38.2704237Z     distributed/checkpoint/_experimental/test_checkpoint_reader 1/1
2025-08-14T21:25:38.2704750Z     distributed/_composable/test_composability/test_pp_composability 1/1
2025-08-14T21:25:38.2705183Z     distributed/tensor/test_tensor_ops 1/1
2025-08-14T21:25:38.2705519Z     distributed/_tools/test_fsdp2_mem_tracker 1/1
2025-08-14T21:25:38.2705867Z     distributed/checkpoint/test_traverse 1/1
2025-08-14T21:25:38.2706392Z     distributed/checkpoint/test_fsspec 1/1
2025-08-14T21:25:38.2706730Z     distributed/tensor/test_redistribute 1/1
2025-08-14T21:25:38.2707064Z     distributed/checkpoint/test_fsdp_optim_state 1/1
2025-08-14T21:25:38.2707406Z     distributed/checkpoint/test_dtensor_resharding 1/1
2025-08-14T21:25:38.2707725Z     distributed/_tools/test_mem_tracker 1/1
2025-08-14T21:25:38.2708016Z     distributed/pipelining/test_schedule 1/1
2025-08-14T21:25:38.2708313Z     distributed/checkpoint/test_save_load_api 1/1
2025-08-14T21:25:38.2708646Z     distributed/checkpoint/e2e/test_e2e_save_and_load 1/1
2025-08-14T21:25:38.2708972Z     distributed/tensor/test_utils 1/1
2025-08-14T21:25:38.2709289Z     distributed/optim/test_apply_optimizer_in_backward 1/1
2025-08-14T21:25:38.2709666Z     distributed/_composable/test_replicate_with_compiler 1/1
2025-08-14T21:25:38.2710057Z     distributed/_composable/fsdp/test_fully_shard_compile 1/1
2025-08-14T21:25:38.2710417Z     distributed/checkpoint/test_fsdp_model_state 1/1
2025-08-14T21:25:38.2710732Z     distributed/fsdp/test_shard_utils 1/1
2025-08-14T21:25:38.2711022Z     distributed/elastic/test_control_plane 1/1
2025-08-14T21:25:38.2711379Z     distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1
2025-08-14T21:25:38.2711810Z     distributed/checkpoint/_experimental/test_checkpoint_process 1/1
2025-08-14T21:25:38.2712213Z     distributed/tensor/parallel/test_parallelize_api 1/1
2025-08-14T21:25:38.2712547Z     distributed/checkpoint/test_checkpoint 1/1
2025-08-14T21:25:38.2712837Z     distributed/test_collective_utils 1/1
2025-08-14T21:25:38.2713113Z     distributed/checkpoint/test_utils 1/1
2025-08-14T21:25:38.2713397Z     distributed/fsdp/test_fsdp_state_dict 2/2
2025-08-14T21:25:38.2713708Z     distributed/checkpoint/test_state_dict_stager 1/1
2025-08-14T21:25:38.2714019Z     distributed/fsdp/test_utils 1/1
2025-08-14T21:25:38.2714288Z     distributed/fsdp/test_fsdp_overlap 1/1
2025-08-14T21:25:38.2714584Z     distributed/fsdp/test_checkpoint_wrapper 1/1
2025-08-14T21:25:38.2714900Z     distributed/fsdp/test_fsdp_flatten_params 1/1
2025-08-14T21:25:38.2715228Z     distributed/optim/test_zero_redundancy_optimizer 1/1
2025-08-14T21:25:38.2715565Z     distributed/fsdp/test_fsdp_ignored_modules 1/1
2025-08-14T21:25:38.2715860Z     distributed/test_nccl 1/1
2025-08-14T21:25:38.2716118Z     distributed/fsdp/test_fsdp_checkpoint 1/1
2025-08-14T21:25:38.2716407Z     distributed/test_distributed_spawn 2/8
2025-08-14T21:25:38.2716689Z     distributed/test_distributed_spawn 5/8
2025-08-14T21:25:38.2716967Z     distributed/test_distributed_spawn 8/8
2025-08-14T21:25:38.2717260Z     distributed/fsdp/test_fsdp_hybrid_shard 1/1
2025-08-14T21:25:38.2717554Z     distributed/fsdp/test_fsdp_exec_order 1/1
2025-08-14T21:25:38.2717835Z     distributed/fsdp/test_wrap 1/1
2025-08-14T21:25:38.2718239Z     distributed/fsdp/test_fsdp_uneven 1/1
2025-08-14T21:25:38.2718570Z     distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1
2025-08-14T21:25:38.2718974Z     distributed/_composable/fsdp/test_fully_shard_state_dict 1/1
2025-08-14T21:25:38.2719322Z     distributed/fsdp/test_fsdp_input 1/1
2025-08-14T21:25:38.2719637Z     distributed/fsdp/test_fsdp_multiple_wrapping 1/1
2025-08-14T21:25:38.2719952Z     distributed/_shard/test_sharder 1/1
2025-08-14T21:25:38.2720232Z     distributed/algorithms/test_join 1/1
2025-08-14T21:25:38.2720507Z     distributed/test_c10d_pypg 1/1
2025-08-14T21:25:38.2720784Z     distributed/fsdp/test_fsdp_use_orig_params 1/1
2025-08-14T21:25:38.2721079Z     distributed/fsdp/test_fsdp_core 3/3
2025-08-14T21:25:38.2721337Z     distributed/test_c10d_gloo 2/2
2025-08-14T21:25:38.2721585Z     distributed/test_c10d_nccl 2/3
2025-08-14T21:25:38.2721860Z     distributed/elastic/timer/api_test 1/1
2025-08-14T21:25:38.2722234Z   Parallel tests (0):
2025-08-14T21:25:38.2722484Z Name: excluded (est. time: 0.0min)
2025-08-14T21:25:38.2722734Z   Serial tests (0):
2025-08-14T21:25:38.2722950Z   Parallel tests (0):
2025-08-14T21:25:38.2723459Z Running distributed/fsdp/test_fsdp_apply 1/1 ... [2025-08-14 21:25:38.269888]
2025-08-14T21:25:38.2723876Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:25:38.2724801Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:25:38.270229]
2025-08-14T21:26:03.8181181Z 
2025-08-14T21:26:03.8182442Z distributed/fsdp/test_fsdp_apply 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_apply_1.1_ea26f67714b2503e_.log
2025-08-14T21:26:03.8185254Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_apply_in_summon_raises_error_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_nested_module_apply_cuda, test/distributed/fsdp/test_fsdp_apply.py::TestApplyCUDA::test_transformer_module_apply_cuda
2025-08-14T21:26:03.8187077Z 
2025-08-14T21:26:03.8187495Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T21:26:03.8188202Z Uploading artifacts took 0.00 seconds
2025-08-14T21:26:03.8188925Z Running distributed/test_dynamo_distributed 2/2 ... [2025-08-14 21:26:03.818662]
2025-08-14T21:26:03.8189656Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:26:03.8191682Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:26:03.818996]
2025-08-14T21:30:06.9057757Z 
2025-08-14T21:30:06.9058810Z distributed/test_dynamo_distributed 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_2.2_ac577dcbcefa81b9_.log
2025-08-14T21:30:06.9072034Z Running 23 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_symbol_splitting, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_direct, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_indirect, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_torture_multi, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_asymmetric_compilation_with_fx_cache, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_graph_break_empty_graph_still_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_optimizer_cudagraph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_setattr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_fsdp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_aot_autograd, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_async_subclass_no_specialize, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_local_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_custom_layer, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_aot_eager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_empty_graph_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_staticmethod, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_training
2025-08-14T21:30:06.9081097Z 
2025-08-14T21:30:06.9081402Z Running distributed/tensor/parallel/test_micro_pipeline_tp 1/1 ... [2025-08-14 21:30:06.906075]
2025-08-14T21:30:06.9081894Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:06.9082998Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_micro_pipeline_tp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:06.906380]
2025-08-14T21:30:30.0513062Z 
2025-08-14T21:30:30.0514727Z distributed/tensor/parallel/test_micro_pipeline_tp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_micro_pipeline_tp_1.1_2020bce4345d6b7e_.log
2025-08-14T21:30:30.0541683Z Running 45 items in this shard: test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_dtensor_seq_par_shard_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_all_gather_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_find_reduce_scatter_patterns, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_2_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_0_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_1_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_False, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_all_gather_scaled_matmul_A_dims_3_gather_dim_2_return_A_True, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_2_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_A_dims_3_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_0, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_1, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_fuse_scaled_matmul_reduce_scatter_rowwise_scales_reshape_mm_reshape_scatter_dim_2, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTPTest::test_get_unexposed_collectives, test/distributed/tensor/parallel/test_micro_pipeline_tp.py::MicroPipelineTP4GPUTest::test_extra_collectives
2025-08-14T21:30:30.0563105Z 
2025-08-14T21:30:30.0563350Z Running distributed/tensor/debug/test_op_coverage 1/1 ... [2025-08-14 21:30:30.051847]
2025-08-14T21:30:30.0563791Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:30.0564745Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/debug/test_op_coverage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:30.052162]
2025-08-14T21:30:34.7230187Z 
2025-08-14T21:30:34.7231524Z distributed/tensor/debug/test_op_coverage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.debug.test_op_coverage_1.1_58261fc93b8708fa_.log
2025-08-14T21:30:34.7232718Z Running 1 items in this shard: test/distributed/tensor/debug/test_op_coverage.py::TestOpCoverage::test_trace_with_inductor_decomp
2025-08-14T21:30:34.7233215Z 
2025-08-14T21:30:34.7237386Z Running distributed/_tools/test_mod_tracker 1/1 ... [2025-08-14 21:30:34.723560]
2025-08-14T21:30:34.7238108Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:34.7241162Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mod_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:34.723871]
2025-08-14T21:30:39.0440806Z 
2025-08-14T21:30:39.0442062Z distributed/_tools/test_mod_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mod_tracker_1.1_e0e18a85f4a1c774_.log
2025-08-14T21:30:39.0445564Z Running 4 items in this shard: test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_ac, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_bw_detection, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_module_hierarchy, test/distributed/_tools/test_mod_tracker.py::TestModTracker::test_user_hooks
2025-08-14T21:30:39.0448111Z 
2025-08-14T21:30:39.0448861Z Running distributed/checkpoint/test_nested_dict 1/1 ... [2025-08-14 21:30:39.044690]
2025-08-14T21:30:39.0449357Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:39.0451953Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_nested_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:39.045023]
2025-08-14T21:30:43.2651068Z 
2025-08-14T21:30:43.2652344Z distributed/checkpoint/test_nested_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_nested_dict_1.1_1f3c47682a74a35d_.log
2025-08-14T21:30:43.2654925Z Running 2 items in this shard: test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_flattening_round_trip, test/distributed/checkpoint/test_nested_dict.py::TestFlattening::test_mapping
2025-08-14T21:30:43.2656752Z 
2025-08-14T21:30:43.2659191Z Running distributed/flight_recorder/test_fr_analysis 1/1 ... [2025-08-14 21:30:43.265698]
2025-08-14T21:30:43.2659854Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:43.2662339Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/flight_recorder/test_fr_analysis.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:43.266016]
2025-08-14T21:30:47.4359615Z 
2025-08-14T21:30:47.4361036Z distributed/flight_recorder/test_fr_analysis 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.flight_recorder.test_fr_analysis_1.1_1557c4d65748f4e3_.log
2025-08-14T21:30:47.4365442Z Running 4 items in this shard: test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_all_events, test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderEventTest::test_match_one_event, test/distributed/flight_recorder/test_fr_analysis.py::FlightMatchInfoTest::test_match_info, test/distributed/flight_recorder/test_fr_analysis.py::FlightRecorderE2ETest::testBuildDB
2025-08-14T21:30:47.4367928Z 
2025-08-14T21:30:47.4368197Z Running distributed/test_compute_comm_reordering 1/1 ... [2025-08-14 21:30:47.436541]
2025-08-14T21:30:47.4368645Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:30:47.4370646Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_compute_comm_reordering.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:30:47.436851]
2025-08-14T21:32:31.5854174Z 
2025-08-14T21:32:31.5855039Z distributed/test_compute_comm_reordering 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_compute_comm_reordering_1.1_ae2291c15e8a2a25_.log
2025-08-14T21:32:31.5860326Z Running 8 items in this shard: test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_grouped_scheduler_node, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_inductor_default_comms_ordering, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_nccl_heuristics, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_raise_comms, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_reorder_compute_for_overlap_custom_runtime_estimation, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits, test/distributed/test_compute_comm_reordering.py::TestComputeCommReorderingMultiProc::test_sink_waits_raise_comms
2025-08-14T21:32:31.5864690Z 
2025-08-14T21:32:31.5865028Z Running distributed/checkpoint/test_consolidate_hf_safetensors 1/1 ... [2025-08-14 21:32:31.585885]
2025-08-14T21:32:31.5865576Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:32:31.5866717Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_consolidate_hf_safetensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:32:31.586193]
2025-08-14T21:33:09.3983337Z 
2025-08-14T21:33:09.3984911Z distributed/checkpoint/test_consolidate_hf_safetensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_consolidate_hf_safetensors_1.1_8bb177e3979031b9_.log
2025-08-14T21:33:09.3991910Z Running 6 items in this shard: test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_calculate_max_contiguous_elements_valid_cases, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_calculate_max_contiguous_elements_validations, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_to_one_file, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_to_two_files, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_consolidate_with_two_ranks, test/distributed/checkpoint/test_consolidate_hf_safetensors.py::TestConsolidateHFSafeTensors::test_write_sub_tensor_to_file_optimized
2025-08-14T21:33:09.3995522Z 
2025-08-14T21:33:09.3995783Z Running distributed/_tools/test_runtime_estimator 1/1 ... [2025-08-14 21:33:09.398517]
2025-08-14T21:33:09.3996219Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:33:09.3997164Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_runtime_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:33:09.398844]
2025-08-14T21:33:54.2712177Z 
2025-08-14T21:33:54.2713507Z distributed/_tools/test_runtime_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_runtime_estimator_1.1_0cbd7454e6b93151_.log
2025-08-14T21:33:54.2715975Z Running 2 items in this shard: test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_conv_model_runtime, test/distributed/_tools/test_runtime_estimator.py::TestRuntimeEstimator::test_transformer_runtime
2025-08-14T21:33:54.2718068Z 
2025-08-14T21:33:54.2721777Z Running distributed/_tools/test_sac_ilp 1/1 ... [2025-08-14 21:33:54.271440]
2025-08-14T21:33:54.2722475Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:33:54.2723509Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_ilp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:33:54.271751]
2025-08-14T21:34:05.6010276Z 
2025-08-14T21:34:05.6011548Z distributed/_tools/test_sac_ilp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_ilp_1.1_589ceedb78d65a6e_.log
2025-08-14T21:34:05.6015171Z Running 4 items in this shard: test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case1, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case2, test/distributed/_tools/test_sac_ilp.py::TestSACILP::test_sac_ilp_case3, test/distributed/_tools/test_sac_ilp.py::TestOptimalCheckpointingPolicy::test_get_optimial_checkpointing_policy_per_module
2025-08-14T21:34:05.6017397Z 
2025-08-14T21:34:05.6017864Z Running distributed/pipelining/test_schedule_multiproc 1/1 ... [2025-08-14 21:34:05.601511]
2025-08-14T21:34:05.6018657Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:34:05.6021071Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule_multiproc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:05.601842]
2025-08-14T21:34:44.4667746Z 
2025-08-14T21:34:44.4669163Z distributed/pipelining/test_schedule_multiproc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_multiproc_1.1_cd73a00df7cabdd0_.log
2025-08-14T21:34:44.4690645Z Running 35 items in this shard: test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_eval_inference_mode_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_eval_inference_mode_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_eval_inference_mode_ScheduleClass2, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_eval_inference_mode_ScheduleClass3, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_eval_inference_mode_ScheduleClass4, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_forward_only_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass0_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_ScheduleClass1_shape_inference_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_manual_interleaved_ScheduleClass2_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_grad_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_kwargs_with_tracer_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_multi_iter_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_schedule_with_weight_update_mlp_e2e_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_v_shape_schedules_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_v_shape_schedules_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_v_shape_schedules_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_v_shape_schedules_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_zero_bubble_with_model_kwargs_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::ScheduleTest::test_zero_bubble_with_model_kwargs_ScheduleClass1, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_non_symmetric_stage_ids_schedule_class0_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_False, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_non_symmetric_stage_ids_schedule_class1_use_new_runtime_True, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_pipeline_schedule_runtime_custom_sched_ScheduleClass0, test/distributed/pipelining/test_schedule_multiproc.py::CustomSchedulesTest::test_schedule_with_native_zero_bubble_ScheduleClass0
2025-08-14T21:34:44.4705707Z 
2025-08-14T21:34:44.4705930Z Running distributed/pipelining/test_pipe 1/1 ... [2025-08-14 21:34:44.467373]
2025-08-14T21:34:44.4706338Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:34:44.4707257Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_pipe.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:44.467684]
2025-08-14T21:34:51.7944487Z 
2025-08-14T21:34:51.7945297Z distributed/pipelining/test_pipe 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_pipe_1.1_2d2d94f343344339_.log
2025-08-14T21:34:51.7948365Z Running 3 items in this shard: test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass0, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass1, test/distributed/pipelining/test_pipe.py::PipeTests::test_model_split_ModelClass2
2025-08-14T21:34:51.7950265Z 
2025-08-14T21:34:51.7950684Z Running distributed/_tools/test_fake_collectives 1/1 ... [2025-08-14 21:34:51.794732]
2025-08-14T21:34:51.7951410Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:34:51.7953451Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fake_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:51.795044]
2025-08-14T21:34:56.4658082Z 
2025-08-14T21:34:56.4660310Z distributed/_tools/test_fake_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fake_collectives_1.1_1febbce34645ed56_.log
2025-08-14T21:34:56.4662213Z Running 1 items in this shard: test/distributed/_tools/test_fake_collectives.py::TestFakeCollectives::test_collectives
2025-08-14T21:34:56.4662969Z 
2025-08-14T21:34:56.4663421Z Running distributed/pipelining/test_backward 1/1 ... [2025-08-14 21:34:56.466051]
2025-08-14T21:34:56.4664295Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:34:56.4666521Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:56.466359]
2025-08-14T21:35:02.4387315Z 
2025-08-14T21:35:02.4388764Z distributed/pipelining/test_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_backward_1.1_882889d28e671bac_.log
2025-08-14T21:35:02.4393268Z Running 5 items in this shard: test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_input_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_weight_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_weight_grad_validation_cuda, test/distributed/pipelining/test_backward.py::StageBackwardTestsCUDA::test_stage_backward_weight_multiple_iters_cuda
2025-08-14T21:35:02.4396830Z 
2025-08-14T21:35:02.4397443Z Running distributed/checkpoint/_experimental/test_barriers 1/1 ... [2025-08-14 21:35:02.439058]
2025-08-14T21:35:02.4398457Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:02.4399720Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_barriers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:02.439358]
2025-08-14T21:35:06.6596040Z 
2025-08-14T21:35:06.6597830Z distributed/checkpoint/_experimental/test_barriers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_barriers_1.1_20cd1e274cec841a_.log
2025-08-14T21:35:06.6600877Z Running 2 items in this shard: test/distributed/checkpoint/_experimental/test_barriers.py::TestBarriers::test_execute_barrier, test/distributed/checkpoint/_experimental/test_barriers.py::TestBarriers::test_tcpstore_barrier_initialization
2025-08-14T21:35:06.6602460Z 
2025-08-14T21:35:06.6602936Z Running distributed/checkpoint/_experimental/test_types 1/1 ... [2025-08-14 21:35:06.660016]
2025-08-14T21:35:06.6604530Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:06.6606561Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_types.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:06.660371]
2025-08-14T21:35:10.8804759Z 
2025-08-14T21:35:10.8806246Z distributed/checkpoint/_experimental/test_types 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_types_1.1_b09322ebbbe5a4df_.log
2025-08-14T21:35:10.8809994Z Running 3 items in this shard: test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_rank_info_default_initialization, test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_rank_info_initialization, test/distributed/checkpoint/_experimental/test_types.py::TestRankInfo::test_state_dict_type_alias
2025-08-14T21:35:10.8811591Z 
2025-08-14T21:35:10.8811886Z Running distributed/_tools/test_sac_estimator 1/1 ... [2025-08-14 21:35:10.880973]
2025-08-14T21:35:10.8812401Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:10.8815255Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_sac_estimator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:10.881281]
2025-08-14T21:35:17.8049127Z 
2025-08-14T21:35:17.8050536Z distributed/_tools/test_sac_estimator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_sac_estimator_1.1_720ee96cf9de8ba1_.log
2025-08-14T21:35:17.8052933Z Running 2 items in this shard: test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_simple_model_sac_estimation, test/distributed/_tools/test_sac_estimator.py::TestSACEstimator::test_transformer_sac_estimation
2025-08-14T21:35:17.8054347Z 
2025-08-14T21:35:17.8055422Z Running distributed/tensor/test_xla_integration 1/1 ... [2025-08-14 21:35:17.805297]
2025-08-14T21:35:17.8056167Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:17.8058132Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_xla_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:17.805605]
2025-08-14T21:35:21.9757388Z 
2025-08-14T21:35:21.9759060Z distributed/tensor/test_xla_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_xla_integration_1.1_e98bb12485d46ca7_.log
2025-08-14T21:35:21.9762567Z Running 3 items in this shard: test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_replicate, test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_1d_shard, test/distributed/tensor/test_xla_integration.py::DTensorXLAIntegrationTest::test_xla_distribute_tensor_2d
2025-08-14T21:35:21.9764802Z 
2025-08-14T21:35:21.9765247Z Running distributed/checkpoint/test_dedup_tensors 1/1 ... [2025-08-14 21:35:21.976118]
2025-08-14T21:35:21.9766005Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:21.9767353Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dedup_tensors.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:21.976437]
2025-08-14T21:35:26.1964828Z 
2025-08-14T21:35:26.1966298Z distributed/checkpoint/test_dedup_tensors 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dedup_tensors_1.1_2edfa5b7e61ec00b_.log
2025-08-14T21:35:26.1968556Z Running 1 items in this shard: test/distributed/checkpoint/test_dedup_tensors.py::TestDedupTensor::test_dedup_shards
2025-08-14T21:35:26.1970300Z 
2025-08-14T21:35:26.1971313Z Running distributed/pipelining/test_unflatten 1/1 ... [2025-08-14 21:35:26.196936]
2025-08-14T21:35:26.1972040Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:26.1974971Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_unflatten.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:26.197245]
2025-08-14T21:35:35.0232878Z 
2025-08-14T21:35:35.0234175Z distributed/pipelining/test_unflatten 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_unflatten_1.1_10468f3ae2a2972c_.log
2025-08-14T21:35:35.0236038Z Running 1 items in this shard: test/distributed/pipelining/test_unflatten.py::UnflattenTestsCUDA::test_unflatten_cuda
2025-08-14T21:35:35.0236832Z 
2025-08-14T21:35:35.0239844Z Running distributed/checkpoint/test_format_utils 1/1 ... [2025-08-14 21:35:35.023747]
2025-08-14T21:35:35.0240598Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:35.0244770Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_format_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:35.024062]
2025-08-14T21:35:54.9141761Z 
2025-08-14T21:35:54.9143152Z distributed/checkpoint/test_format_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_format_utils_1.1_3882d1bcfd9e1114_.log
2025-08-14T21:35:54.9146125Z Running 3 items in this shard: test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_dcp_to_torch_save, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_online_torch_save_to_dcp, test/distributed/checkpoint/test_format_utils.py::TestFormatUtils::test_torch_save_to_dcp
2025-08-14T21:35:54.9148034Z 
2025-08-14T21:35:54.9148456Z Running distributed/tensor/test_dtensor_ops 1/1 ... [2025-08-14 21:35:54.914413]
2025-08-14T21:35:54.9149285Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:54.9154148Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:54.914726]
2025-08-14T21:36:00.0595260Z 
2025-08-14T21:36:00.0596592Z distributed/tensor/test_dtensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_ops_1.1_811cbed382e40702_.log
2025-08-14T21:36:00.0598072Z Running 0 items in this shard:
2025-08-14T21:36:00.0598398Z 
2025-08-14T21:36:00.0600998Z Running distributed/pipelining/test_microbatch 1/1 ... [2025-08-14 21:36:00.059899]
2025-08-14T21:36:00.0601767Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:36:00.0604640Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_microbatch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:36:00.060198]
2025-08-14T21:36:08.4356824Z 
2025-08-14T21:36:08.4358207Z distributed/pipelining/test_microbatch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_microbatch_1.1_b1ac0353b8962de6_.log
2025-08-14T21:36:08.4360717Z Running 2 items in this shard: test/distributed/pipelining/test_microbatch.py::MicrobatchTestsCUDA::test_chunk_spec_cuda, test/distributed/pipelining/test_microbatch.py::MicrobatchTestsCUDA::test_split_and_merge_cuda
2025-08-14T21:36:08.4362069Z 
2025-08-14T21:36:08.4362607Z Running distributed/tensor/test_api 1/1 ... [2025-08-14 21:36:08.436027]
2025-08-14T21:36:08.4364156Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:36:08.4366304Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:36:08.436348]
2025-08-14T21:37:10.0300152Z 
2025-08-14T21:37:10.0301540Z distributed/tensor/test_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_api_1.1_e51fb76ee2fb3651_.log
2025-08-14T21:37:10.0307616Z Running 8 items in this shard: test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_casting, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_input_fn_output_fn_warning, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_module_meta, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_errors, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_rank, test/distributed/tensor/test_api.py::DTensorAPITest::test_distribute_tensor_uneven_sharding
2025-08-14T21:37:10.0312599Z 
2025-08-14T21:37:10.0313121Z Running distributed/tensor/test_convolution_ops 1/1 ... [2025-08-14 21:37:10.030376]
2025-08-14T21:37:10.0313716Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:37:10.0314728Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_convolution_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:10.030685]
2025-08-14T21:38:10.4228744Z 
2025-08-14T21:38:10.4230143Z distributed/tensor/test_convolution_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_convolution_ops_1.1_1c014c304b01b585_.log
2025-08-14T21:38:10.4233910Z Running 3 items in this shard: test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_conv_backward_none_grad_inp, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_depthwise_convolution, test/distributed/tensor/test_convolution_ops.py::DistConvolutionOpsTest::test_downsampling_convolution
2025-08-14T21:38:10.4235607Z 
2025-08-14T21:38:10.4235957Z Running distributed/checkpoint/_experimental/test_checkpoint_reader 1/1 ... [2025-08-14 21:38:10.423328]
2025-08-14T21:38:10.4236510Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:38:10.4239562Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpoint_reader.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:38:10.423766]
2025-08-14T21:38:14.8441028Z 
2025-08-14T21:38:14.8442811Z distributed/checkpoint/_experimental/test_checkpoint_reader 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpoint_reader_1.1_f59e75291b176ca5_.log
2025-08-14T21:38:14.8450250Z Running 7 items in this shard: test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_partial_read, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_partial_read_different_dtypes, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_partial_read_missing_keys, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_read_checkpoint, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_read_nonexistent_checkpoint, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_read_with_kwargs, test/distributed/checkpoint/_experimental/test_checkpoint_reader.py::TestCheckpointReader::test_read_with_map_location
2025-08-14T21:38:14.8454278Z 
2025-08-14T21:38:14.8454656Z Running distributed/_composable/test_composability/test_pp_composability 1/1 ... [2025-08-14 21:38:14.844595]
2025-08-14T21:38:14.8455232Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:38:14.8456400Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_pp_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:38:14.845023]
2025-08-14T21:39:11.0809755Z 
2025-08-14T21:39:11.0811671Z distributed/_composable/test_composability/test_pp_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_pp_composability_1.1_1d3697a6356dc45b_.log
2025-08-14T21:39:11.0821751Z Running 11 items in this shard: test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass0_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass0_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass1_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass1_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass2_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass2_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass3_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass3_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass4_bfloat16, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_3d_with_tp_dp_pp_ScheduleClass4_float32, test/distributed/_composable/test_composability/test_pp_composability.py::ComposabilityTest::test_pp_and_dcp
2025-08-14T21:39:11.0827170Z 
2025-08-14T21:39:11.0827419Z Running distributed/tensor/test_tensor_ops 1/1 ... [2025-08-14 21:39:11.081505]
2025-08-14T21:39:11.0827857Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:39:11.0828793Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:39:11.081817]
2025-08-14T21:42:37.8692415Z 
2025-08-14T21:42:37.8693814Z distributed/tensor/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_tensor_ops_1.1_475905be94123610_.log
2025-08-14T21:42:37.8706589Z Running 29 items in this shard: test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_aten_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_clone, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_contiguous, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_copy_, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_detach, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_dtensor_dtype_conversion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_empty_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_equal, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_fill_inplace_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_full_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_gather, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index_put_scalar, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_index_put_tensor, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_inplace_op, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_empty_strided, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_new_full, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_ones_like_partial_sum, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_op_out_variant, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_scatter, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_slice, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_split_on_partial, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_stack, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_where_type_promotion, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zero_inplace, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like, test/distributed/tensor/test_tensor_ops.py::DistTensorOpsTest::test_zeros_like_partial_sum
2025-08-14T21:42:37.8715404Z 
2025-08-14T21:42:37.8715662Z Running distributed/_tools/test_fsdp2_mem_tracker 1/1 ... [2025-08-14 21:42:37.869886]
2025-08-14T21:42:37.8716105Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:42:37.8717048Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_fsdp2_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:42:37.870199]
2025-08-14T21:43:24.3364935Z 
2025-08-14T21:43:24.3366187Z distributed/_tools/test_fsdp2_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_fsdp2_mem_tracker_1.1_971dc702c3eb34f3_.log
2025-08-14T21:43:24.3369987Z Running 3 items in this shard: test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_multi_group_eager, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCore::test_tracker_non_root_forward_backward, test/distributed/_tools/test_fsdp2_mem_tracker.py::TestTrackerFullyShard1DTrainingCompose::test_tracker_with_activation_checkpointing
2025-08-14T21:43:24.3373140Z 
2025-08-14T21:43:24.3373437Z Running distributed/checkpoint/test_traverse 1/1 ... [2025-08-14 21:43:24.336675]
2025-08-14T21:43:24.3373927Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:43:24.3374938Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_traverse.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:24.336986]
2025-08-14T21:43:28.5570259Z 
2025-08-14T21:43:28.5571841Z distributed/checkpoint/test_traverse 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_traverse_1.1_05aa8fd4f50b7a0c_.log
2025-08-14T21:43:28.5575894Z Running 7 items in this shard: test/distributed/checkpoint/test_traverse.py::TestTraverse::test_get_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_set_element, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_doesnt_ignore_intermediate_collections, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_dict, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_nested_list, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_shallow, test/distributed/checkpoint/test_traverse.py::TestTraverse::test_traverse_with_ordered_dict
2025-08-14T21:43:28.5579348Z 
2025-08-14T21:43:28.5579643Z Running distributed/checkpoint/test_fsspec 1/1 ... [2025-08-14 21:43:28.557423]
2025-08-14T21:43:28.5580075Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:43:28.5581063Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsspec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:28.557731]
2025-08-14T21:43:47.5964296Z 
2025-08-14T21:43:47.5965656Z distributed/checkpoint/test_fsspec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsspec_1.1_e15af4ae76d2fe05_.log
2025-08-14T21:43:47.5969167Z Running 3 items in this shard: test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_fsspec, test/distributed/checkpoint/test_fsspec.py::TestFSSpec::test_overwrite, test/distributed/checkpoint/test_fsspec.py::TestFileSystem::test_remove_on_fail
2025-08-14T21:43:47.5971082Z 
2025-08-14T21:43:47.5971587Z Running distributed/tensor/test_redistribute 1/1 ... [2025-08-14 21:43:47.596726]
2025-08-14T21:43:47.5972449Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:43:47.5973711Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_redistribute.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:47.597036]
2025-08-14T21:46:18.1507429Z 
2025-08-14T21:46:18.1509060Z distributed/tensor/test_redistribute 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_redistribute_1.1_82f2c2d609046587_.log
2025-08-14T21:46:18.1523443Z Running 21 items in this shard: test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_replicate_forward_backward_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_partial_to_shard_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_negative_shard_dim, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_shard_dim_change_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_redistribute_uneven_sharding, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_local_partial_grad_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_partial, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_replicate_forward_backward_datatype_conversion, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_replicate_to_shard_forward_backward, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_dim_alltoall_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_dim_alltoall_float32, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward_complex64, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward_datatype_conversion, test/distributed/tensor/test_redistribute.py::RedistributeTest::test_shard_to_replicate_forward_backward_float32, test/distributed/tensor/test_redistribute.py::MultiDimRedistributeTest::test_multi_dim_mesh, test/distributed/tensor/test_redistribute.py::MultiDimRedistributeTest::test_redistribute_shard_dim_multi_dim_mesh
2025-08-14T21:46:18.1532028Z 
2025-08-14T21:46:18.1532256Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T21:46:18.1532669Z Uploading artifacts took 0.00 seconds
2025-08-14T21:46:18.1533101Z Running distributed/checkpoint/test_fsdp_optim_state 1/1 ... [2025-08-14 21:46:18.151205]
2025-08-14T21:46:18.1533539Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:46:18.1534484Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:46:18.151518]
2025-08-14T21:46:38.3419977Z 
2025-08-14T21:46:38.3422153Z distributed/checkpoint/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_optim_state_1.1_695d909228bab4b7_.log
2025-08-14T21:46:38.3425127Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_False, test/distributed/checkpoint/test_fsdp_optim_state.py::FsdpOptimStateCheckpoint::test_load_sharded_optimizer_state_dict_pass_planner_True
2025-08-14T21:46:38.3427033Z 
2025-08-14T21:46:38.3427489Z Running distributed/checkpoint/test_dtensor_resharding 1/1 ... [2025-08-14 21:46:38.342474]
2025-08-14T21:46:38.3428289Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:46:38.3429985Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_resharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:46:38.342788]
2025-08-14T21:47:59.4599281Z 
2025-08-14T21:47:59.4600736Z distributed/checkpoint/test_dtensor_resharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_resharding_1.1_823b0f7354a17cfd_.log
2025-08-14T21:47:59.4607830Z Running 7 items in this shard: test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions0, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions1, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change_extensions2, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_dtensor_resharding.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard
2025-08-14T21:47:59.4611236Z 
2025-08-14T21:47:59.4611472Z Running distributed/_tools/test_mem_tracker 1/1 ... [2025-08-14 21:47:59.460181]
2025-08-14T21:47:59.4611891Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:47:59.4612816Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_mem_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:47:59.460512]
2025-08-14T21:48:06.1338858Z 
2025-08-14T21:48:06.1340241Z distributed/_tools/test_mem_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_mem_tracker_1.1_ec1f5aa3d05c3fa9_.log
2025-08-14T21:48:06.1343198Z Running 3 items in this shard: test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_accelerator_tracker_equivalence, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_attribution, test/distributed/_tools/test_mem_tracker.py::TestMemTracker::test_tracker_with_activation_checkpointing
2025-08-14T21:48:06.1345298Z 
2025-08-14T21:48:06.1345712Z Running distributed/pipelining/test_schedule 1/1 ... [2025-08-14 21:48:06.134088]
2025-08-14T21:48:06.1346449Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:48:06.1348066Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_schedule.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:48:06.134410]
2025-08-14T21:48:50.9069276Z 
2025-08-14T21:48:50.9071347Z distributed/pipelining/test_schedule 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_schedule_1.1_1634d5140102730f_.log
2025-08-14T21:48:50.9086712Z Running 33 items in this shard: test/distributed/pipelining/test_schedule.py::ScheduleTest::test_get_schedule_class, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass0, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass1, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass2, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass3, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_schedule_with_single_stage_ScheduleClass4, test/distributed/pipelining/test_schedule.py::ScheduleTest::test_zero_bubble_schedule_errors_with_compile, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_flex_and_zero_bubble_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_for_v_schedules_ScheduleClass0, test/distributed/pipelining/test_schedule.py::TestSchedulePlan::test_pipeline_order_for_v_schedules_ScheduleClass1, test/distributed/pipelining/test_schedule.py::TestScheduleCsv::test_csv_compare_ScheduleClass0_csv_name_dualpipev_4rank_10mb, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref2, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref3, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref4, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref5, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref6, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_action_parse_action_str_and_ref7, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_csv_csv_name_zb1p_2rank_2stagep, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_split_b_w, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_grad_with_v_schedule, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_merge_bw_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info0, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_send_recv_test_info1, test/distributed/pipelining/test_schedule.py::TestScheduleLowering::test_unshard_reshard_test_info0, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_action, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_invalid_schedule_missing_rank, test/distributed/pipelining/test_schedule.py::TestValidateSchedule::test_valid_schedule, test/distributed/pipelining/test_schedule.py::ScheduleUtilTests::test_generate_stage_to_rank_mapping
2025-08-14T21:48:50.9099015Z 
2025-08-14T21:48:50.9099268Z Running distributed/checkpoint/test_save_load_api 1/1 ... [2025-08-14 21:48:50.907268]
2025-08-14T21:48:50.9099704Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:48:50.9100806Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_save_load_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:48:50.907578]
2025-08-14T21:49:07.8937198Z 
2025-08-14T21:49:07.8938617Z distributed/checkpoint/test_save_load_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_save_load_api_1.1_ce29323c80d42cd8_.log
2025-08-14T21:49:07.8941047Z Running 2 items in this shard: test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_assert_same_keys, test/distributed/checkpoint/test_save_load_api.py::TestSaveAndLoadAPI::test_auto_detect
2025-08-14T21:49:07.8942309Z 
2025-08-14T21:49:07.8942787Z Running distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 ... [2025-08-14 21:49:07.893951]
2025-08-14T21:49:07.8943557Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:49:07.8945904Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_e2e_save_and_load.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:49:07.894272]
2025-08-14T21:51:29.1895960Z 
2025-08-14T21:51:29.1897525Z distributed/checkpoint/e2e/test_e2e_save_and_load 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_e2e_save_and_load_1.1_1863c9838bb70810_.log
2025-08-14T21:51:29.1907431Z Running 19 items in this shard: test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_different_ordered_state_dict_keys, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_False_async_checkpointer_type0_zoc_False, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_False_async_checkpointer_type2_zoc_False, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_False_async_checkpointer_type4_zoc_True, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_False_async_checkpointer_type5_zoc_True, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_True_async_checkpointer_type1_zoc_False, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_async_cached_cache_staged_state_dict_True_async_checkpointer_type3_zoc_False, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_False_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type0, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type1, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_e2e_compile_True_model_type2, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_no_dist, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_overwrite, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_partial_load, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestE2ESaveAndLoad::test_stateful_and_non_stateful_loads, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestNoCPU::test_no_cpu, test/distributed/checkpoint/e2e/test_e2e_save_and_load.py::TestInitStateDict::test_init_state_dict
2025-08-14T21:51:29.1922340Z 
2025-08-14T21:51:29.1922558Z Running distributed/tensor/test_utils 1/1 ... [2025-08-14 21:51:29.190209]
2025-08-14T21:51:29.1922959Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:51:29.1924066Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:51:29.190518]
2025-08-14T21:52:51.4095448Z 
2025-08-14T21:52:51.4096660Z distributed/tensor/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_utils_1.1_98a8a830380d7f81_.log
2025-08-14T21:52:51.4106605Z Running 16 items in this shard: test/distributed/tensor/test_utils.py::LocalTest::test_compute_local_shape_and_global_offset_uneven, test/distributed/tensor/test_utils.py::LocalTest::test_explicit_order_placements, test/distributed/tensor/test_utils.py::UtilTest::test_compute_global_tensor_shape_1D, test/distributed/tensor/test_utils.py::UtilTest::test_compute_global_tensor_shape_1D_invalid_shape, test/distributed/tensor/test_utils.py::UtilTest::test_compute_global_tensor_shape_failure_2D, test/distributed/tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_1D, test/distributed/tensor/test_utils.py::UtilTest::test_compute_local_shape_and_global_offset_2D, test/distributed/tensor/test_utils.py::UtilTest::test_fsdp_tp_meta_compute, test/distributed/tensor/test_utils.py::UtilTest::test_hsdp_tp_meta_compute, test/distributed/tensor/test_utils.py::UtilTest::test_strided_sharding_assumption_in_meta_compute, test/distributed/tensor/test_utils.py::UtilTest::test_uneven_fsdp_tp_meta_compute, test/distributed/tensor/test_utils.py::TestStridedSharding::test_1d_mesh_strided_sharding, test/distributed/tensor/test_utils.py::TestStridedSharding::test_2d_mesh_2d_tensor_strided_sharding, test/distributed/tensor/test_utils.py::TestStridedSharding::test_2d_mesh_strided_sharding, test/distributed/tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp1_tp_2d_dtensor_local_shards_and_offsets, test/distributed/tensor/test_utils.py::Test2DStridedLocalShard::test_fsdp2_tp_2d_dtensor_local_shards_and_offsets
2025-08-14T21:52:51.4112691Z 
2025-08-14T21:52:51.4112996Z Running distributed/optim/test_apply_optimizer_in_backward 1/1 ... [2025-08-14 21:52:51.409967]
2025-08-14T21:52:51.4113528Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:52:51.4114632Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_apply_optimizer_in_backward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:52:51.410279]
2025-08-14T21:52:53.2335052Z 
2025-08-14T21:52:53.2335907Z distributed/optim/test_apply_optimizer_in_backward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_apply_optimizer_in_backward_1.1_e87b4a4707a79d96_.log
2025-08-14T21:52:53.2337514Z 
2025-08-14T21:52:53.2340054Z Running distributed/_composable/test_replicate_with_compiler 1/1 ... [2025-08-14 21:52:53.233814]
2025-08-14T21:52:53.2340583Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:52:53.2342668Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_compiler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:52:53.234093]
2025-08-14T21:55:27.8309106Z 
2025-08-14T21:55:27.8310702Z distributed/_composable/test_replicate_with_compiler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_compiler_1.1_94c9956e06bb0d13_.log
2025-08-14T21:55:27.8317213Z Running 10 items in this shard: test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_coalesced_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_bucketing_concat_op, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_backward_only, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_bf16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_cpu_no_sync, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_fp16, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu, test/distributed/_composable/test_replicate_with_compiler.py::ReplicateTest::test_compile_gpu_ac, test/distributed/_composable/test_replicate_with_compiler.py::DDP_TP_Test::test_ddp_tp
2025-08-14T21:55:27.8320915Z 
2025-08-14T21:55:27.8321223Z Running distributed/_composable/fsdp/test_fully_shard_compile 1/1 ... [2025-08-14 21:55:27.831172]
2025-08-14T21:55:27.8321725Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:55:27.8322910Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:55:27.831520]
2025-08-14T21:59:03.7259138Z 
2025-08-14T21:59:03.7260831Z distributed/_composable/fsdp/test_fully_shard_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_compile_1.1_0d9eeb11e93aab02_.log
2025-08-14T21:59:03.7275009Z Running 18 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompileCompute::test_disable_compiling_hooks, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_compiled_autograd_ctx, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_recompiles_on_fsdp_layers, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_dynamo_trace_use_training_state, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_nested_fully_shard_backend_inductor_fullgraph_True_graph_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_simple_mlp_fullgraph_backend_inductor, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_trace_fsdp_copy_, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_aot_eager_decomp_partition, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_False, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True, test/distributed/_composable/fsdp/test_fully_shard_compile.py::TestFullyShardCompile::test_transformer_backend_inductor_fullgraph_True_graph_partition
2025-08-14T21:59:03.7284403Z 
2025-08-14T21:59:03.7284887Z Running distributed/checkpoint/test_fsdp_model_state 1/1 ... [2025-08-14 21:59:03.726209]
2025-08-14T21:59:03.7285349Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:03.7286316Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_model_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:03.726525]
2025-08-14T21:59:24.3179349Z 
2025-08-14T21:59:24.3180673Z distributed/checkpoint/test_fsdp_model_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_model_state_1.1_7409be4eef9734ca_.log
2025-08-14T21:59:24.3183469Z Running 2 items in this shard: test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_no_resharding, test/distributed/checkpoint/test_fsdp_model_state.py::FsdpModelStateCheckpoint::test_fsdp_model_state_with_resharding
2025-08-14T21:59:24.3185184Z 
2025-08-14T21:59:24.3185592Z Running distributed/fsdp/test_shard_utils 1/1 ... [2025-08-14 21:59:24.318290]
2025-08-14T21:59:24.3186305Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:24.3188117Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_shard_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:24.318603]
2025-08-14T21:59:40.9549990Z 
2025-08-14T21:59:40.9551385Z distributed/fsdp/test_shard_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_shard_utils_1.1_c92b0a82855818ab_.log
2025-08-14T21:59:40.9554337Z Running 2 items in this shard: test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributed::test_create_chunk_sharded_tensor, test/distributed/fsdp/test_shard_utils.py::TestShardUtilsDistributedDTensor::test_create_chunk_dtensor
2025-08-14T21:59:40.9556042Z 
2025-08-14T21:59:40.9560745Z Running distributed/elastic/test_control_plane 1/1 ... [2025-08-14 21:59:40.955517]
2025-08-14T21:59:40.9561578Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:40.9563041Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/test_control_plane.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:40.955828]
2025-08-14T21:59:45.6266775Z 
2025-08-14T21:59:45.6268145Z distributed/elastic/test_control_plane 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.test_control_plane_1.1_03e11ba81d144293_.log
2025-08-14T21:59:45.6274566Z Running 9 items in this shard: test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_json, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_nccl_trace_pickle_with_params, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_dump_traceback, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_names, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_get_handler_nonexistant, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_run_handler, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_tcp, test/distributed/elastic/test_control_plane.py::WorkerServerTest::test_worker_server
2025-08-14T21:59:45.6280106Z 
2025-08-14T21:59:45.6280449Z Running distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 ... [2025-08-14 21:59:45.627108]
2025-08-14T21:59:45.6280958Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:45.6282381Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:45.627521]
2025-08-14T22:00:01.2628254Z 
2025-08-14T22:00:01.2630141Z distributed/_composable/fsdp/test_fully_shard_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_grad_scaler_1.1_f5d6c1bce3cf4507_.log
2025-08-14T22:00:01.2632505Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_grad_scaler.py::TestFullyShardGradientScaler::test_gradient_scaler
2025-08-14T22:00:01.2633453Z 
2025-08-14T22:00:01.2634063Z Running distributed/checkpoint/_experimental/test_checkpoint_process 1/1 ... [2025-08-14 22:00:01.263016]
2025-08-14T22:00:01.2635017Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:00:01.2637150Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpoint_process.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:00:01.263323]
2025-08-14T22:00:37.8758490Z 
2025-08-14T22:00:37.8760446Z distributed/checkpoint/_experimental/test_checkpoint_process 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpoint_process_1.1_f351cce671be2c05_.log
2025-08-14T22:00:37.8772291Z Running 15 items in this shard: test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_request_type_enum, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_worker_request, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestRequestTypes::test_worker_response, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcessConfig::test_custom_options, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcessConfig::test_default_options, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_process_initialization, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_future_state_dict, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_sync_state_dict, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_checkpoint_write_with_kwargs, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_communication_error_handling, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_forced_termination, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_graceful_termination, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_shared_memory_tensor_ipc, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_subprocess_initialization_failure, test/distributed/checkpoint/_experimental/test_checkpoint_process.py::TestCheckpointProcess::test_subprocess_initialization_timeout
2025-08-14T22:00:37.8780531Z 
2025-08-14T22:00:37.8780825Z Running distributed/tensor/parallel/test_parallelize_api 1/1 ... [2025-08-14 22:00:37.876319]
2025-08-14T22:00:37.8781316Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:00:37.8782304Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:00:37.876635]
2025-08-14T22:02:45.5542629Z 
2025-08-14T22:02:45.5544858Z distributed/tensor/parallel/test_parallelize_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_parallelize_api_1.1_8fe72d84b998e439_.log
2025-08-14T22:02:45.5555856Z Running 16 items in this shard: test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_empty_plan, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_col_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_linear_row_wise_parallel, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_mlp_with_module_api_nested, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_multi_wildcard, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_src_data_rank, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_digit, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_no_match, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_question, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_root_module, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_parallelize_module_with_star, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_input_output, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_prepare_module_output, test/distributed/tensor/parallel/test_parallelize_api.py::TensorParallelAPITests::test_under_devicemesh_context
2025-08-14T22:02:45.5562756Z 
2025-08-14T22:02:45.5563009Z Running distributed/checkpoint/test_checkpoint 1/1 ... [2025-08-14 22:02:45.554666]
2025-08-14T22:02:45.5563449Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:02:45.5564378Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:02:45.554982]
2025-08-14T22:03:39.3392235Z 
2025-08-14T22:03:39.3393898Z distributed/checkpoint/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_checkpoint_1.1_1e2b3155b1487350_.log
2025-08-14T22:03:39.3401031Z Running 8 items in this shard: test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_default_metadata, test/distributed/checkpoint/test_checkpoint.py::TestDistributedCheckpointing::test_tensor_metadata_with_missing_rank_spec, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_reader_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_dummy_writer_works, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_load_error_handling_no_dist, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling, test/distributed/checkpoint/test_checkpoint.py::TestDistributedFailure::test_save_error_handling_no_dist
2025-08-14T22:03:39.3404235Z 
2025-08-14T22:03:39.3404453Z Running distributed/test_collective_utils 1/1 ... [2025-08-14 22:03:39.339740]
2025-08-14T22:03:39.3404870Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:03:39.3405994Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_collective_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:03:39.340048]
2025-08-14T22:04:16.1522294Z 
2025-08-14T22:04:16.1523671Z distributed/test_collective_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_collective_utils_1.1_594587dd745f46df_.log
2025-08-14T22:04:16.1530145Z Running 8 items in this shard: test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_all_gather_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_no_pg, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_broadcast_result_raises_exceptions_from_func, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cpu, test/distributed/test_collective_utils.py::TestCollectiveUtils::test_check_rng_sync_device_cuda
2025-08-14T22:04:16.1533144Z 
2025-08-14T22:04:16.1533365Z Running distributed/checkpoint/test_utils 1/1 ... [2025-08-14 22:04:16.152551]
2025-08-14T22:04:16.1533775Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:04:16.1534698Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:04:16.152853]
2025-08-14T22:04:56.1697210Z 
2025-08-14T22:04:56.1698559Z distributed/checkpoint/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_utils_1.1_c870dbf366c43342_.log
2025-08-14T22:04:56.1706546Z Running 16 items in this shard: test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_dcp_logger, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_flat_data, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_equals, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_index_hint_ignored_on_hash, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_init_convert_offset, test/distributed/checkpoint/test_utils.py::TestMedatadaIndex::test_sharded_tensor_lookup, test/distributed/checkpoint/test_utils.py::TestReaderView::testAllRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testLongReadinto, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortRead, test/distributed/checkpoint/test_utils.py::TestReaderView::testShortReadinto, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_barrier, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_broadcast_object_global_local_mismatch, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_broadcast_object_with_nonzero_coordinator, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_gather_object, test/distributed/checkpoint/test_utils.py::TestDistWrapper::test_scatter_object
2025-08-14T22:04:56.1720605Z 
2025-08-14T22:04:56.1720856Z Running distributed/fsdp/test_fsdp_state_dict 2/2 ... [2025-08-14 22:04:56.170063]
2025-08-14T22:04:56.1721277Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:04:56.1722330Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:04:56.170381]
2025-08-14T22:16:14.2504897Z 
2025-08-14T22:16:14.2506919Z distributed/fsdp/test_fsdp_state_dict 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_2.2_d05be69bfac023c5_.log
2025-08-14T22:16:14.2574699Z Running 101 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_local_state_dict_with_empty_ranks, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_sharded_load_multi_backend_pg, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_shared_module_and_shared_parameter, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_local_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_sharded_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_skip_module_state_dict_type_state_dict_double_nest_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_sharded_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_manual_ac_wrapper_state_dict_type_state_dict_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_world_size_one, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_wrong_state_dict_config
2025-08-14T22:16:14.2637037Z 
2025-08-14T22:16:14.2637261Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T22:16:14.2637674Z Uploading artifacts took 0.00 seconds
2025-08-14T22:16:14.2638259Z Running distributed/checkpoint/test_state_dict_stager 1/1 ... [2025-08-14 22:16:14.252235]
2025-08-14T22:16:14.2638708Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:16:14.2639664Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_stager.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:16:14.252540]
2025-08-14T22:16:53.7677112Z 
2025-08-14T22:16:53.7678661Z distributed/checkpoint/test_state_dict_stager 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_stager_1.1_02ca26d782586fc3_.log
2025-08-14T22:16:53.7691418Z Running 14 items in this shard: test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_caching, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_complex_storage_sharing, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_cpu_storage_independence, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_dataclasses, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_different_dtypes, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_empty_tensors, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_tensor_attrs, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_tensor_pinned_and_shared, test/distributed/checkpoint/test_state_dict_stager.py::TestStateDictStager::test_views, test/distributed/checkpoint/test_state_dict_stager.py::TestDTensorStateDictStager::test_dtensor, test/distributed/checkpoint/test_state_dict_stager.py::TestReplicationStager::test_replication_basic, test/distributed/checkpoint/test_state_dict_stager.py::TestReplicationStager::test_replication_dtensors, test/distributed/checkpoint/test_state_dict_stager.py::TestReplicationStager::test_replication_persistence, test/distributed/checkpoint/test_state_dict_stager.py::TestReplicationStager::test_replication_sharded_tensors
2025-08-14T22:16:53.7697496Z 
2025-08-14T22:16:53.7697699Z Running distributed/fsdp/test_utils 1/1 ... [2025-08-14 22:16:53.768234]
2025-08-14T22:16:53.7698094Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:16:53.7698999Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:16:53.768548]
2025-08-14T22:16:58.2891223Z 
2025-08-14T22:16:58.2892685Z distributed/fsdp/test_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_utils_1.1_0ff662f30b6f2c33_.log
2025-08-14T22:16:58.2897783Z Running 5 items in this shard: test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_cpu_cuda_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list0_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_apply_to_tensors_device_list1_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_packed_sequence_cuda, test/distributed/fsdp/test_utils.py::TestUtilsCUDA::test_replace_by_prefix_cuda
2025-08-14T22:16:58.2900600Z 
2025-08-14T22:16:58.2900990Z Running distributed/fsdp/test_fsdp_overlap 1/1 ... [2025-08-14 22:16:58.289529]
2025-08-14T22:16:58.2901736Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:16:58.2903094Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:16:58.289827]
2025-08-14T22:17:25.3381573Z 
2025-08-14T22:17:25.3383087Z distributed/fsdp/test_fsdp_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_overlap_1.1_4d872913d4c859d9_.log
2025-08-14T22:17:25.3384549Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_overlap.py::TestForwardOverlapWorldSizeOneCUDA::test_forward_overlap_cuda
2025-08-14T22:17:25.3385211Z 
2025-08-14T22:17:25.3387344Z Running distributed/fsdp/test_checkpoint_wrapper 1/1 ... [2025-08-14 22:17:25.338523]
2025-08-14T22:17:25.3387920Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:17:25.3390454Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:17:25.338833]
2025-08-14T22:17:30.5604428Z 
2025-08-14T22:17:30.5605866Z distributed/fsdp/test_checkpoint_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_checkpoint_wrapper_1.1_c017e8287cda3bdf_.log
2025-08-14T22:17:30.5612336Z Running 8 items in this shard: test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_apply_activation_checkpointing, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_args_kwargs, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_cpu_offload, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_kwarg_support, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_checkpoint_wrapper_parity, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_forward_missing_attributes, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_fqn, test/distributed/fsdp/test_checkpoint_wrapper.py::CheckpointWrapperTest::test_load_activation_checkpointed_module
2025-08-14T22:17:30.5616986Z 
2025-08-14T22:17:30.5617251Z Running distributed/fsdp/test_fsdp_flatten_params 1/1 ... [2025-08-14 22:17:30.560712]
2025-08-14T22:17:30.5617704Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:17:30.5618684Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_flatten_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:17:30.561010]
2025-08-14T22:19:00.6918032Z 
2025-08-14T22:19:00.6919618Z distributed/fsdp/test_fsdp_flatten_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_flatten_params_1.1_60ee84df82e788b8_.log
2025-08-14T22:19:00.6931730Z Running 14 items in this shard: test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_empty_module, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_full_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_aligned_mixed_precision, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_unaligned, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format0, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flat_param_shard_metadata_with_memory_format_memory_format1, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_flatten_nothing, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_numel_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_output_without_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_partial_flattening, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_pnorm_after_step_with_shared_params, test/distributed/fsdp/test_fsdp_flatten_params.py::TestFlattenParams::test_writeback_orig_params_no_shard
2025-08-14T22:19:00.6939086Z 
2025-08-14T22:19:00.6939393Z Running distributed/optim/test_zero_redundancy_optimizer 1/1 ... [2025-08-14 22:19:00.692098]
2025-08-14T22:19:00.6939910Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:19:00.6941011Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_zero_redundancy_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:19:00.692431]
2025-08-14T22:23:17.0429451Z 
2025-08-14T22:23:17.0430899Z distributed/optim/test_zero_redundancy_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_zero_redundancy_optimizer_1.1_7a9fa1e84c9be96c_.log
2025-08-14T22:23:17.0462256Z Running 42 items in this shard: test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_constructor, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_same_dense_param_type, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_state_dict, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_extra_inner_key, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_with_kwargs, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_step_without_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerSingleRank::test_zero_grad, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_add_param_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_collect_shards, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_False_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_False_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_False_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_ddp_zero_overlap_use_gpu_True_use_interleaved_hook_True_gradient_as_bucket_view_True_static_graph_True_shard_buckets_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_AdamW_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_Adam_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_local_optimizer_parity_optimizer_class_str_SGD_maximize_True, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_lr_scheduler, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_multiple_param_groups, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_nondefault_process_group, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_sharding, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_step_with_closure, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_cpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_join_gpu, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_False, test/distributed/optim/test_zero_redundancy_optimizer.py::TestZeroRedundancyOptimizerDistributed::test_zero_model_parallel_parameters_as_bucket_view_True
2025-08-14T22:23:17.0487209Z 
2025-08-14T22:23:17.0487461Z Running distributed/fsdp/test_fsdp_ignored_modules 1/1 ... [2025-08-14 22:23:17.043545]
2025-08-14T22:23:17.0487904Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:23:17.0488852Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_ignored_modules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:23:17.043859]
2025-08-14T22:24:17.5356008Z 
2025-08-14T22:24:17.5357469Z distributed/fsdp/test_fsdp_ignored_modules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_ignored_modules_1.1_ef9abfc7917a3e01_.log
2025-08-14T22:24:17.5364517Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_diff_ignored_modules_across_ranks, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_invalid, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_nested, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_False, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_not_under_wrapped_root_ignore_modules_True, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_modules_transformer, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_auto_wrap, test/distributed/fsdp/test_fsdp_ignored_modules.py::TestFSDPIgnoredModules::test_ignored_states_check
2025-08-14T22:24:17.5367916Z 
2025-08-14T22:24:17.5368104Z Running distributed/test_nccl 1/1 ... [2025-08-14 22:24:17.535771]
2025-08-14T22:24:17.5368484Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:24:17.5369377Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:24:17.536089]
2025-08-14T22:24:34.4224878Z 
2025-08-14T22:24:34.4225757Z distributed/test_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nccl_1.1_a9aa63db35f03fe8_.log
2025-08-14T22:24:34.4231208Z Running 15 items in this shard: test/distributed/test_nccl.py::NCCLSymmetricMemoryTest::test_nccl_symmem_alloc, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_gather_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_all_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e4m3fnuz, test/distributed/test_nccl.py::TestNCCLCUDA::test_broadcast_cuda_float8_e5m2fnuz, test/distributed/test_nccl.py::TestNCCLCUDA::test_collective_errors_cuda, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_bfloat16, test/distributed/test_nccl.py::TestNCCLCUDA::test_reduce_scatter_cuda_float32, test/distributed/test_nccl.py::TestNCCLCUDA::test_unique_id_cuda
2025-08-14T22:24:34.4236453Z 
2025-08-14T22:24:34.4236719Z Running distributed/fsdp/test_fsdp_checkpoint 1/1 ... [2025-08-14 22:24:34.422657]
2025-08-14T22:24:34.4237196Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:24:34.4238478Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:24:34.422977]
2025-08-14T22:27:03.9799869Z 
2025-08-14T22:27:03.9801373Z distributed/fsdp/test_fsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_checkpoint_1.1_cd41d0d5cbd6a9dd_.log
2025-08-14T22:27:03.9817591Z Running 17 items in this shard: test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_basic_checkpoint_end_to_end_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload0_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpoint::test_checkpoint_fsdp_wrapping_cpu_offload1_offload_activations_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_checkpoint.py::TestFSDPCheckpointSubmoduleCUDA::test_checkpoint_submodule_use_reentrant_False_cuda
2025-08-14T22:27:03.9827521Z 
2025-08-14T22:27:03.9827752Z Running distributed/test_distributed_spawn 2/8 ... [2025-08-14 22:27:03.980662]
2025-08-14T22:27:03.9828215Z MPI not available -- MPI backend tests will be skipped
2025-08-14T22:27:03.9828635Z Running distributed tests for the test backend with env init_method
2025-08-14T22:27:03.9829011Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:27:03.9829980Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:27:03.982821]
2025-08-14T22:27:07.9870701Z 
2025-08-14T22:27:07.9872702Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_cf7ebc138665e4f1_.log
2025-08-14T22:27:07.9874205Z Running 0 items in this shard:
2025-08-14T22:27:07.9874571Z 
2025-08-14T22:27:07.9880678Z Running distributed tests for the test backend with file init_method
2025-08-14T22:27:07.9881921Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:27:07.9885524Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:27:07.988368]
2025-08-14T22:27:11.9946908Z 
2025-08-14T22:27:11.9948029Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_6ae96fd3e576b2b2_.log
2025-08-14T22:27:11.9949407Z Running 0 items in this shard:
2025-08-14T22:27:11.9949759Z 
2025-08-14T22:27:11.9955974Z Running distributed tests for the nccl backend with env init_method
2025-08-14T22:27:11.9957745Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:27:11.9960819Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:27:11.995904]
2025-08-14T22:32:36.6205103Z 
2025-08-14T22:32:36.6206444Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_8982c8042ee162c8_.log
2025-08-14T22:32:36.6223427Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:32:36.6237706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2025-08-14T22:32:36.6238742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2025-08-14T22:32:36.6239678Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream
2025-08-14T22:32:36.6240629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync
2025-08-14T22:32:36.6241512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2025-08-14T22:32:36.6242479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup
2025-08-14T22:32:36.6243378Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min
2025-08-14T22:32:36.6244496Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-08-14T22:32:36.6245379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-08-14T22:32:36.6246148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2025-08-14T22:32:36.6246940Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group
2025-08-14T22:32:36.6247742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2025-08-14T22:32:36.6248547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group
2025-08-14T22:32:36.6249358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2025-08-14T22:32:36.6250444Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger
2025-08-14T22:32:36.6251382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device
2025-08-14T22:32:36.6252159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2025-08-14T22:32:36.6252968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-08-14T22:32:36.6253854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error
2025-08-14T22:32:36.6254817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params
2025-08-14T22:32:36.6255766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph
2025-08-14T22:32:36.6256650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-08-14T22:32:36.6257494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-08-14T22:32:36.6258305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-08-14T22:32:36.6259072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2025-08-14T22:32:36.6259859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-08-14T22:32:36.6260620Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-08-14T22:32:36.6261418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler
2025-08-14T22:32:36.6262376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2025-08-14T22:32:36.6263359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity
2025-08-14T22:32:36.6264323Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2025-08-14T22:32:36.6265250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min
2025-08-14T22:32:36.6266063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-08-14T22:32:36.6266983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min
2025-08-14T22:32:36.6267731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda
2025-08-14T22:32:36.6268514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-08-14T22:32:36.6269325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2025-08-14T22:32:36.6270156Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:32:36.6270617Z 
2025-08-14T22:32:36.6270807Z Running distributed tests for the nccl backend with file init_method
2025-08-14T22:32:36.6271191Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:32:36.6272288Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:32:36.622295]
2025-08-14T22:38:02.1849894Z 
2025-08-14T22:38:02.1851283Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_6880ae3841337f15_.log
2025-08-14T22:38:02.1869998Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:38:02.1884565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2025-08-14T22:38:02.1885602Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2025-08-14T22:38:02.1886543Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream
2025-08-14T22:38:02.1887502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync
2025-08-14T22:38:02.1888382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2025-08-14T22:38:02.1889245Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup
2025-08-14T22:38:02.1890138Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min
2025-08-14T22:38:02.1891078Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-08-14T22:38:02.1891951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-08-14T22:38:02.1892717Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2025-08-14T22:38:02.1893520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group
2025-08-14T22:38:02.1894322Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2025-08-14T22:38:02.1895115Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group
2025-08-14T22:38:02.1895911Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2025-08-14T22:38:02.1896832Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger
2025-08-14T22:38:02.1897750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device
2025-08-14T22:38:02.1898707Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2025-08-14T22:38:02.1899516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-08-14T22:38:02.1900394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error
2025-08-14T22:38:02.1901351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params
2025-08-14T22:38:02.1902298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph
2025-08-14T22:38:02.1903175Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-08-14T22:38:02.1904011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-08-14T22:38:02.1904974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-08-14T22:38:02.1905743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2025-08-14T22:38:02.1906507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-08-14T22:38:02.1907263Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-08-14T22:38:02.1908049Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler
2025-08-14T22:38:02.1908994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2025-08-14T22:38:02.1909976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity
2025-08-14T22:38:02.1910938Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2025-08-14T22:38:02.1911862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min
2025-08-14T22:38:02.1912668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-08-14T22:38:02.1913440Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min
2025-08-14T22:38:02.1914187Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda
2025-08-14T22:38:02.1914974Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-08-14T22:38:02.1915781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2025-08-14T22:38:02.1916592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:38:02.1917048Z 
2025-08-14T22:38:02.1917244Z Running distributed tests for the gloo backend with env init_method
2025-08-14T22:38:02.1917628Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:38:02.1918596Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:38:02.186786]
2025-08-14T22:43:30.9145513Z 
2025-08-14T22:43:30.9146825Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_7b37e2bed49829bd_.log
2025-08-14T22:43:30.9167224Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:43:30.9181552Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2025-08-14T22:43:30.9182616Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2025-08-14T22:43:30.9183554Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream
2025-08-14T22:43:30.9184487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync
2025-08-14T22:43:30.9185362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2025-08-14T22:43:30.9186228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup
2025-08-14T22:43:30.9187268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min
2025-08-14T22:43:30.9188212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-08-14T22:43:30.9189072Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-08-14T22:43:30.9189827Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2025-08-14T22:43:30.9190611Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group
2025-08-14T22:43:30.9191394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2025-08-14T22:43:30.9192178Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group
2025-08-14T22:43:30.9192965Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2025-08-14T22:43:30.9193886Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger
2025-08-14T22:43:30.9194797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device
2025-08-14T22:43:30.9195562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2025-08-14T22:43:30.9196358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-08-14T22:43:30.9197241Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error
2025-08-14T22:43:30.9198199Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params
2025-08-14T22:43:30.9199134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph
2025-08-14T22:43:30.9200006Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-08-14T22:43:30.9200840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-08-14T22:43:30.9201641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-08-14T22:43:30.9202488Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2025-08-14T22:43:30.9203460Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-08-14T22:43:30.9204225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-08-14T22:43:30.9204990Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler
2025-08-14T22:43:30.9205926Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2025-08-14T22:43:30.9206896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity
2025-08-14T22:43:30.9207844Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2025-08-14T22:43:30.9208910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min
2025-08-14T22:43:30.9209719Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-08-14T22:43:30.9210482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min
2025-08-14T22:43:30.9211220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda
2025-08-14T22:43:30.9211981Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-08-14T22:43:30.9212774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2025-08-14T22:43:30.9213583Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:43:30.9214038Z 
2025-08-14T22:43:30.9214230Z Running distributed tests for the gloo backend with file init_method
2025-08-14T22:43:30.9214602Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:43:30.9215560Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=2', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:43:30.916590]
2025-08-14T22:48:58.9174370Z 
2025-08-14T22:48:58.9175383Z distributed/test_distributed_spawn 2/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_2.8_95582a440a761cd1_.log
2025-08-14T22:48:58.9191912Z Running 38 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:48:58.9206198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager
2025-08-14T22:48:58.9207225Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel
2025-08-14T22:48:58.9208159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_non_default_stream
2025-08-14T22:48:58.9209091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync
2025-08-14T22:48:58.9209966Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_simple
2025-08-14T22:48:58.9210840Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_subgroup
2025-08-14T22:48:58.9211718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_min
2025-08-14T22:48:58.9212648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_product
2025-08-14T22:48:58.9213688Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum
2025-08-14T22:48:58.9214450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all
2025-08-14T22:48:58.9215236Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group
2025-08-14T22:48:58.9216029Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group
2025-08-14T22:48:58.9216825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_group
2025-08-14T22:48:58.9217615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_cuda
2025-08-14T22:48:58.9218542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_without_logger
2025-08-14T22:48:58.9219609Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device
2025-08-14T22:48:58.9220381Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_has_finalized
2025-08-14T22:48:58.9221177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_ignore_params_arg
2025-08-14T22:48:58.9222054Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_error
2025-08-14T22:48:58.9222998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_ignored_params
2025-08-14T22:48:58.9223937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd_static_graph
2025-08-14T22:48:58.9224814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_python_error_logged
2025-08-14T22:48:58.9225653Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_remove_autograd_hooks
2025-08-14T22:48:58.9226456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_group
2025-08-14T22:48:58.9227207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_cuda
2025-08-14T22:48:58.9227969Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_full_group
2025-08-14T22:48:58.9228722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_irecv
2025-08-14T22:48:58.9229508Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_autograd_profiler
2025-08-14T22:48:58.9230455Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_input_rank_exceeds_world_size
2025-08-14T22:48:58.9231432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity
2025-08-14T22:48:58.9232396Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd
2025-08-14T22:48:58.9233314Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_min
2025-08-14T22:48:58.9234122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_sum
2025-08-14T22:48:58.9235030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_min
2025-08-14T22:48:58.9235776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda
2025-08-14T22:48:58.9236633Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_cuda_complex
2025-08-14T22:48:58.9237454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum
2025-08-14T22:48:58.9238275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_stateless_api_with_ddp
2025-08-14T22:48:58.9238727Z 
2025-08-14T22:48:58.9238955Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T22:48:58.9239362Z Uploading artifacts took 0.00 seconds
2025-08-14T22:48:58.9239754Z Running distributed/test_distributed_spawn 5/8 ... [2025-08-14 22:48:58.919107]
2025-08-14T22:48:58.9240212Z MPI not available -- MPI backend tests will be skipped
2025-08-14T22:48:58.9240744Z Running distributed tests for the test backend with env init_method
2025-08-14T22:48:58.9241132Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:48:58.9242090Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:48:58.921295]
2025-08-14T22:49:02.9348960Z 
2025-08-14T22:49:02.9350280Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_6e66a8065bbb3fa3_.log
2025-08-14T22:49:02.9351638Z Running 0 items in this shard:
2025-08-14T22:49:02.9351925Z 
2025-08-14T22:49:02.9356634Z Running distributed tests for the test backend with file init_method
2025-08-14T22:49:02.9357922Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:49:02.9360852Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:49:02.935891]
2025-08-14T22:49:06.9302661Z 
2025-08-14T22:49:06.9303929Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_68e6f7dce9be1481_.log
2025-08-14T22:49:06.9305249Z Running 0 items in this shard:
2025-08-14T22:49:06.9305540Z 
2025-08-14T22:49:06.9310498Z Running distributed tests for the nccl backend with env init_method
2025-08-14T22:49:06.9321069Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:49:06.9322341Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:49:06.931328]
2025-08-14T22:54:53.2213972Z 
2025-08-14T22:54:53.2215013Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_517a67e08f4c2360_.log
2025-08-14T22:54:53.2233823Z Running 39 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T22:54:53.2248891Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2025-08-14T22:54:53.2249874Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-08-14T22:54:53.2250883Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view
2025-08-14T22:54:53.2252100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook
2025-08-14T22:54:53.2253120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2025-08-14T22:54:53.2254034Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-08-14T22:54:53.2254806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-08-14T22:54:53.2255663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda
2025-08-14T22:54:53.2256581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda
2025-08-14T22:54:53.2257626Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-08-14T22:54:53.2258487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-08-14T22:54:53.2259327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async
2025-08-14T22:54:53.2260159Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex
2025-08-14T22:54:53.2260957Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda
2025-08-14T22:54:53.2261805Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl
2025-08-14T22:54:53.2262722Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook
2025-08-14T22:54:53.2263624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2025-08-14T22:54:53.2264520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD
2025-08-14T22:54:53.2265351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2025-08-14T22:54:53.2266177Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2025-08-14T22:54:53.2267036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-08-14T22:54:53.2267898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-08-14T22:54:53.2268823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-08-14T22:54:53.2269783Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2025-08-14T22:54:53.2270668Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks
2025-08-14T22:54:53.2271442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group
2025-08-14T22:54:53.2272208Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-08-14T22:54:53.2272956Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2025-08-14T22:54:53.2273927Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order
2025-08-14T22:54:53.2274859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2025-08-14T22:54:53.2275763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2025-08-14T22:54:53.2276641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2025-08-14T22:54:53.2277587Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank
2025-08-14T22:54:53.2278591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-08-14T22:54:53.2279512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-08-14T22:54:53.2280536Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-08-14T22:54:53.2281450Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2025-08-14T22:54:53.2282353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-08-14T22:54:53.2283232Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T22:54:53.2283757Z 
2025-08-14T22:54:53.2283945Z Running distributed tests for the nccl backend with file init_method
2025-08-14T22:54:53.2284333Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:54:53.2285305Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:54:53.222463]
2025-08-14T23:00:35.9563194Z 
2025-08-14T23:00:35.9564676Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_9645513bb5242292_.log
2025-08-14T23:00:35.9585051Z Running 39 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:00:35.9599810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2025-08-14T23:00:35.9600774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-08-14T23:00:35.9601788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view
2025-08-14T23:00:35.9602910Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook
2025-08-14T23:00:35.9603923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2025-08-14T23:00:35.9604826Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-08-14T23:00:35.9605592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-08-14T23:00:35.9606416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda
2025-08-14T23:00:35.9607487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda
2025-08-14T23:00:35.9608359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-08-14T23:00:35.9609188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-08-14T23:00:35.9610009Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async
2025-08-14T23:00:35.9610821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex
2025-08-14T23:00:35.9611605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda
2025-08-14T23:00:35.9612442Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl
2025-08-14T23:00:35.9613494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook
2025-08-14T23:00:35.9614380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2025-08-14T23:00:35.9615282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD
2025-08-14T23:00:35.9616105Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2025-08-14T23:00:35.9616919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2025-08-14T23:00:35.9617757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-08-14T23:00:35.9618612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-08-14T23:00:35.9619512Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-08-14T23:00:35.9620457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2025-08-14T23:00:35.9621332Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks
2025-08-14T23:00:35.9622093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group
2025-08-14T23:00:35.9622845Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-08-14T23:00:35.9623600Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2025-08-14T23:00:35.9624438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order
2025-08-14T23:00:35.9625368Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2025-08-14T23:00:35.9626273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2025-08-14T23:00:35.9627141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2025-08-14T23:00:35.9628079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank
2025-08-14T23:00:35.9629073Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-08-14T23:00:35.9630116Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-08-14T23:00:35.9630999Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-08-14T23:00:35.9631899Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2025-08-14T23:00:35.9632735Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-08-14T23:00:35.9633605Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:00:35.9634134Z 
2025-08-14T23:00:35.9634318Z Running distributed tests for the gloo backend with env init_method
2025-08-14T23:00:35.9634701Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:00:35.9635796Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:00:35.958246]
2025-08-14T23:06:23.9483377Z 
2025-08-14T23:06:23.9484658Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_933aafeca48bbd4f_.log
2025-08-14T23:06:23.9502355Z Running 39 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:06:23.9517211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2025-08-14T23:06:23.9518196Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-08-14T23:06:23.9519201Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view
2025-08-14T23:06:23.9520212Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook
2025-08-14T23:06:23.9521227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2025-08-14T23:06:23.9522377Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-08-14T23:06:23.9523152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-08-14T23:06:23.9523997Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda
2025-08-14T23:06:23.9524919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda
2025-08-14T23:06:23.9525800Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-08-14T23:06:23.9526651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-08-14T23:06:23.9527486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async
2025-08-14T23:06:23.9528308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex
2025-08-14T23:06:23.9529109Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda
2025-08-14T23:06:23.9530150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl
2025-08-14T23:06:23.9531075Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook
2025-08-14T23:06:23.9531972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2025-08-14T23:06:23.9532858Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD
2025-08-14T23:06:23.9533689Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2025-08-14T23:06:23.9534507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2025-08-14T23:06:23.9535357Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-08-14T23:06:23.9536355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-08-14T23:06:23.9537261Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-08-14T23:06:23.9538209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2025-08-14T23:06:23.9539088Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks
2025-08-14T23:06:23.9539851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group
2025-08-14T23:06:23.9540615Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-08-14T23:06:23.9541383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2025-08-14T23:06:23.9542215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order
2025-08-14T23:06:23.9543134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2025-08-14T23:06:23.9544043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2025-08-14T23:06:23.9544919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2025-08-14T23:06:23.9545853Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank
2025-08-14T23:06:23.9546854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-08-14T23:06:23.9547760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-08-14T23:06:23.9548646Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-08-14T23:06:23.9549547Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2025-08-14T23:06:23.9550380Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-08-14T23:06:23.9551260Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:06:23.9551924Z 
2025-08-14T23:06:23.9552115Z Running distributed tests for the gloo backend with file init_method
2025-08-14T23:06:23.9552500Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:06:23.9553470Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=5', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:06:23.950245]
2025-08-14T23:12:09.9840742Z 
2025-08-14T23:12:09.9842065Z distributed/test_distributed_spawn 5/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_5.8_997e70845e08d3ce_.log
2025-08-14T23:12:09.9863624Z Running 39 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:12:09.9878418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm
2025-08-14T23:12:09.9879401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_requires_grad
2025-08-14T23:12:09.9880551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_with_amp_and_grad_is_view
2025-08-14T23:12:09.9881571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_hook
2025-08-14T23:12:09.9882778Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_allreduce_with_then_hook
2025-08-14T23:12:09.9893405Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather
2025-08-14T23:12:09.9894215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda
2025-08-14T23:12:09.9895080Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_cat_tensor_cuda
2025-08-14T23:12:09.9895998Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_into_stack_tensor_cuda
2025-08-14T23:12:09.9896875Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max
2025-08-14T23:12:09.9897720Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_product
2025-08-14T23:12:09.9898542Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_async
2025-08-14T23:12:09.9899360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda_complex
2025-08-14T23:12:09.9900166Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_cuda
2025-08-14T23:12:09.9901017Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_ring_exchange_nccl
2025-08-14T23:12:09.9901937Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer_via_hook
2025-08-14T23:12:09.9902822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_device_mesh_initialization
2025-08-14T23:12:09.9903700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_post_localSGD
2025-08-14T23:12:09.9904523Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_inference
2025-08-14T23:12:09.9905333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_join_model_equivalence
2025-08-14T23:12:09.9906423Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_cpu
2025-08-14T23:12:09.9907275Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_execution_trace
2025-08-14T23:12:09.9908158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_shared_grad_acc_unused_params
2025-08-14T23:12:09.9909097Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_unused_params_rebuild_buckets_exception
2025-08-14T23:12:09.9909977Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_checks
2025-08-14T23:12:09.9910728Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_group
2025-08-14T23:12:09.9911490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object
2025-08-14T23:12:09.9912392Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_backend
2025-08-14T23:12:09.9913218Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_failure_order
2025-08-14T23:12:09.9914133Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_rank_0_timeout
2025-08-14T23:12:09.9915021Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_broadcast
2025-08-14T23:12:09.9915880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration
2025-08-14T23:12:09.9916818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_by_enumeration_negative_input_rank
2025-08-14T23:12:09.9917809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_group_size_exceeds_world_size
2025-08-14T23:12:09.9918713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_max
2025-08-14T23:12:09.9919591Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_autograd_profiler
2025-08-14T23:12:09.9920492Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_autograd_profiler
2025-08-14T23:12:09.9921322Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag
2025-08-14T23:12:09.9922294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_undefined_grad_parity_unused_parameters
2025-08-14T23:12:09.9922815Z 
2025-08-14T23:12:09.9923042Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T23:12:09.9923457Z Uploading artifacts took 0.00 seconds
2025-08-14T23:12:09.9923848Z Running distributed/test_distributed_spawn 8/8 ... [2025-08-14 23:12:09.985952]
2025-08-14T23:12:09.9924308Z MPI not available -- MPI backend tests will be skipped
2025-08-14T23:12:09.9924731Z Running distributed tests for the test backend with env init_method
2025-08-14T23:12:09.9925113Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:12:09.9926071Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:12:09.988188]
2025-08-14T23:12:13.9725621Z 
2025-08-14T23:12:13.9727127Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_b12f31cbfea55d5b_.log
2025-08-14T23:12:13.9728724Z Running 0 items in this shard:
2025-08-14T23:12:13.9729076Z 
2025-08-14T23:12:13.9734560Z Running distributed tests for the test backend with file init_method
2025-08-14T23:12:13.9735849Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:12:13.9739106Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:12:13.973730]
2025-08-14T23:12:17.9634262Z 
2025-08-14T23:12:17.9635456Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_260d4d6550d1f13a_.log
2025-08-14T23:12:17.9636808Z Running 0 items in this shard:
2025-08-14T23:12:17.9637107Z 
2025-08-14T23:12:17.9642537Z Running distributed tests for the nccl backend with env init_method
2025-08-14T23:12:17.9644422Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:12:17.9646687Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:12:17.964481]
2025-08-14T23:17:48.0419438Z 
2025-08-14T23:17:48.0420807Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_df471ff5826f10de_.log
2025-08-14T23:17:48.0448055Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:17:48.0462386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-08-14T23:17:48.0463499Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-08-14T23:17:48.0464474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2025-08-14T23:17:48.0465370Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2025-08-14T23:17:48.0466209Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group
2025-08-14T23:17:48.0467046Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min
2025-08-14T23:17:48.0467894Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-08-14T23:17:48.0468763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product
2025-08-14T23:17:48.0469669Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2025-08-14T23:17:48.0470595Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-08-14T23:17:48.0471477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-08-14T23:17:48.0472351Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl
2025-08-14T23:17:48.0473250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-08-14T23:17:48.0474114Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2025-08-14T23:17:48.0475173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping
2025-08-14T23:17:48.0476076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2025-08-14T23:17:48.0476975Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks
2025-08-14T23:17:48.0477890Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types
2025-08-14T23:17:48.0478785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-08-14T23:17:48.0479721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn
2025-08-14T23:17:48.0480586Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-08-14T23:17:48.0481525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-08-14T23:17:48.0482432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group
2025-08-14T23:17:48.0483211Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2025-08-14T23:17:48.0484085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2025-08-14T23:17:48.0485030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce
2025-08-14T23:17:48.0485884Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce
2025-08-14T23:17:48.0486734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream
2025-08-14T23:17:48.0487624Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2025-08-14T23:17:48.0488598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-08-14T23:17:48.0489510Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max
2025-08-14T23:17:48.0490294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-08-14T23:17:48.0491098Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-08-14T23:17:48.0491915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-08-14T23:17:48.0492780Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler
2025-08-14T23:17:48.0493697Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-08-14T23:17:48.0494618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:17:48.0495125Z 
2025-08-14T23:17:48.0495318Z Running distributed tests for the nccl backend with file init_method
2025-08-14T23:17:48.0495709Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:17:48.0496675Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:17:48.043671]
2025-08-14T23:23:18.8807223Z 
2025-08-14T23:23:18.8808511Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_32df0d8562b300ed_.log
2025-08-14T23:23:18.8826240Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:23:18.8840389Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-08-14T23:23:18.8841475Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-08-14T23:23:18.8842558Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2025-08-14T23:23:18.8843414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2025-08-14T23:23:18.8844228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group
2025-08-14T23:23:18.8845203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min
2025-08-14T23:23:18.8846044Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-08-14T23:23:18.8846893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product
2025-08-14T23:23:18.8847805Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2025-08-14T23:23:18.8848706Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-08-14T23:23:18.8849578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-08-14T23:23:18.8850454Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl
2025-08-14T23:23:18.8851333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-08-14T23:23:18.8852191Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2025-08-14T23:23:18.8853079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping
2025-08-14T23:23:18.8853962Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2025-08-14T23:23:18.8854842Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks
2025-08-14T23:23:18.8855742Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types
2025-08-14T23:23:18.8856618Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-08-14T23:23:18.8857520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn
2025-08-14T23:23:18.8858360Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-08-14T23:23:18.8859134Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-08-14T23:23:18.8859950Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group
2025-08-14T23:23:18.8860868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2025-08-14T23:23:18.8861727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2025-08-14T23:23:18.8862647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce
2025-08-14T23:23:18.8863485Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce
2025-08-14T23:23:18.8864313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream
2025-08-14T23:23:18.8865180Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2025-08-14T23:23:18.8866137Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-08-14T23:23:18.8867167Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max
2025-08-14T23:23:18.8867945Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-08-14T23:23:18.8868727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-08-14T23:23:18.8869520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-08-14T23:23:18.8870359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler
2025-08-14T23:23:18.8871250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-08-14T23:23:18.8872157Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:23:18.8872653Z 
2025-08-14T23:23:18.8872834Z Running distributed tests for the gloo backend with env init_method
2025-08-14T23:23:18.8873204Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:23:18.8874158Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:23:18.882442]
2025-08-14T23:28:08.1819070Z 
2025-08-14T23:28:08.1820454Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_4ab5ad4edac89536_.log
2025-08-14T23:28:08.1837184Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:28:08.1851355Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-08-14T23:28:08.1852456Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-08-14T23:28:08.1853400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2025-08-14T23:28:08.1854276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2025-08-14T23:28:08.1855087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group
2025-08-14T23:28:08.1855893Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min
2025-08-14T23:28:08.1856721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-08-14T23:28:08.1857763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product
2025-08-14T23:28:08.1858641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2025-08-14T23:28:08.1859540Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-08-14T23:28:08.1860398Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-08-14T23:28:08.1861256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl
2025-08-14T23:28:08.1862125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-08-14T23:28:08.1862978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2025-08-14T23:28:08.1864020Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping
2025-08-14T23:28:08.1864897Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2025-08-14T23:28:08.1865767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks
2025-08-14T23:28:08.1866659Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types
2025-08-14T23:28:08.1867518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-08-14T23:28:08.1868412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn
2025-08-14T23:28:08.1869254Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-08-14T23:28:08.1870028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-08-14T23:28:08.1870837Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group
2025-08-14T23:28:08.1871585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2025-08-14T23:28:08.1872432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2025-08-14T23:28:08.1873344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce
2025-08-14T23:28:08.1874182Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce
2025-08-14T23:28:08.1875015Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream
2025-08-14T23:28:08.1875881Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2025-08-14T23:28:08.1876836Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-08-14T23:28:08.1877734Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max
2025-08-14T23:28:08.1878502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-08-14T23:28:08.1879432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-08-14T23:28:08.1880228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-08-14T23:28:08.1881062Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler
2025-08-14T23:28:08.1881947Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-08-14T23:28:08.1882914Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:28:08.1883412Z 
2025-08-14T23:28:08.1883613Z Running distributed tests for the gloo backend with file init_method
2025-08-14T23:28:08.1883995Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:28:08.1885128Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=8', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:28:08.183658]
2025-08-14T23:32:57.8689065Z 
2025-08-14T23:32:57.8690324Z distributed/test_distributed_spawn 8/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_8.8_f2055b184e306514_.log
2025-08-14T23:32:57.8711402Z Running 37 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:32:57.8725920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value
2025-08-14T23:32:57.8727016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_half
2025-08-14T23:32:57.8727960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_with_empty
2025-08-14T23:32:57.8728824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_cuda_complex
2025-08-14T23:32:57.8729650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_group
2025-08-14T23:32:57.8730458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_min
2025-08-14T23:32:57.8731291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_sum
2025-08-14T23:32:57.8732144Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_product
2025-08-14T23:32:57.8733036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split
2025-08-14T23:32:57.8733946Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_mixed_backend_err
2025-08-14T23:32:57.8734821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_nccl
2025-08-14T23:32:57.8735692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_no_rank_zero_nccl
2025-08-14T23:32:57.8736572Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_self_nccl
2025-08-14T23:32:57.8737428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward
2025-08-14T23:32:57.8738327Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping
2025-08-14T23:32:57.8739220Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_powerSGD
2025-08-14T23:32:57.8740286Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_num_params_across_ranks
2025-08-14T23:32:57.8741190Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_static_graph_nested_types
2025-08-14T23:32:57.8742058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_join_disable
2025-08-14T23:32:57.8742968Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs_stop_iteration_sync_bn
2025-08-14T23:32:57.8743818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank
2025-08-14T23:32:57.8744599Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_full_group
2025-08-14T23:32:57.8745418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_rank_size_group
2025-08-14T23:32:57.8746334Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend
2025-08-14T23:32:57.8747195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang_wait_all_ranks
2025-08-14T23:32:57.8748123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allreduce
2025-08-14T23:32:57.8748976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_reduce
2025-08-14T23:32:57.8749809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_high_priority_stream
2025-08-14T23:32:57.8750681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_overlap_not_allowed
2025-08-14T23:32:57.8751651Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_world_size_not_divisible_by_group_size
2025-08-14T23:32:57.8752562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_max
2025-08-14T23:32:57.8753356Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_product
2025-08-14T23:32:57.8754151Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda_twice
2025-08-14T23:32:57.8754960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_full_group
2025-08-14T23:32:57.8755807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_autograd_profiler
2025-08-14T23:32:57.8756713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_torch_profiler
2025-08-14T23:32:57.8757610Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_skip_all_reduce_unused_parameters
2025-08-14T23:32:57.8758112Z 
2025-08-14T23:32:57.8758328Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T23:32:57.8758731Z Uploading artifacts took 0.00 seconds
2025-08-14T23:32:57.8759135Z Running distributed/fsdp/test_fsdp_hybrid_shard 1/1 ... [2025-08-14 23:32:57.870586]
2025-08-14T23:32:57.8759545Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:32:57.8760468Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:32:57.870885]
2025-08-14T23:34:15.7356209Z 
2025-08-14T23:34:15.7357658Z distributed/fsdp/test_fsdp_hybrid_shard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_hybrid_shard_1.1_b20104ae976f7ba0_.log
2025-08-14T23:34:15.7362957Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_basic_setup, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_fsdp_hybrid_shard_parity, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_save_load_state_dict, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_hsdp_sync_module_state, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_invalid_pg_specification_raises, test/distributed/fsdp/test_fsdp_hybrid_shard.py::TestFSDPHybridShard::test_raises_manual_wrap_hybrid_shard_when_none_policy
2025-08-14T23:34:15.7367297Z 
2025-08-14T23:34:15.7367774Z Running distributed/fsdp/test_fsdp_exec_order 1/1 ... [2025-08-14 23:34:15.736292]
2025-08-14T23:34:15.7368623Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:34:15.7370262Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_exec_order.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:34:15.736613]
2025-08-14T23:35:26.3916997Z 
2025-08-14T23:35:26.3918311Z distributed/fsdp/test_fsdp_exec_order 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_exec_order_1.1_9cf1eb09bd441f69_.log
2025-08-14T23:35:26.3926283Z Running 8 items in this shard: test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_first_iter_order_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy0_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_1_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_invalid_later_iter_order_sharding_strategy1_iters_before_path_change_3_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_exec_order.py::TestFSDPExecOrderCUDA::test_train_eval_sharding_strategy1_cuda
2025-08-14T23:35:26.3930475Z 
2025-08-14T23:35:26.3930690Z Running distributed/fsdp/test_wrap 1/1 ... [2025-08-14 23:35:26.391916]
2025-08-14T23:35:26.3931096Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:35:26.3932033Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_wrap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:35:26.392263]
2025-08-14T23:38:32.6453152Z 
2025-08-14T23:38:32.6454263Z distributed/fsdp/test_wrap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_wrap_1.1_fff56e74d37c787a_.log
2025-08-14T23:38:32.6479272Z Running 52 items in this shard: test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_bn_always_wrapped_individually, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_error_already_wrapped_nested_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload0_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch0_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_False_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode0, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_main_wrap_api_cpu_offload1_backward_prefetch1_forward_prefetch_True_device_init_mode1, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_False, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_wrap_batchnorm_individually_use_or_policy_True, test/distributed/fsdp/test_wrap.py::TestFSDPWrap::test_zero_argument, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_always_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_api, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_exclude_wrap_include_children, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_preset_force_leaf_custom, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode0_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload0_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_False, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_smoke_test_device_init_mode1_cpu_offload1_use_device_id_True, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_auto_wrap_with_ignored_modules_wrap_method1, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_custom_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_frozen_params, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_module_wrap_policy_callable, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_transformer_auto_wrap_policy, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_disabled_outside_context, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_override_defaults, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method0, test/distributed/fsdp/test_wrap.py::TestAutoWrap::test_wrap_wrap_method1, test/distributed/fsdp/test_wrap.py::TestWrapUtils::test_validate_frozen_params
2025-08-14T23:38:32.6498295Z 
2025-08-14T23:38:32.6498507Z Running distributed/fsdp/test_fsdp_uneven 1/1 ... [2025-08-14 23:38:32.646153]
2025-08-14T23:38:32.6498922Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:38:32.6499843Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_uneven.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:38:32.646475]
2025-08-14T23:38:45.6279272Z 
2025-08-14T23:38:45.6280640Z distributed/fsdp/test_fsdp_uneven 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_uneven_1.1_055f1220bc596660_.log
2025-08-14T23:38:45.6282926Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_uneven.py::TestUnevenParamShardCUDA::test_one_iteration_cuda
2025-08-14T23:38:45.6283881Z 
2025-08-14T23:38:45.6286484Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 ... [2025-08-14 23:38:45.628401]
2025-08-14T23:38:45.6287289Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:38:45.6289638Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:38:45.628726]
2025-08-14T23:39:30.1999063Z 
2025-08-14T23:39:30.2000635Z distributed/algorithms/ddp_comm_hooks/test_ddp_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks_1.1_637299e0a848a3d5_.log
2025-08-14T23:39:30.2007474Z Running 6 items in this shard: test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_allreduce_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_fp16compress_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_noop_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_channel_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_ddp_comm_hook_quantize_per_tensor_hook, test/distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py::DistributedDataParallelCommHookTest::test_is_last_hook
2025-08-14T23:39:30.2010478Z 
2025-08-14T23:39:30.2010784Z Running distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 ... [2025-08-14 23:39:30.200096]
2025-08-14T23:39:30.2011685Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:39:30.2012694Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:39:30.200468]
2025-08-14T23:40:16.1744072Z 
2025-08-14T23:40:16.1745665Z distributed/_composable/fsdp/test_fully_shard_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_dict_1.1_312d2b2db67fcc23_.log
2025-08-14T23:40:16.1752698Z Running 6 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_2d_state_dict_correctness, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_cpu_offload, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_dp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiProcess::test_hsdp_tp_state_dict_save_load, test/distributed/_composable/fsdp/test_fully_shard_state_dict.py::TestFullyShardStateDictMultiThread::test_rank0_offload_full_state_dict
2025-08-14T23:40:16.1756422Z 
2025-08-14T23:40:16.1756661Z Running distributed/fsdp/test_fsdp_input 1/1 ... [2025-08-14 23:40:16.174667]
2025-08-14T23:40:16.1757109Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:40:16.1758141Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:40:16.174967]
2025-08-14T23:40:34.0123173Z 
2025-08-14T23:40:34.0124399Z distributed/fsdp/test_fsdp_input 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_input_1.1_c0d3889e17cfbc09_.log
2025-08-14T23:40:34.0126558Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_dict_cuda, test/distributed/fsdp/test_fsdp_input.py::TestInputCUDA::test_input_type_list_cuda
2025-08-14T23:40:34.0127730Z 
2025-08-14T23:40:34.0131598Z Running distributed/fsdp/test_fsdp_multiple_wrapping 1/1 ... [2025-08-14 23:40:34.012663]
2025-08-14T23:40:34.0132667Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:40:34.0134432Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_wrapping.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:40:34.012987]
2025-08-14T23:40:46.7937780Z 
2025-08-14T23:40:46.7939275Z distributed/fsdp/test_fsdp_multiple_wrapping 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_wrapping_1.1_884712620a91cfc3_.log
2025-08-14T23:40:46.7941324Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_multiple_wrapping.py::TestMultipleWrappingCUDA::test_multiple_wrapping_cuda
2025-08-14T23:40:46.7942198Z 
2025-08-14T23:40:46.7942461Z Running distributed/_shard/test_sharder 1/1 ... [2025-08-14 23:40:46.793978]
2025-08-14T23:40:46.7942864Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:40:46.7944812Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:40:46.794308]
2025-08-14T23:41:04.8322722Z 
2025-08-14T23:41:04.8324182Z distributed/_shard/test_sharder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.test_sharder_1.1_f6c9a9990290eb62_.log
2025-08-14T23:41:04.8326813Z Running 2 items in this shard: test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder, test/distributed/_shard/test_sharder.py::TestCustomSharder::test_custom_sharder_errors
2025-08-14T23:41:04.8328029Z 
2025-08-14T23:41:04.8329792Z Running distributed/algorithms/test_join 1/1 ... [2025-08-14 23:41:04.832789]
2025-08-14T23:41:04.8330515Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:41:04.8333502Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/test_join.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:41:04.833102]
2025-08-14T23:42:09.7808167Z 
2025-08-14T23:42:09.7809521Z distributed/algorithms/test_join 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.test_join_1.1_84ef22482874e972_.log
2025-08-14T23:42:09.7816179Z Running 9 items in this shard: test/distributed/algorithms/test_join.py::TestJoin::test_join_kwargs, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables, test/distributed/algorithms/test_join.py::TestJoin::test_multiple_joinables_throw, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_disable, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_main_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_post_hooks, test/distributed/algorithms/test_join.py::TestJoin::test_single_joinable_throw
2025-08-14T23:42:09.7819429Z 
2025-08-14T23:42:09.7819677Z Running distributed/test_c10d_pypg 1/1 ... [2025-08-14 23:42:09.781050]
2025-08-14T23:42:09.7820161Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:42:09.7821268Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:42:09.781366]
2025-08-14T23:42:18.2072231Z 
2025-08-14T23:42:18.2073656Z distributed/test_c10d_pypg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_pypg_1.1_442f6cca3471be5e_.log
2025-08-14T23:42:18.2094750Z Running 48 items in this shard: test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkSubclass::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_dataclass_output_unused_param, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_invoke_work_object, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_no_init_sync, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_ddp_with_pypg_with_grad_views, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_invalid_powerSGD_state, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_empty_input, test/distributed/test_c10d_pypg.py::TestDDPWithWorkWrapper::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_abort_shutdown, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_attr_overrides, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_block_current_stream, test/distributed/test_c10d_pypg.py::TestPyProcessGroup::test_block_current_stream_use_after_free
2025-08-14T23:42:18.2111327Z 
2025-08-14T23:42:18.2111572Z Running distributed/fsdp/test_fsdp_use_orig_params 1/1 ... [2025-08-14 23:42:18.207497]
2025-08-14T23:42:18.2112000Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:42:18.2112929Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:42:18.207816]
2025-08-14T23:50:21.9372020Z 
2025-08-14T23:50:21.9373206Z distributed/fsdp/test_fsdp_use_orig_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_use_orig_params_1.1_492da08f9df66dbb_.log
2025-08-14T23:50:21.9394810Z Running 25 items in this shard: test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_full_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_no_shard, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_hyperparams_sharding_strategy_str_shard_grad_op, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_diff_trainability, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_fsdp_compile, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsMultipleParamGroups::test_multiple_optimizers, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_multiple_forward_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_False, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsUnshardReshard::test_summon_between_two_forwards_offload_params_True, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsParamAccess::test_access_params_after_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_grad_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_no_reshard_and_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_param_writeback, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_between_fwd_and_bwd_for_no_reshard_raises, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsWriteback::test_writeback_shape_mismatch, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsFQNs::test_named_parameters_in_forward, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_correctness, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsNoSync::test_no_sync_mixed_precision, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestFSDPUseOrigParamsInit::test_non_uniform_requires_grad, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cpu, test/distributed/fsdp/test_fsdp_use_orig_params.py::TestMultiTensorApply::test_multi_tensor_apply_size0_tensors_cuda
2025-08-14T23:50:21.9406326Z 
2025-08-14T23:50:21.9406539Z Running distributed/fsdp/test_fsdp_core 3/3 ... [2025-08-14 23:50:21.938388]
2025-08-14T23:50:21.9406949Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:50:21.9407872Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=3', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:50:21.938705]
2025-08-14T23:56:41.8418713Z 
2025-08-14T23:56:41.8421852Z distributed/fsdp/test_fsdp_core 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_3.3_251216a52ac74ac9_.log
2025-08-14T23:56:41.8434467Z Running 21 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_True_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestAutogradCUDA::test_unshard_params_as_tensors_cuda
2025-08-14T23:56:41.8443474Z 
2025-08-14T23:56:41.8443703Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T23:56:41.8444131Z Uploading artifacts took 0.00 seconds
2025-08-14T23:56:41.8444515Z Running distributed/test_c10d_gloo 2/2 ... [2025-08-14 23:56:41.844168]
2025-08-14T23:56:41.8444904Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:56:41.8446735Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=2', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:56:41.844490]
2025-08-15T00:12:51.8656398Z 
2025-08-15T00:12:51.8657532Z distributed/test_c10d_gloo 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_2.2_6fde8c94fe12282f_.log
2025-08-15T00:12:51.8698560Z Running 116 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_op_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_block_current_stream_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_long, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_pickle, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_cuda_dispatched, test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2025-08-15T00:12:51.8735072Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousEnvTest::test_logging_init
2025-08-15T00:12:51.8735778Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics
2025-08-15T00:12:51.8736490Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_basics_cuda
2025-08-15T00:12:51.8737198Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_checks
2025-08-15T00:12:51.8737911Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_inference_mode
2025-08-15T00:12:51.8738667Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_noncontiguous_input
2025-08-15T00:12:51.8739436Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_async
2025-08-15T00:12:51.8740194Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks
2025-08-15T00:12:51.8740935Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_stress
2025-08-15T00:12:51.8741664Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_op_timeout
2025-08-15T00:12:51.8742405Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress_cuda
2025-08-15T00:12:51.8743143Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_block_current_stream_cuda
2025-08-15T00:12:51.8743889Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_checks
2025-08-15T00:12:51.8744751Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics
2025-08-15T00:12:51.8745438Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_basics_cuda
2025-08-15T00:12:51.8746121Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_checks
2025-08-15T00:12:51.8746832Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_noncontiguous_input
2025-08-15T00:12:51.8747544Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress
2025-08-15T00:12:51.8748227Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics_cuda
2025-08-15T00:12:51.8748904Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_checks
2025-08-15T00:12:51.8749594Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter
2025-08-15T00:12:51.8750461Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor
2025-08-15T00:12:51.8751250Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_scatter_tensor_coalesced
2025-08-15T00:12:51.8751988Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_checks
2025-08-15T00:12:51.8752685Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress_cuda
2025-08-15T00:12:51.8753414Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_checks
2025-08-15T00:12:51.8754200Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_cuda_dispatched
2025-08-15T00:12:51.8755041Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output_unused_param
2025-08-15T00:12:51.8755977Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False
2025-08-15T00:12:51.8756940Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing
2025-08-15T00:12:51.8757876Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2025-08-15T00:12:51.8758849Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False
2025-08-15T00:12:51.8759767Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_register_just_once
2025-08-15T00:12:51.8760602Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_init
2025-08-15T00:12:51.8761434Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_invalid_comm_hook_return_type
2025-08-15T00:12:51.8762413Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_find_unused_parameters_when_unused_parameters_empty
2025-08-15T00:12:51.8763385Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_static_graph
2025-08-15T00:12:51.8764337Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_integer_list
2025-08-15T00:12:51.8765299Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_1gpu_module_device_ids_torch_device_list
2025-08-15T00:12:51.8766361Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_2gpu_module
2025-08-15T00:12:51.8767155Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_4gpu_module
2025-08-15T00:12:51.8767995Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output_with_unused_parameters
2025-08-15T00:12:51.8768839Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_invalid_powerSGD_state
2025-08-15T00:12:51.8769617Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_save_load_checkpoint
2025-08-15T00:12:51.8770422Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input
2025-08-15T00:12:51.8771195Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_optimizer
2025-08-15T00:12:51.8771922Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward_unused_parameters
2025-08-15T00:12:51.8772772Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_multi_bucket
2025-08-15T00:12:51.8773487Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics
2025-08-15T00:12:51.8774258Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_basics_cuda
2025-08-15T00:12:51.8775057Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_async
2025-08-15T00:12:51.8775905Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_noncontiguous_input
2025-08-15T00:12:51.8776722Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress_cuda
2025-08-15T00:12:51.8777491Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics
2025-08-15T00:12:51.8778275Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_basics
2025-08-15T00:12:51.8779074Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_barrier_implies_wait
2025-08-15T00:12:51.8779845Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics_cuda
2025-08-15T00:12:51.8780604Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_checks
2025-08-15T00:12:51.8781355Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics_cuda
2025-08-15T00:12:51.8782101Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress
2025-08-15T00:12:51.8782868Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_multi_device_constructor
2025-08-15T00:12:51.8783640Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics
2025-08-15T00:12:51.8784374Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_basics_cuda
2025-08-15T00:12:51.8785107Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_checks
2025-08-15T00:12:51.8785834Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter
2025-08-15T00:12:51.8786573Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress_cuda
2025-08-15T00:12:51.8787466Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics_cuda
2025-08-15T00:12:51.8788214Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress
2025-08-15T00:12:51.8788955Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_stress_cuda
2025-08-15T00:12:51.8789750Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics_cuda
2025-08-15T00:12:51.8790569Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_checks
2025-08-15T00:12:51.8791401Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_cuda_dispatched
2025-08-15T00:12:51.8792195Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_checks
2025-08-15T00:12:51.8792949Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_into_tensor_coalesced
2025-08-15T00:12:51.8793875Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_noncontiguous_input
2025-08-15T00:12:51.8794624Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics
2025-08-15T00:12:51.8795320Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_checks
2025-08-15T00:12:51.8796051Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_async
2025-08-15T00:12:51.8796811Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_basics
2025-08-15T00:12:51.8797576Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks
2025-08-15T00:12:51.8798366Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_checks_cuda
2025-08-15T00:12:51.8799138Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress_cuda
2025-08-15T00:12:51.8799871Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_barrier_implies_wait
2025-08-15T00:12:51.8800580Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics
2025-08-15T00:12:51.8801291Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_basics_cuda
2025-08-15T00:12:51.8802006Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_checks
2025-08-15T00:12:51.8802756Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_empty_tensors
2025-08-15T00:12:51.8803438Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics
2025-08-15T00:12:51.8804166Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_noncontiguous_input
2025-08-15T00:12:51.8804884Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress
2025-08-15T00:12:51.8805536Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_long
2025-08-15T00:12:51.8806239Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_multi_device_constructor
2025-08-15T00:12:51.8806950Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics
2025-08-15T00:12:51.8807638Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_basics_cuda
2025-08-15T00:12:51.8808753Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor
2025-08-15T00:12:51.8809460Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress
2025-08-15T00:12:51.8810150Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_stress_cuda
2025-08-15T00:12:51.8810843Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics
2025-08-15T00:12:51.8811517Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_checks
2025-08-15T00:12:51.8812195Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress
2025-08-15T00:12:51.8812874Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_pickle
2025-08-15T00:12:51.8813587Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics
2025-08-15T00:12:51.8814546Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_cuda_dispatched
2025-08-15T00:12:51.8815257Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_bool_tensors
2025-08-15T00:12:51.8815876Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_warn_not_in_group
2025-08-15T00:12:51.8816560Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_default
2025-08-15T00:12:51.8817290Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_incremented_gloo_subgroup
2025-08-15T00:12:51.8817984Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_mismatch
2025-08-15T00:12:51.8818776Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single
2025-08-15T00:12:51.8819736Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced
2025-08-15T00:12:51.8820666Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_collectives
2025-08-15T00:12:51.8821589Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-08-15T00:12:51.8822601Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-08-15T00:12:51.8823607Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_monitored_barrier
2025-08-15T00:12:51.8824423Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync
2025-08-15T00:12:51.8825143Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_sanity_check
2025-08-15T00:12:51.8825574Z 
2025-08-15T00:12:51.8825763Z Running distributed/test_c10d_nccl 2/3 ... [2025-08-15 00:12:51.868724]
2025-08-15T00:12:51.8826153Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:12:51.8827089Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=2', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-15 00:12:51.869037]
2025-08-15T00:26:15.6713334Z 
2025-08-15T00:26:15.6714759Z distributed/test_c10d_nccl 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_2.3_6eeda8751605dc8f_.log
2025-08-15T00:26:15.6743128Z Running 75 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_deterministic_mode_no_break, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off, test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k, test/distributed/test_c10d_nccl.py::CommTest::test_unwaited, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group
2025-08-15T00:26:15.6767782Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::TimeoutTest::test_default_store_timeout_nccl
2025-08-15T00:26:15.6768504Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_scalable_init
2025-08-15T00:26:15.6769224Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_pg
2025-08-15T00:26:15.6769969Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_subgroup
2025-08-15T00:26:15.6770739Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_cuda_event_cache_mthd_race
2025-08-15T00:26:15.6771714Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_destruct_before_terminate_pg
2025-08-15T00:26:15.6772529Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_deterministic_mode_no_break
2025-08-15T00:26:15.6773352Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend0
2025-08-15T00:26:15.6774137Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float16
2025-08-15T00:26:15.6774870Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float64
2025-08-15T00:26:15.6775624Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e4m3fn
2025-08-15T00:26:15.6776358Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_check
2025-08-15T00:26:15.6777080Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_rank_filter
2025-08-15T00:26:15.6777989Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_False
2025-08-15T00:26:15.6778755Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_p2p
2025-08-15T00:26:15.6779554Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend0
2025-08-15T00:26:15.6780352Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_process_group_desc
2025-08-15T00:26:15.6781141Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_True
2025-08-15T00:26:15.6782027Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module_with_grad_is_view
2025-08-15T00:26:15.6782948Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value
2025-08-15T00:26:15.6783776Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_nccl
2025-08-15T00:26:15.6784636Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl_grad_is_view
2025-08-15T00:26:15.6785525Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module
2025-08-15T00:26:15.6786428Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False
2025-08-15T00:26:15.6787388Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_False
2025-08-15T00:26:15.6788361Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_grad_is_view
2025-08-15T00:26:15.6789295Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl_static_graph
2025-08-15T00:26:15.6790217Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_nccl
2025-08-15T00:26:15.6791087Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_multi_device_module_config
2025-08-15T00:26:15.6791893Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_weight_sharing
2025-08-15T00:26:15.6792680Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl
2025-08-15T00:26:15.6793691Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_detail
2025-08-15T00:26:15.6794561Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_2devicemodule
2025-08-15T00:26:15.6795363Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_invalid_powerSGD_state
2025-08-15T00:26:15.6796199Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward
2025-08-15T00:26:15.6797127Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_integer_list
2025-08-15T00:26:15.6798102Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_1gpu_module_device_ids_torch_device_list
2025-08-15T00:26:15.6799084Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_module_device_ids_None
2025-08-15T00:26:15.6800093Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_pass_default_pg
2025-08-15T00:26:15.6800925Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl_grad_is_view
2025-08-15T00:26:15.6801744Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_mixed_ops
2025-08-15T00:26:15.6802521Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_nonblocking
2025-08-15T00:26:15.6803243Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_manager_nccl
2025-08-15T00:26:15.6803939Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_all_reduce_coalesced_nccl
2025-08-15T00:26:15.6804609Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_broadcast_coalesced_nccl
2025-08-15T00:26:15.6805237Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier
2025-08-15T00:26:15.6805869Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_barrier_device_ids
2025-08-15T00:26:15.6806540Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_off
2025-08-15T00:26:15.6807212Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nncl_rank_membership
2025-08-15T00:26:15.6807906Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_high_priority_stream
2025-08-15T00:26:15.6808593Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_base_k
2025-08-15T00:26:15.6809204Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_unwaited
2025-08-15T00:26:15.6809931Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_collectives
2025-08-15T00:26:15.6810856Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-08-15T00:26:15.6811740Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_False
2025-08-15T00:26:15.6812588Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_True
2025-08-15T00:26:15.6813428Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_False
2025-08-15T00:26:15.6814190Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_subgroup_group_rank_True
2025-08-15T00:26:15.6815124Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_False
2025-08-15T00:26:15.6815889Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_False
2025-08-15T00:26:15.6816594Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync
2025-08-15T00:26:15.6817335Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_True
2025-08-15T00:26:15.6818193Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_True
2025-08-15T00:26:15.6819086Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_True
2025-08-15T00:26:15.6819960Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_True
2025-08-15T00:26:15.6820991Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_False
2025-08-15T00:26:15.6821924Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_False
2025-08-15T00:26:15.6822838Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce1_timing_enabled_True
2025-08-15T00:26:15.6823731Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_False
2025-08-15T00:26:15.6824594Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes0_timing_enabled_True
2025-08-15T00:26:15.6825471Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_True
2025-08-15T00:26:15.6826370Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_False
2025-08-15T00:26:15.6827261Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_True
2025-08-15T00:26:15.6828149Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_False
2025-08-15T00:26:15.6829017Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_recursive_split_group
2025-08-15T00:26:15.6829490Z 
2025-08-15T00:26:15.6829718Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:26:15.6830127Z Uploading artifacts took 0.00 seconds
2025-08-15T00:26:15.6830528Z Running distributed/elastic/timer/api_test 1/1 ... [2025-08-15 00:26:15.672990]
2025-08-15T00:26:15.6830947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:26:15.6831878Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-15 00:26:15.673299]
2025-08-15T00:26:17.1841427Z 
2025-08-15T00:26:17.1842750Z distributed/elastic/timer/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.api_test_1.1_4bc763c5db8316ac_.log
2025-08-15T00:26:17.1843897Z 
2025-08-15T00:26:20.6256875Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.6259764Z   import pkg_resources
2025-08-15T00:26:20.7671246Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.7673292Z   import pkg_resources
2025-08-15T00:26:20.7901050Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.7903054Z   import pkg_resources
2025-08-15T00:26:20.8222266Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.8223831Z   import pkg_resources
2025-08-15T00:26:20.8238901Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.8240156Z   import pkg_resources
2025-08-15T00:26:20.8241370Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.8242714Z   import pkg_resources
2025-08-15T00:26:20.8272552Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.8274533Z   import pkg_resources
2025-08-15T00:26:20.8277392Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:20.8278830Z   import pkg_resources
2025-08-15T00:26:22.7168162Z Running test batch 'tests to run' cost 10844.45 seconds
2025-08-15T00:26:23.3620911Z 
2025-08-15T00:26:23.3621152Z real	180m50.695s
2025-08-15T00:26:23.3621462Z user	508m45.052s
2025-08-15T00:26:23.3621730Z sys	513m30.626s
2025-08-15T00:26:23.3622041Z + assert_git_not_dirty
2025-08-15T00:26:23.3622369Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-08-15T00:26:23.3622760Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-08-15T00:26:23.3623125Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-15T00:26:23.3623461Z + [[ 1 == 1 ]]
2025-08-15T00:26:23.3623746Z + echo 'Testing distributed C++ tests'
2025-08-15T00:26:23.3624097Z Testing distributed C++ tests
2025-08-15T00:26:23.3626633Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:26:23.3650147Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:26:23.3676957Z + export CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:26:23.3677856Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:26:23.3678663Z + python test/run_test.py --cpp --verbose -i cpp/FileStoreTest
2025-08-15T00:26:26.3343802Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:26.3345974Z   import pkg_resources
2025-08-15T00:26:28.5924466Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-15T00:26:28.8640572Z Ignoring disabled issues:  ['']
2025-08-15T00:26:28.8768554Z Found test times from artifacts
2025-08-15T00:26:28.9266577Z Found test times from artifacts
2025-08-15T00:26:28.9279725Z Running all tests
2025-08-15T00:26:28.9284472Z Running parallel tests on 8 processes
2025-08-15T00:26:28.9284961Z Name: tests to run (est. time: 0.0min)
2025-08-15T00:26:28.9285313Z   Serial tests (1):
2025-08-15T00:26:28.9285606Z     cpp/FileStoreTest 1/1
2025-08-15T00:26:28.9285914Z   Parallel tests (0):
2025-08-15T00:26:28.9286220Z Name: excluded (est. time: 0.0min)
2025-08-15T00:26:28.9286554Z   Serial tests (0):
2025-08-15T00:26:28.9286830Z   Parallel tests (0):
2025-08-15T00:26:28.9289262Z Running cpp/FileStoreTest 1/1 ... [2025-08-15 00:26:28.928809]
2025-08-15T00:26:28.9289652Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:26:28.9296762Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/FileStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-e32030784a2cf141.xml', '-x', '--reruns=2'] ... [2025-08-15 00:26:28.929487]
2025-08-15T00:26:31.2480898Z 
2025-08-15T00:26:31.2482644Z cpp/FileStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.FileStoreTest_1.1_e9c690394659c096_.log
2025-08-15T00:26:31.2483806Z 
2025-08-15T00:26:31.2484287Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:26:31.2485098Z Uploading artifacts took 0.00 seconds
2025-08-15T00:26:34.6438083Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.6439770Z   import pkg_resources
2025-08-15T00:26:34.7770427Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.7771927Z   import pkg_resources
2025-08-15T00:26:34.8206171Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8213421Z   import pkg_resources
2025-08-15T00:26:34.8394108Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8395575Z   import pkg_resources
2025-08-15T00:26:34.8445527Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8446968Z   import pkg_resources
2025-08-15T00:26:34.8462715Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8464380Z   import pkg_resources
2025-08-15T00:26:34.8471460Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8472635Z   import pkg_resources
2025-08-15T00:26:34.8527376Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:34.8528835Z   import pkg_resources
2025-08-15T00:26:36.3512677Z Running test batch 'tests to run' cost 7.42 seconds
2025-08-15T00:26:37.3177475Z + python test/run_test.py --cpp --verbose -i cpp/HashStoreTest
2025-08-15T00:26:40.2739339Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:40.2741609Z   import pkg_resources
2025-08-15T00:26:42.5485702Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-15T00:26:42.5600877Z Found test times from artifacts
2025-08-15T00:26:42.6093485Z Found test times from artifacts
2025-08-15T00:26:42.6107409Z Running all tests
2025-08-15T00:26:42.6110045Z Running parallel tests on 8 processes
2025-08-15T00:26:42.6110546Z Name: tests to run (est. time: 0.0min)
2025-08-15T00:26:42.6110876Z   Serial tests (1):
2025-08-15T00:26:42.6111126Z     cpp/HashStoreTest 1/1
2025-08-15T00:26:42.6111405Z   Parallel tests (0):
2025-08-15T00:26:42.6111666Z Name: excluded (est. time: 0.0min)
2025-08-15T00:26:42.6111931Z   Serial tests (0):
2025-08-15T00:26:42.6112151Z   Parallel tests (0):
2025-08-15T00:26:42.6112458Z Running cpp/HashStoreTest 1/1 ... [2025-08-15 00:26:42.611116]
2025-08-15T00:26:42.6112820Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:26:42.6117393Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/HashStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-0d1a62809c5f4061.xml', '-x', '--reruns=2'] ... [2025-08-15 00:26:42.611556]
2025-08-15T00:26:44.6295852Z 
2025-08-15T00:26:44.6297149Z cpp/HashStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.HashStoreTest_1.1_00ea5cdca11d6b32_.log
2025-08-15T00:26:44.6298503Z 
2025-08-15T00:26:44.6298811Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:26:44.6299384Z Uploading artifacts took 0.00 seconds
2025-08-15T00:26:48.1297823Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.1299489Z   import pkg_resources
2025-08-15T00:26:48.2082462Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2084043Z   import pkg_resources
2025-08-15T00:26:48.2228703Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2230238Z   import pkg_resources
2025-08-15T00:26:48.2244775Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2245982Z   import pkg_resources
2025-08-15T00:26:48.2262082Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2263623Z   import pkg_resources
2025-08-15T00:26:48.2432642Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2530490Z   import pkg_resources
2025-08-15T00:26:48.2531951Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2533453Z   import pkg_resources
2025-08-15T00:26:48.2546917Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:48.2548180Z   import pkg_resources
2025-08-15T00:26:50.1495725Z Running test batch 'tests to run' cost 7.54 seconds
2025-08-15T00:26:50.7946391Z + python test/run_test.py --cpp --verbose -i cpp/TCPStoreTest
2025-08-15T00:26:53.8060769Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:26:53.8062949Z   import pkg_resources
2025-08-15T00:26:56.1044975Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-15T00:26:56.1158567Z Found test times from artifacts
2025-08-15T00:26:56.1642687Z Found test times from artifacts
2025-08-15T00:26:56.1657178Z Running all tests
2025-08-15T00:26:56.1662625Z Running parallel tests on 8 processes
2025-08-15T00:26:56.1663075Z Name: tests to run (est. time: 0.0min)
2025-08-15T00:26:56.1663398Z   Serial tests (1):
2025-08-15T00:26:56.1663655Z     cpp/TCPStoreTest 1/1
2025-08-15T00:26:56.1663939Z   Parallel tests (0):
2025-08-15T00:26:56.1664197Z Name: excluded (est. time: 0.0min)
2025-08-15T00:26:56.1664480Z   Serial tests (0):
2025-08-15T00:26:56.1664699Z   Parallel tests (0):
2025-08-15T00:26:56.1665109Z Running cpp/TCPStoreTest 1/1 ... [2025-08-15 00:26:56.166409]
2025-08-15T00:26:56.1665479Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:26:56.1671391Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/TCPStoreTest', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-3e36e5f6f1b9177b.xml', '-x', '--reruns=2'] ... [2025-08-15 00:26:56.166909]
2025-08-15T00:27:08.1480122Z 
2025-08-15T00:27:08.1481646Z cpp/TCPStoreTest 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.TCPStoreTest_1.1_0103eebb128b7931_.log
2025-08-15T00:27:08.1482720Z 
2025-08-15T00:27:08.1483131Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:27:08.1483872Z Uploading artifacts took 0.00 seconds
2025-08-15T00:27:11.5842861Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.5844487Z   import pkg_resources
2025-08-15T00:27:11.7701251Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.7702740Z   import pkg_resources
2025-08-15T00:27:11.7773725Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.7775182Z   import pkg_resources
2025-08-15T00:27:11.7942008Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.7943490Z   import pkg_resources
2025-08-15T00:27:11.7946269Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.7947736Z   import pkg_resources
2025-08-15T00:27:11.7957818Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.7959492Z   import pkg_resources
2025-08-15T00:27:11.8033678Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.8035139Z   import pkg_resources
2025-08-15T00:27:11.8041345Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:11.8042639Z   import pkg_resources
2025-08-15T00:27:13.6444433Z Running test batch 'tests to run' cost 17.48 seconds
2025-08-15T00:27:14.3358350Z + echo 'Testing multi-GPU linalg tests'
2025-08-15T00:27:14.3358934Z Testing multi-GPU linalg tests
2025-08-15T00:27:14.3359711Z + python test/run_test.py -i test_linalg.py -k test_matmul_offline_mgpu_tunable --verbose
2025-08-15T00:27:17.3491725Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:17.3493771Z   import pkg_resources
2025-08-15T00:27:19.6259860Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-15T00:27:19.6373074Z Found test times from artifacts
2025-08-15T00:27:19.6860100Z Found test times from artifacts
2025-08-15T00:27:19.6874284Z Running all tests
2025-08-15T00:27:19.6879582Z Running parallel tests on 8 processes
2025-08-15T00:27:19.6880123Z Name: tests to run (est. time: 0.2min)
2025-08-15T00:27:19.6880617Z   Serial tests (1):
2025-08-15T00:27:19.6880968Z     test_linalg 1/1
2025-08-15T00:27:19.6881312Z   Parallel tests (0):
2025-08-15T00:27:19.6881710Z Name: excluded (est. time: 0.0min)
2025-08-15T00:27:19.6882298Z   Serial tests (0):
2025-08-15T00:27:19.6882673Z   Parallel tests (0):
2025-08-15T00:27:19.6883133Z Running test_linalg 1/1 ... [2025-08-15 00:27:19.688016]
2025-08-15T00:27:19.6883682Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:27:19.6886082Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'test_linalg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-k', 'test_matmul_offline_mgpu_tunable', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-15 00:27:19.688359]
2025-08-15T00:27:37.6264645Z 
2025-08-15T00:27:37.6265759Z test_linalg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/test_linalg_1.1_4325435ae53cefa9_.log
2025-08-15T00:27:37.6267523Z Running 1 items in this shard: test/test_linalg.py::TestLinalgCUDA::test_matmul_offline_mgpu_tunableop_cuda_float32
2025-08-15T00:27:37.6268365Z 
2025-08-15T00:27:37.6268836Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:27:37.6269668Z Uploading artifacts took 0.00 seconds
2025-08-15T00:27:41.0381152Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.0382857Z   import pkg_resources
2025-08-15T00:27:41.1215197Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.1217317Z   import pkg_resources
2025-08-15T00:27:41.1837123Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.1838628Z   import pkg_resources
2025-08-15T00:27:41.2041448Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.2042995Z   import pkg_resources
2025-08-15T00:27:41.2089319Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.2090855Z   import pkg_resources
2025-08-15T00:27:41.2149551Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.2150995Z   import pkg_resources
2025-08-15T00:27:41.2208982Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.2210432Z   import pkg_resources
2025-08-15T00:27:41.2799995Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:41.2801563Z   import pkg_resources
2025-08-15T00:27:42.9467976Z Running test batch 'tests to run' cost 23.26 seconds
2025-08-15T00:27:43.6390105Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-08-15T00:27:43.6390623Z + [[ 1 == 1 ]]
2025-08-15T00:27:43.6390916Z + test_rpc
2025-08-15T00:27:43.6391177Z + echo 'Testing RPC C++ tests'
2025-08-15T00:27:43.6391495Z Testing RPC C++ tests
2025-08-15T00:27:43.6395011Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:27:43.6423467Z + ln -sf /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib/libc10_hip.so /opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:27:43.6445825Z + CPP_TESTS_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-15T00:27:43.6446679Z + python test/run_test.py --cpp --verbose -i cpp/test_cpp_rpc
2025-08-15T00:27:46.6095943Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:46.6098919Z   import pkg_resources
2025-08-15T00:27:48.8918889Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-15T00:27:48.9034124Z Found test times from artifacts
2025-08-15T00:27:48.9528026Z Found test times from artifacts
2025-08-15T00:27:48.9542717Z Running all tests
2025-08-15T00:27:48.9548060Z Running parallel tests on 8 processes
2025-08-15T00:27:48.9548404Z Name: tests to run (est. time: 0.0min)
2025-08-15T00:27:48.9548682Z   Serial tests (1):
2025-08-15T00:27:48.9548907Z     cpp/test_cpp_rpc 1/1
2025-08-15T00:27:48.9549147Z   Parallel tests (0):
2025-08-15T00:27:48.9549396Z Name: excluded (est. time: 0.0min)
2025-08-15T00:27:48.9549650Z   Serial tests (0):
2025-08-15T00:27:48.9549897Z   Parallel tests (0):
2025-08-15T00:27:48.9550218Z Running cpp/test_cpp_rpc 1/1 ... [2025-08-15 00:27:48.954913]
2025-08-15T00:27:48.9550579Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-15T00:27:48.9555728Z Executing ['pytest', '/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin/test_cpp_rpc', '-v', '-vv', '-rfEX', '-n', '8', '--junit-xml-reruns', 'test-reports/python-pytest/test.run_test/test.run_test-b76948e2c7f31c83.xml', '-x', '--reruns=2'] ... [2025-08-15 00:27:48.955355]
2025-08-15T00:27:50.0724931Z 
2025-08-15T00:27:50.0726204Z cpp/test_cpp_rpc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/cpp.test_cpp_rpc_1.1_512b235737cf70c6_.log
2025-08-15T00:27:50.0727161Z 
2025-08-15T00:27:50.0727573Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-15T00:27:50.0728318Z Uploading artifacts took 0.00 seconds
2025-08-15T00:27:53.4666392Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.4667993Z   import pkg_resources
2025-08-15T00:27:53.5070301Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.5071826Z   import pkg_resources
2025-08-15T00:27:53.5434085Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.5435566Z   import pkg_resources
2025-08-15T00:27:53.6280188Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.6281640Z   import pkg_resources
2025-08-15T00:27:53.6329299Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.6330741Z   import pkg_resources
2025-08-15T00:27:53.6392296Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.6394234Z   import pkg_resources
2025-08-15T00:27:53.6410343Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.6411811Z   import pkg_resources
2025-08-15T00:27:53.6443540Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-15T00:27:53.6445011Z   import pkg_resources
2025-08-15T00:27:55.1979520Z Running test batch 'tests to run' cost 6.24 seconds
2025-08-15T00:27:55.8578597Z + sccache_epilogue
2025-08-15T00:27:55.8579719Z + echo '::group::Sccache Compilation Log'
2025-08-15T00:27:55.8580745Z ##[group]Sccache Compilation Log
2025-08-15T00:27:55.8581320Z + echo '=================== sccache compilation log ==================='
2025-08-15T00:27:55.8581877Z =================== sccache compilation log ===================
2025-08-15T00:27:55.8582561Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-08-15T00:27:55.8730558Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-08-15T00:27:55.8731678Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-08-15T00:27:55.8732537Z + sccache --show-stats
2025-08-15T00:27:55.8772148Z Compile requests                    806
2025-08-15T00:27:55.8772586Z Compile requests executed             0
2025-08-15T00:27:55.8772925Z Cache hits                            0
2025-08-15T00:27:55.8773253Z Cache misses                          0
2025-08-15T00:27:55.8773583Z Cache hits rate                       -
2025-08-15T00:27:55.8773908Z Cache timeouts                        0
2025-08-15T00:27:55.8774227Z Cache read errors                     0
2025-08-15T00:27:55.8774546Z Forced recaches                       0
2025-08-15T00:27:55.8774859Z Cache write errors                    0
2025-08-15T00:27:55.8775178Z Cache errors                          0
2025-08-15T00:27:55.8775497Z Compilations                          0
2025-08-15T00:27:55.8775824Z Compilation failures                  0
2025-08-15T00:27:55.8776167Z Non-cacheable compilations            0
2025-08-15T00:27:55.8776507Z Non-cacheable calls                   7
2025-08-15T00:27:55.8776840Z Non-compilation calls               799
2025-08-15T00:27:55.8777189Z Unsupported compiler calls            0
2025-08-15T00:27:55.8777534Z Average cache write               0.000 s
2025-08-15T00:27:55.8777876Z Average compiler                  0.000 s
2025-08-15T00:27:55.8778233Z Average cache read hit            0.000 s
2025-08-15T00:27:55.8778590Z Failed distributed compilations       0
2025-08-15T00:27:55.8778830Z 
2025-08-15T00:27:55.8778945Z Non-cacheable reasons:
2025-08-15T00:27:55.8779232Z -E                                    7
2025-08-15T00:27:55.8779457Z 
2025-08-15T00:27:55.8779681Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-08-15T00:27:55.8780145Z Use direct/preprocessor mode?   yes
2025-08-15T00:27:55.8780501Z Version (client)                0.10.0
2025-08-15T00:27:55.8780848Z Max cache size                       10 GiB
2025-08-15T00:27:55.8781200Z + sccache --stop-server
2025-08-15T00:27:55.8810038Z Stopping sccache server...
2025-08-15T00:27:55.8814182Z Compile requests                    806
2025-08-15T00:27:55.8815284Z Compile requests executed             0
2025-08-15T00:27:55.8815762Z Cache hits                            0
2025-08-15T00:27:55.8816119Z Cache misses                          0
2025-08-15T00:27:55.8816460Z Cache hits rate                       -
2025-08-15T00:27:55.8816801Z Cache timeouts                        0
2025-08-15T00:27:55.8817129Z Cache read errors                     0
2025-08-15T00:27:55.8817461Z Forced recaches                       0
2025-08-15T00:27:55.8817791Z Cache write errors                    0
2025-08-15T00:27:55.8818114Z Cache errors                          0
2025-08-15T00:27:55.8818433Z Compilations                          0
2025-08-15T00:27:55.8818752Z Compilation failures                  0
2025-08-15T00:27:55.8819096Z Non-cacheable compilations            0
2025-08-15T00:27:55.8819432Z Non-cacheable calls                   7
2025-08-15T00:27:55.8819763Z Non-compilation calls               799
2025-08-15T00:27:55.8820106Z Unsupported compiler calls            0
2025-08-15T00:27:55.8820456Z Average cache write               0.000 s
2025-08-15T00:27:55.8820811Z Average compiler                  0.000 s
2025-08-15T00:27:55.8821149Z Average cache read hit            0.000 s
2025-08-15T00:27:55.8821813Z Failed distributed compilations       0
2025-08-15T00:27:55.8822072Z 
2025-08-15T00:27:55.8822192Z Non-cacheable reasons:
2025-08-15T00:27:55.8822475Z -E                                    7
2025-08-15T00:27:55.8822700Z 
2025-08-15T00:27:55.8822916Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-08-15T00:27:55.8823401Z Use direct/preprocessor mode?   yes
2025-08-15T00:27:55.8823754Z Version (client)                0.10.0
2025-08-15T00:27:55.8824103Z Max cache size                       10 GiB
2025-08-15T00:27:55.8824844Z + echo ::endgroup::
2025-08-15T00:27:55.8825306Z ##[endgroup]
2025-08-15T00:27:55.8927856Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-08-15T00:27:55.8928633Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-08-15T00:27:55.8929504Z docker exec -t "2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-08-15T00:27:55.8950106Z shell: /usr/bin/bash -e {0}
2025-08-15T00:27:55.8950352Z env:
2025-08-15T00:27:55.8950551Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:27:55.8950927Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:27:55.8951487Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:27:55.8951996Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:27:55.8952870Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:27:55.8953623Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:27:55.8953872Z   AWS_REGION: us-east-1
2025-08-15T00:27:55.8954171Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:27:55.8954496Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:27:55.8959311Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:27:55.8959677Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:27:55.8960074Z ##[endgroup]
2025-08-15T00:27:56.0643243Z ##[group]Run cat test/**/*_toprint.log || true
2025-08-15T00:27:56.0643610Z cat test/**/*_toprint.log || true
2025-08-15T00:27:56.0664492Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:27:56.0664825Z env:
2025-08-15T00:27:56.0665030Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:27:56.0665407Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:27:56.0665957Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:27:56.0666454Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:27:56.0667594Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:27:56.0668343Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:27:56.0668588Z   AWS_REGION: us-east-1
2025-08-15T00:27:56.0668917Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:27:56.0669247Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:27:56.0673953Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:27:56.0674327Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:27:56.0674727Z ##[endgroup]
2025-08-15T00:27:56.0822036Z cat: 'test/**/*_toprint.log': No such file or directory
2025-08-15T00:27:56.0942804Z Prepare all required actions
2025-08-15T00:27:56.0943412Z Getting action download info
2025-08-15T00:27:56.4733104Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-08-15T00:27:57.2285040Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-08-15T00:27:58.0485673Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-08-15T00:27:58.0486002Z with:
2025-08-15T00:27:58.0486197Z   use-gha: true
2025-08-15T00:27:58.0486500Z   file-suffix: test-distributed-1-3-linux.rocm.gpu.4_48128182516
2025-08-15T00:27:58.0486867Z   s3-bucket: gha-artifacts
2025-08-15T00:27:58.0487099Z env:
2025-08-15T00:27:58.0487298Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:27:58.0487670Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:27:58.0488211Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:27:58.0488780Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:27:58.0489643Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:27:58.0490425Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:27:58.0490683Z   AWS_REGION: us-east-1
2025-08-15T00:27:58.0490976Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:27:58.0491318Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:27:58.0496007Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:27:58.0496395Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:27:58.0496806Z ##[endgroup]
2025-08-15T00:27:58.0565981Z ##[group]Run actions/upload-artifact@v4
2025-08-15T00:27:58.0566258Z with:
2025-08-15T00:27:58.0566634Z   name: test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip
2025-08-15T00:27:58.0567098Z   retention-days: 14
2025-08-15T00:27:58.0567333Z   if-no-files-found: warn
2025-08-15T00:27:58.0567568Z   path: test/**/*.json
2025-08-15T00:27:58.0567801Z   compression-level: 6
2025-08-15T00:27:58.0568021Z   overwrite: false
2025-08-15T00:27:58.0568255Z   include-hidden-files: false
2025-08-15T00:27:58.0568490Z env:
2025-08-15T00:27:58.0568683Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:27:58.0569046Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:27:58.0569582Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:27:58.0570086Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:27:58.0570922Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:27:58.0571677Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:27:58.0571929Z   AWS_REGION: us-east-1
2025-08-15T00:27:58.0572196Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:27:58.0572524Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:27:58.0577271Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:27:58.0577643Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:27:58.0578286Z ##[endgroup]
2025-08-15T00:27:58.7981286Z With the provided path, there will be 11 files uploaded
2025-08-15T00:27:58.7987354Z Artifact name is valid!
2025-08-15T00:27:58.7988534Z Root directory input is valid!
2025-08-15T00:27:59.0053782Z Beginning upload of artifact content to blob storage
2025-08-15T00:27:59.3278863Z Uploaded bytes 45092
2025-08-15T00:27:59.3867645Z Finished uploading artifact content to blob storage!
2025-08-15T00:27:59.3870563Z SHA256 digest of uploaded artifact zip is 42e0a41e737e960a65f8cb5422edfa5cc8465697a406b8cd3092e513dfb1f005
2025-08-15T00:27:59.3871693Z Finalizing artifact upload
2025-08-15T00:27:59.4927427Z Artifact test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip.zip successfully finalized. Artifact ID 3770440545
2025-08-15T00:27:59.4928855Z Artifact test-jsons-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip has been successfully uploaded! Final size is 45092 bytes. Artifact ID is 3770440545
2025-08-15T00:27:59.4933362Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770440545
2025-08-15T00:27:59.5196532Z ##[group]Run actions/upload-artifact@v4
2025-08-15T00:27:59.5196876Z with:
2025-08-15T00:27:59.5197372Z   name: test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip
2025-08-15T00:27:59.5197942Z   retention-days: 14
2025-08-15T00:27:59.5198228Z   if-no-files-found: ignore
2025-08-15T00:27:59.5198549Z   path: test/**/*.xml
test/**/*.csv

2025-08-15T00:27:59.5198893Z   compression-level: 6
2025-08-15T00:27:59.5199175Z   overwrite: false
2025-08-15T00:27:59.5199450Z   include-hidden-files: false
2025-08-15T00:27:59.5199749Z env:
2025-08-15T00:27:59.5199994Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:27:59.5200457Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:27:59.5200998Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:27:59.5201502Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:27:59.5202539Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:27:59.5203301Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:27:59.5203549Z   AWS_REGION: us-east-1
2025-08-15T00:27:59.5203826Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:27:59.5204151Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:27:59.5208833Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:27:59.5209205Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:27:59.5209609Z ##[endgroup]
2025-08-15T00:28:00.3200339Z With the provided path, there will be 729 files uploaded
2025-08-15T00:28:00.3205733Z Artifact name is valid!
2025-08-15T00:28:00.3206858Z Root directory input is valid!
2025-08-15T00:28:00.5320209Z Beginning upload of artifact content to blob storage
2025-08-15T00:28:01.3193807Z Uploaded bytes 595289
2025-08-15T00:28:01.3776794Z Finished uploading artifact content to blob storage!
2025-08-15T00:28:01.3779704Z SHA256 digest of uploaded artifact zip is 4b62465c4e2d9ff1741c62db31d4ab8b5f849ff813116adba7812c1266d9df94
2025-08-15T00:28:01.3781112Z Finalizing artifact upload
2025-08-15T00:28:01.4963363Z Artifact test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip.zip successfully finalized. Artifact ID 3770440712
2025-08-15T00:28:01.4965300Z Artifact test-reports-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip has been successfully uploaded! Final size is 595289 bytes. Artifact ID is 3770440712
2025-08-15T00:28:01.4970994Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770440712
2025-08-15T00:28:01.5244414Z ##[group]Run actions/upload-artifact@v4
2025-08-15T00:28:01.5244735Z with:
2025-08-15T00:28:01.5245432Z   name: logs-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip
2025-08-15T00:28:01.5245880Z   retention-days: 14
2025-08-15T00:28:01.5246147Z   if-no-files-found: ignore
2025-08-15T00:28:01.5246424Z   path: usage_log.txt
test/**/*.log

2025-08-15T00:28:01.5246740Z   compression-level: 6
2025-08-15T00:28:01.5246988Z   overwrite: false
2025-08-15T00:28:01.5247228Z   include-hidden-files: false
2025-08-15T00:28:01.5247501Z env:
2025-08-15T00:28:01.5247712Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:01.5248099Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:01.5248681Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:01.5249466Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:01.5250341Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:01.5251113Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:01.5251391Z   AWS_REGION: us-east-1
2025-08-15T00:28:01.5251720Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:01.5252084Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:01.5256809Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:01.5257204Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:01.5257642Z ##[endgroup]
2025-08-15T00:28:02.3213196Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-08-15T00:28:02.3214403Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-08-15T00:28:02.3215163Z With the provided path, there will be 101 files uploaded
2025-08-15T00:28:02.3215755Z Artifact name is valid!
2025-08-15T00:28:02.3216039Z Root directory input is valid!
2025-08-15T00:28:02.5268834Z Beginning upload of artifact content to blob storage
2025-08-15T00:28:03.0905728Z Uploaded bytes 322257
2025-08-15T00:28:03.1540819Z Finished uploading artifact content to blob storage!
2025-08-15T00:28:03.1543939Z SHA256 digest of uploaded artifact zip is a6d0dcf8d7753586e3312e21d5dac4ee64aeb3846be1bcbaca3e00c806443f9a
2025-08-15T00:28:03.1545322Z Finalizing artifact upload
2025-08-15T00:28:03.2723128Z Artifact logs-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip.zip successfully finalized. Artifact ID 3770440872
2025-08-15T00:28:03.2725339Z Artifact logs-runattempt1-test-distributed-1-3-linux.rocm.gpu.4_48128182516.zip has been successfully uploaded! Final size is 322257 bytes. Artifact ID is 3770440872
2025-08-15T00:28:03.2731672Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770440872
2025-08-15T00:28:03.2946848Z ##[group]Run # shellcheck disable=SC2156
2025-08-15T00:28:03.2947236Z # shellcheck disable=SC2156
2025-08-15T00:28:03.2947760Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-08-15T00:28:03.2969499Z shell: /usr/bin/bash -e {0}
2025-08-15T00:28:03.2969767Z env:
2025-08-15T00:28:03.2969990Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:03.2970403Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:03.2970983Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:03.2971531Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:03.2972415Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:03.2973243Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:03.2973568Z   AWS_REGION: us-east-1
2025-08-15T00:28:03.2973885Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:03.2974241Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:03.2979230Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:03.2979630Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:03.2980058Z ##[endgroup]
2025-08-15T00:28:03.7132701Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-08-15T00:28:03.7133210Z with:
2025-08-15T00:28:03.7133592Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
2025-08-15T00:28:03.7134041Z   role-duration-seconds: 18000
2025-08-15T00:28:03.7134352Z   aws-region: us-east-1
2025-08-15T00:28:03.7134624Z   audience: sts.amazonaws.com
2025-08-15T00:28:03.7134900Z env:
2025-08-15T00:28:03.7135115Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:03.7135512Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:03.7136080Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:03.7136605Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:03.7137546Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:03.7138322Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:03.7138594Z   AWS_REGION: us-east-1
2025-08-15T00:28:03.7138946Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:03.7139318Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:03.7144087Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:03.7144489Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:03.7144944Z ##[endgroup]
2025-08-15T00:28:04.0434969Z Assuming role with OIDC
2025-08-15T00:28:04.2614226Z Authenticated as assumedRoleId AROAUPVRELQNA5GQHA6IA:GitHubActions
2025-08-15T00:28:04.3359634Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-08-15T00:28:04.3360118Z with:
2025-08-15T00:28:04.3360412Z   benchmark-results-dir: test/test-reports
2025-08-15T00:28:04.3360744Z   dry-run: false
2025-08-15T00:28:04.3360999Z   schema-version: v3
2025-08-15T00:28:04.3361555Z   github-token: ***
2025-08-15T00:28:04.3361815Z env:
2025-08-15T00:28:04.3362049Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:04.3362572Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:04.3363160Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:04.3363706Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:04.3364598Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:04.3365433Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:04.3365744Z   AWS_REGION: us-east-1
2025-08-15T00:28:04.3366048Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:04.3366422Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:04.3371254Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:04.3371686Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:04.3372140Z ##[endgroup]
2025-08-15T00:28:04.3391529Z ##[group]Run set -eux
2025-08-15T00:28:04.3391842Z set -eux
2025-08-15T00:28:04.3392211Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-08-15T00:28:04.3392617Z 
2025-08-15T00:28:04.3392862Z DEVICE_NAME=""
2025-08-15T00:28:04.3393141Z DEVICE_TYPE=""
2025-08-15T00:28:04.3393399Z 
2025-08-15T00:28:04.3393656Z if command -v nvidia-smi; then
2025-08-15T00:28:04.3394095Z   # NB: I'm using PyTorch here to get the device name, however, it needs to
2025-08-15T00:28:04.3394653Z   # install the correct version of PyTorch manually for now. Any PyTorch
2025-08-15T00:28:04.3395166Z   # version is fine, I just use 2.7.1 to satify PYPIDEP linter
2025-08-15T00:28:04.3395895Z   python3 -mpip install torch==2.7.1
2025-08-15T00:28:04.3396265Z elif command -v rocminfo; then
2025-08-15T00:28:04.3396690Z   # NB: Installing torch on ROCm runner with pip here causes CI to fail
2025-08-15T00:28:04.3397215Z   # with a memoryview is too large error only on MI300 runners. Is pip
2025-08-15T00:28:04.3397734Z   # version on ROCm runner there too old? As a workaround, let's use the
2025-08-15T00:28:04.3398190Z   # GPU device name coming from rocminfo instead
2025-08-15T00:28:04.3398576Z   DEVICE_NAME=rocm
2025-08-15T00:28:04.3399052Z   DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs)
2025-08-15T00:28:04.3399526Z fi
2025-08-15T00:28:04.3399762Z 
2025-08-15T00:28:04.3400057Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV
2025-08-15T00:28:04.3400466Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV
2025-08-15T00:28:04.3422868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:04.3423260Z env:
2025-08-15T00:28:04.3423502Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:04.3423918Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:04.3424509Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:04.3425066Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:04.3425942Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:04.3426740Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:04.3427270Z   AWS_REGION: us-east-1
2025-08-15T00:28:04.3427602Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:04.3427976Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:04.3432811Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:04.3433228Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:04.3433695Z ##[endgroup]
2025-08-15T00:28:04.3485541Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-08-15T00:28:04.6490078Z Defaulting to user installation because normal site-packages is not writeable
2025-08-15T00:28:04.7597191Z Requirement already satisfied: boto3==1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (1.35.33)
2025-08-15T00:28:04.7602209Z Requirement already satisfied: psutil==7.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (7.0.0)
2025-08-15T00:28:04.7608384Z Requirement already satisfied: pynvml==12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (12.0.0)
2025-08-15T00:28:04.7648161Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (0.10.4)
2025-08-15T00:28:04.7654495Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from boto3==1.35.33) (0.10.0)
2025-08-15T00:28:04.7659543Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (1.35.99)
2025-08-15T00:28:04.7841715Z Requirement already satisfied: nvidia-ml-py<13.0.0a0,>=12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from pynvml==12.0.0) (12.575.51)
2025-08-15T00:28:04.7900554Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.26.5)
2025-08-15T00:28:04.7906011Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.2)
2025-08-15T00:28:04.7956564Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.16.0)
2025-08-15T00:28:05.2002851Z + DEVICE_NAME=
2025-08-15T00:28:05.2003297Z + DEVICE_TYPE=
2025-08-15T00:28:05.2003740Z + command -v nvidia-smi
2025-08-15T00:28:05.2004200Z + command -v rocminfo
2025-08-15T00:28:05.2004651Z + DEVICE_NAME=rocm
2025-08-15T00:28:05.2005067Z /opt/rocm/bin/rocminfo
2025-08-15T00:28:05.2016212Z ++ rocminfo
2025-08-15T00:28:05.2017881Z ++ grep 'Marketing Name'
2025-08-15T00:28:05.2021082Z ++ tail -n1
2025-08-15T00:28:05.2022944Z ++ awk -F: '{print $2}'
2025-08-15T00:28:05.2025246Z ++ xargs
2025-08-15T00:28:05.3492461Z + DEVICE_TYPE='AMD Instinct MI250X/MI250'
2025-08-15T00:28:05.3493122Z + echo DEVICE_NAME=rocm
2025-08-15T00:28:05.3493710Z + echo 'DEVICE_TYPE=AMD Instinct MI250X/MI250'
2025-08-15T00:28:05.3517922Z ##[group]Run set -eux
2025-08-15T00:28:05.3518224Z set -eux
2025-08-15T00:28:05.3518474Z 
2025-08-15T00:28:05.3518730Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-08-15T00:28:05.3519081Z   echo "Missing github-token input"
2025-08-15T00:28:05.3519406Z   exit 1
2025-08-15T00:28:05.3519639Z fi
2025-08-15T00:28:05.3547647Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:05.3548016Z env:
2025-08-15T00:28:05.3548247Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:05.3548649Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:05.3549227Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:05.3549755Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:05.3550613Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:05.3551675Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:05.3551992Z   AWS_REGION: us-east-1
2025-08-15T00:28:05.3552309Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:05.3552693Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:05.3557519Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:05.3557948Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:05.3558384Z   DEVICE_NAME: rocm
2025-08-15T00:28:05.3558649Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:05.3559123Z   GITHUB_TOKEN: ***
2025-08-15T00:28:05.3559396Z ##[endgroup]
2025-08-15T00:28:05.3606454Z + [[ -z *** ]]
2025-08-15T00:28:05.3666188Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-08-15T00:28:05.3666629Z with:
2025-08-15T00:28:05.3667008Z   github-token: ***
2025-08-15T00:28:05.3667278Z env:
2025-08-15T00:28:05.3667516Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:05.3667958Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:05.3668585Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:05.3669128Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:05.3670033Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:05.3670838Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:05.3671126Z   AWS_REGION: us-east-1
2025-08-15T00:28:05.3671427Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:05.3671800Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:05.3676642Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:05.3677054Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:05.3677502Z   DEVICE_NAME: rocm
2025-08-15T00:28:05.3677764Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:05.3678069Z ##[endgroup]
2025-08-15T00:28:05.3694985Z ##[group]Run set -eux
2025-08-15T00:28:05.3695280Z set -eux
2025-08-15T00:28:05.3695517Z 
2025-08-15T00:28:05.3695974Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-08-15T00:28:05.3718257Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:05.3718637Z env:
2025-08-15T00:28:05.3718876Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:05.3719291Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:05.3719897Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:05.3720455Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:05.3721343Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:05.3722257Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:05.3722557Z   AWS_REGION: us-east-1
2025-08-15T00:28:05.3722858Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:05.3723224Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:05.3728070Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:05.3728501Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:05.3728954Z   DEVICE_NAME: rocm
2025-08-15T00:28:05.3729235Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:05.3729648Z   GITHUB_TOKEN: ***
2025-08-15T00:28:05.3729932Z ##[endgroup]
2025-08-15T00:28:05.3774036Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16976370303 gpud515
2025-08-15T00:28:07.0769292Z setting job-id=48128182516
2025-08-15T00:28:07.0770332Z setting job-name=linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-15T00:28:07.0922674Z ##[group]Run set -eux
2025-08-15T00:28:07.0923017Z set -eux
2025-08-15T00:28:07.0923275Z 
2025-08-15T00:28:07.0923661Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-08-15T00:28:07.0924190Z   --schema-version "${SCHEMA_VERSION}" \
2025-08-15T00:28:07.0924536Z   --repo "${REPO}" \
2025-08-15T00:28:07.0924850Z   --head-branch "${HEAD_BRANCH}" \
2025-08-15T00:28:07.0925184Z   --head-sha "${HEAD_SHA}" \
2025-08-15T00:28:07.0925525Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-08-15T00:28:07.0925878Z   --run-attempt "${RUN_ATTEMPT}" \
2025-08-15T00:28:07.0926200Z   --job-id "${JOB_ID}" \
2025-08-15T00:28:07.0926511Z   --job-name "${JOB_NAME}"
2025-08-15T00:28:07.0948212Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:07.0948585Z env:
2025-08-15T00:28:07.0948824Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:07.0949266Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:07.0949851Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:07.0950416Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:07.0951296Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:07.0952106Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:07.0952405Z   AWS_REGION: us-east-1
2025-08-15T00:28:07.0952753Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:07.0953118Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:07.0957982Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:07.0958405Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:07.0958860Z   DEVICE_NAME: rocm
2025-08-15T00:28:07.0959129Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:07.0959458Z   SCHEMA_VERSION: v3
2025-08-15T00:28:07.0959711Z   REPO: pytorch/pytorch
2025-08-15T00:28:07.0959988Z   HEAD_BRANCH: refs/heads/main
2025-08-15T00:28:07.0960609Z   HEAD_SHA: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-15T00:28:07.0960946Z   WORKFLOW_RUN_ID: 16976370303
2025-08-15T00:28:07.0961231Z   RUN_ATTEMPT: 1
2025-08-15T00:28:07.0961473Z   JOB_ID: 48128182516
2025-08-15T00:28:07.0961962Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-15T00:28:07.0962571Z ##[endgroup]
2025-08-15T00:28:07.1010819Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 1fc683cf17c8c673044538d10266c00f92987be2 --workflow-id 16976370303 --run-attempt 1 --job-id 48128182516 --job-name 'linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)'
2025-08-15T00:28:07.1347320Z ##[group]Run set -eux
2025-08-15T00:28:07.1347621Z set -eux
2025-08-15T00:28:07.1347933Z 
2025-08-15T00:28:07.1348324Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py"
2025-08-15T00:28:07.1369467Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:07.1369856Z env:
2025-08-15T00:28:07.1370099Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:07.1370521Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:07.1371105Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:07.1371653Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:07.1372772Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:07.1373572Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:07.1373863Z   AWS_REGION: us-east-1
2025-08-15T00:28:07.1374188Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:07.1374599Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:07.1379477Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:07.1379909Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:07.1380349Z   DEVICE_NAME: rocm
2025-08-15T00:28:07.1380625Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:07.1380927Z ##[endgroup]
2025-08-15T00:28:07.1425016Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py
2025-08-15T00:28:08.6511877Z ##[group]Run set -eux
2025-08-15T00:28:08.6512185Z set -eux
2025-08-15T00:28:08.6512430Z 
2025-08-15T00:28:08.6512708Z # TODO (huydhn): Implement this part
2025-08-15T00:28:08.6513120Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-08-15T00:28:08.6534142Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:08.6534547Z env:
2025-08-15T00:28:08.6534796Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:08.6535212Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:08.6535807Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:08.6536358Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:08.6537250Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:08.6538053Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:08.6538349Z   AWS_REGION: us-east-1
2025-08-15T00:28:08.6538698Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:08.6539084Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:08.6543975Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:08.6544413Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:08.6545168Z   DEVICE_NAME: rocm
2025-08-15T00:28:08.6545462Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:08.6545786Z ##[endgroup]
2025-08-15T00:28:08.6589319Z + echo 'dependencies={}'
2025-08-15T00:28:08.6616125Z ##[group]Run set -eux
2025-08-15T00:28:08.6616438Z set -eux
2025-08-15T00:28:08.6616692Z 
2025-08-15T00:28:08.6616968Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-08-15T00:28:08.6617400Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-08-15T00:28:08.6617867Z   # We don't want the job to fail if the directory doesn't exist
2025-08-15T00:28:08.6618273Z   exit 0
2025-08-15T00:28:08.6618510Z fi
2025-08-15T00:28:08.6618747Z 
2025-08-15T00:28:08.6618989Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-08-15T00:28:08.6619487Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-08-15T00:28:08.6620028Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-08-15T00:28:08.6620469Z     --metadata "${BENCHMARK_METADATA}" \
2025-08-15T00:28:08.6620821Z     --runners "${RUNNER_INFO}" \
2025-08-15T00:28:08.6621158Z     --dependencies "${DEPENDENCIES}" \
2025-08-15T00:28:08.6621470Z     --dry-run
2025-08-15T00:28:08.6621721Z else
2025-08-15T00:28:08.6622093Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-08-15T00:28:08.6622630Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-08-15T00:28:08.6623039Z     --metadata "${BENCHMARK_METADATA}" \
2025-08-15T00:28:08.6623386Z     --runners "${RUNNER_INFO}" \
2025-08-15T00:28:08.6623722Z     --dependencies "${DEPENDENCIES}"
2025-08-15T00:28:08.6624273Z fi
2025-08-15T00:28:08.6644413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:08.6644791Z env:
2025-08-15T00:28:08.6645014Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:08.6645445Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:08.6646032Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:08.6646570Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:08.6647456Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:08.6648258Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:08.6648548Z   AWS_REGION: us-east-1
2025-08-15T00:28:08.6648879Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:08.6649252Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:08.6654112Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:08.6654533Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:08.6654988Z   DEVICE_NAME: rocm
2025-08-15T00:28:08.6655270Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:08.6655599Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-08-15T00:28:08.6655913Z   DRY_RUN: false
2025-08-15T00:28:08.6657112Z   BENCHMARK_METADATA: {"timestamp": 1755217687, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182516}
2025-08-15T00:28:08.6658710Z   RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpud515.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]
2025-08-15T00:28:08.6659385Z   DEPENDENCIES: {}
2025-08-15T00:28:08.6659640Z ##[endgroup]
2025-08-15T00:28:08.6705937Z + [[ ! -d test/test-reports ]]
2025-08-15T00:28:08.6706319Z + [[ false == \t\r\u\e ]]
2025-08-15T00:28:08.6709700Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1755217687, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 1, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182516}' --runners '[{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpud515.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]' --dependencies '{}'
2025-08-15T00:28:08.8249138Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/test_cpp_rpc'}], 'excluded': []} from test/test-reports/td_exclusions-9698fc50a1b5ec08dc3d.json is not a benchmark record, skipping
2025-08-15T00:28:08.8251639Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8254164Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/HashStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-a507fa5f99c4841bd1bb.json is not a benchmark record, skipping
2025-08-15T00:28:08.8256584Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8261837Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'test_linalg'}], 'excluded': []} from test/test-reports/td_exclusions-39314337d05733d13222.json is not a benchmark record, skipping
2025-08-15T00:28:08.8263255Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8264682Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/TCPStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-49aed89193d1c9ded7a2.json is not a benchmark record, skipping
2025-08-15T00:28:08.8266103Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8267501Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'cpp/FileStoreTest'}], 'excluded': []} from test/test-reports/td_exclusions-11b7707c55e2377c3c17.json is not a benchmark record, skipping
2025-08-15T00:28:08.8268883Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8309323Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'distributed/fsdp/test_fsdp_apply'}, {'test_file': 'distributed/fsdp/test_distributed_checkpoint'}, {'test_file': 'distributed/test_store'}, {'test_file': 'distributed/test_dynamo_distributed'}, {'test_file': 'distributed/test_inductor_collectives'}, {'test_file': 'distributed/tensor/test_dtensor_compile'}, {'test_file': 'distributed/test_c10d_functional_native'}, {'test_file': 'distributed/tensor/parallel/test_micro_pipeline_tp'}, {'test_file': 'distributed/tensor/debug/test_op_coverage'}, {'test_file': 'distributed/_tools/test_mod_tracker'}, {'test_file': 'distributed/checkpoint/test_nested_dict'}, {'test_file': 'distributed/flight_recorder/test_fr_analysis'}, {'test_file': 'distributed/test_compute_comm_reordering'}, {'test_file': 'distributed/checkpoint/test_consolidate_hf_safetensors'}, {'test_file': 'distributed/_tools/test_runtime_estimator'}, {'test_file': 'distributed/_tools/test_sac_ilp'}, {'test_file': 'distributed/pipelining/test_schedule_multiproc'}, {'test_file': 'distributed/_composable/test_contract'}, {'test_file': 'distributed/_shard/sharded_tensor/test_logger'}, {'test_file': 'distributed/pipelining/test_transformer'}, {'test_file': 'distributed/tensor/test_dtensor'}, {'test_file': 'distributed/tensor/test_attention'}, {'test_file': 'distributed/pipelining/test_pipe'}, {'test_file': 'distributed/_tools/test_fake_collectives'}, {'test_file': 'distributed/pipelining/test_backward'}, {'test_file': 'distributed/checkpoint/_experimental/test_barriers'}, {'test_file': 'distributed/checkpoint/_experimental/test_types'}, {'test_file': 'distributed/_tools/test_sac_estimator'}, {'test_file': 'distributed/tensor/test_xla_integration'}, {'test_file': 'distributed/checkpoint/test_dedup_tensors'}, {'test_file': 'distributed/pipelining/test_unflatten'}, {'test_file': 'distributed/checkpoint/test_format_utils'}, {'test_file': 'distributed/tensor/test_dtensor_ops'}, {'test_file': 'distributed/pipelining/test_microbatch'}, {'test_file': 'distributed/tensor/test_api'}, {'test_file': 'distributed/fsdp/test_fsdp_fx'}, {'test_file': 'distributed/test_nvshmem_triton'}, {'test_file': 'distributed/tensor/test_experimental_ops'}, {'test_file': 'distributed/checkpoint/test_hf_storage'}, {'test_file': 'distributed/tensor/test_common_rules'}, {'test_file': 'distributed/tensor/test_convolution_ops'}, {'test_file': 'distributed/tensor/test_matrix_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_reader'}, {'test_file': 'distributed/_composable/test_composability/test_pp_composability'}, {'test_file': 'distributed/tensor/test_tensor_ops'}, {'test_file': 'distributed/tensor/debug/test_comm_mode_features'}, {'test_file': 'distributed/tensor/experimental/test_register_sharding'}, {'test_file': 'distributed/checkpoint/_experimental/test_builder'}, {'test_file': 'distributed/tensor/test_init'}, {'test_file': 'distributed/optim/test_named_optimizer'}, {'test_file': 'distributed/_composable/test_replicate_with_fsdp'}, {'test_file': 'distributed/tensor/parallel/test_tp_style'}, {'test_file': 'distributed/_tools/test_memory_tracker'}, {'test_file': 'distributed/_composable/test_checkpoint'}, {'test_file': 'distributed/checkpoint/_experimental/test_staging'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_writer'}, {'test_file': 'distributed/tensor/test_pointwise_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpointer'}, {'test_file': 'distributed/checkpoint/fsdp/test_fsdp_dsd'}, {'test_file': 'distributed/tensor/experimental/test_local_map'}, {'test_file': 'distributed/checkpoint/e2e/test_fsdp_ep'}, {'test_file': 'distributed/test_functional_api'}, {'test_file': 'distributed/tensor/test_random_ops'}, {'test_file': 'distributed/_tools/test_fsdp2_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_traverse'}, {'test_file': 'distributed/checkpoint/test_fsspec'}, {'test_file': 'distributed/tensor/test_redistribute'}, {'test_file': 'distributed/checkpoint/test_async_process_executor'}, {'test_file': 'distributed/tensor/test_math_ops'}, {'test_file': 'distributed/test_backends'}, {'test_file': 'distributed/tensor/experimental/test_tp_transform'}, {'test_file': 'distributed/test_device_mesh'}, {'test_file': 'distributed/checkpoint/test_fsdp_optim_state'}, {'test_file': 'distributed/checkpoint/test_dtensor_resharding'}, {'test_file': 'distributed/test_symmetric_memory'}, {'test_file': 'distributed/checkpoint/e2e/test_fine_tuning'}, {'test_file': 'distributed/pipelining/test_stage'}, {'test_file': 'distributed/_tools/test_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_tp_checkpoint'}, {'test_file': 'distributed/pipelining/test_schedule'}, {'test_file': 'distributed/test_nvshmem'}, {'test_file': 'distributed/tensor/test_embedding_ops'}, {'test_file': 'distributed/checkpoint/test_save_load_api'}, {'test_file': 'distributed/test_control_collectives'}, {'test_file': 'distributed/checkpoint/test_hf_safetensor_e2e'}, {'test_file': 'distributed/checkpoint/e2e/test_e2e_save_and_load'}, {'test_file': 'distributed/checkpoint/test_planner'}, {'test_file': 'distributed/launcher/test_run'}, {'test_file': 'distributed/checkpoint/test_hsdp_checkpoint'}, {'test_file': 'distributed/checkpoint/test_dtensor_checkpoint'}, {'test_file': 'distributed/tensor/test_view_ops'}, {'test_file': 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion'}, {'test_file': 'distributed/tensor/test_optimizers'}, {'test_file': 'distributed/test_p2p_ipc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_ignore_params'}, {'test_file': 'distributed/checkpoint/test_compatibility'}, {'test_file': 'distributed/tensor/parallel/test_tp_examples'}, {'test_file': 'distributed/tensor/test_utils'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_overlap'}, {'test_file': 'distributed/tensor/parallel/test_tp_random_state'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_logging'}, {'test_file': 'distributed/optim/test_apply_optimizer_in_backward'}, {'test_file': 'distributed/_composable/test_composability/test_2d_composability'}, {'test_file': 'distributed/_composable/test_replicate_with_compiler'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_compile'}, {'test_file': 'distributed/checkpoint/test_fsdp_model_state'}, {'test_file': 'distributed/fsdp/test_shard_utils'}, {'test_file': 'distributed/elastic/test_control_plane'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_grad_scaler'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_process'}, {'test_file': 'distributed/tensor/parallel/test_parallelize_api'}, {'test_file': 'distributed/fsdp/test_fsdp_optim_state'}, {'test_file': 'distributed/tensor/test_op_strategy'}, {'test_file': 'distributed/checkpoint/test_checkpoint'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_'}, {'test_file': 'distributed/checkpoint/test_pg_transport'}, {'test_file': 'distributed/test_collective_utils'}, {'test_file': 'distributed/tensor/debug/test_comm_mode'}, {'test_file': 'distributed/fsdp/test_fsdp_pure_fp16'}, {'test_file': 'distributed/checkpoint/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_state_dict'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state'}, {'test_file': 'distributed/test_dist2'}, {'test_file': 'distributed/fsdp/test_fsdp_memory'}, {'test_file': 'distributed/_composable/test_replicate'}, {'test_file': 'distributed/test_fake_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_backward_prefetch'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_autograd'}, {'test_file': 'distributed/checkpoint/test_state_dict_stager'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_frozen'}, {'test_file': 'distributed/fsdp/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_overlap'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_init'}, {'test_file': 'distributed/fsdp/test_checkpoint_wrapper'}, {'test_file': 'distributed/fsdp/test_fsdp_grad_acc'}, {'test_file': 'distributed/fsdp/test_fsdp_flatten_params'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_mixed_precision'}, {'test_file': 'distributed/test_c10d_logger'}, {'test_file': 'distributed/test_c10d_object_collectives'}, {'test_file': 'distributed/test_serialization'}, {'test_file': 'distributed/checkpoint/test_state_dict'}, {'test_file': 'distributed/optim/test_zero_redundancy_optimizer'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_comm'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint_cpu'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_extensions'}, {'test_file': 'distributed/fsdp/test_fsdp_ignored_modules'}, {'test_file': 'distributed/fsdp/test_fsdp_freezing_weights'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint'}, {'test_file': 'distributed/test_nccl'}, {'test_file': 'distributed/fsdp/test_fsdp_checkpoint'}, {'test_file': 'distributed/test_distributed_spawn'}, {'test_file': 'distributed/fsdp/test_fsdp_unshard_params'}, {'test_file': 'distributed/test_multi_threaded_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_tp_integration'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_training'}, {'test_file': 'distributed/fsdp/test_fsdp_dtensor_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_fine_tune'}, {'test_file': 'distributed/fsdp/test_fsdp_hybrid_shard'}, {'test_file': 'distributed/fsdp/test_fsdp_comm_hooks'}, {'test_file': 'distributed/fsdp/test_fsdp_exec_order'}, {'test_file': 'distributed/fsdp/test_fsdp_misc'}, {'test_file': 'distributed/fsdp/test_wrap'}, {'test_file': 'distributed/fsdp/test_fsdp_clip_grad_norm'}, {'test_file': 'distributed/fsdp/test_fsdp_meta'}, {'test_file': 'distributed/_shard/sharding_spec/test_sharding_spec'}, {'test_file': 'distributed/fsdp/test_hsdp_dtensor_state_dict'}, {'test_file': 'distributed/checkpoint/test_state_dict_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_uneven'}, {'test_file': 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks'}, {'test_file': 'distributed/algorithms/quantization/test_quantization'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_comm'}, {'test_file': 'distributed/fsdp/test_fsdp_input'}, {'test_file': 'distributed/fsdp/test_fsdp_traversal'}, {'test_file': 'distributed/fsdp/test_fsdp_sharded_grad_scaler'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_wrapping'}, {'test_file': 'distributed/_shard/test_sharder'}, {'test_file': 'distributed/algorithms/test_join'}, {'test_file': 'distributed/test_data_parallel'}, {'test_file': 'distributed/_shard/sharding_plan/test_sharding_plan'}, {'test_file': 'distributed/nn/jit/test_instantiator'}, {'test_file': 'distributed/test_pg_wrapper'}, {'test_file': 'distributed/test_c10d_pypg'}, {'test_file': 'distributed/fsdp/test_fsdp_use_orig_params'}, {'test_file': 'distributed/test_composability'}, {'test_file': 'distributed/fsdp/test_fsdp_core'}, {'test_file': 'distributed/_shard/sharded_optim/test_sharded_optim'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_forward'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_tensor_ops'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_binary_cmp'}, {'test_file': 'distributed/fsdp/test_fsdp_mixed_precision'}, {'test_file': 'distributed/test_c10d_ucc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_memory'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_init'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard'}, {'test_file': 'distributed/test_c10d_gloo'}, {'test_file': 'distributed/test_c10d_common'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding_bag'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor'}, {'test_file': 'distributed/test_c10d_spawn_gloo'}, {'test_file': 'distributed/test_c10d_spawn_nccl'}, {'test_file': 'distributed/test_launcher'}, {'test_file': 'distributed/test_c10d_spawn_ucc'}, {'test_file': 'distributed/test_c10d_nccl'}, {'test_file': 'distributed/test_c10d_ops_nccl'}, {'test_file': 'distributed/elastic/events/lib_test'}, {'test_file': 'distributed/elastic/metrics/api_test'}, {'test_file': 'distributed/elastic/multiprocessing/api_test'}, {'test_file': 'distributed/elastic/timer/api_test'}, {'test_file': 'distributed/elastic/timer/local_timer_example'}, {'test_file': 'distributed/elastic/timer/local_timer_test'}, {'test_file': 'distributed/elastic/utils/distributed_test'}, {'test_file': 'distributed/elastic/utils/logging_test'}, {'test_file': 'distributed/elastic/utils/util_test'}, {'test_file': 'distributed/tensor/test_dtensor_dispatch_overhead'}], 'excluded': []} from test/test-reports/td_exclusions-6fcc6c2419a10c464f36.json is not a benchmark record, skipping
2025-08-15T00:28:08.8349618Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-15T00:28:08.8497852Z Prepare all required actions
2025-08-15T00:28:08.8498330Z Getting action download info
2025-08-15T00:28:08.8543609Z ##[group]Run ./.github/actions/teardown-rocm
2025-08-15T00:28:08.8544022Z env:
2025-08-15T00:28:08.8544278Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:08.8544713Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:08.8545337Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:08.8545902Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:08.8546794Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:08.8547595Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:08.8547895Z   AWS_REGION: us-east-1
2025-08-15T00:28:08.8548226Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:08.8548649Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:08.8553483Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:08.8553902Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:08.8554357Z   DEVICE_NAME: rocm
2025-08-15T00:28:08.8554627Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:08.8554931Z ##[endgroup]
2025-08-15T00:28:08.8571550Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-08-15T00:28:08.8572066Z # ignore expansion of "docker ps -q" since it could be empty
2025-08-15T00:28:08.8572474Z # shellcheck disable=SC2046
2025-08-15T00:28:08.8572808Z docker stop $(docker ps -q) || true
2025-08-15T00:28:08.8573178Z # Prune all stopped containers.
2025-08-15T00:28:08.8573510Z docker container prune -f
2025-08-15T00:28:08.8593387Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:08.8593754Z env:
2025-08-15T00:28:08.8593987Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:08.8594408Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:08.8594989Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:08.8595524Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:08.8596398Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:08.8597187Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:08.8597475Z   AWS_REGION: us-east-1
2025-08-15T00:28:08.8597794Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:08.8598156Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:08.8603136Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:08.8603550Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:08.8603999Z   DEVICE_NAME: rocm
2025-08-15T00:28:08.8604286Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:08.8604595Z ##[endgroup]
2025-08-15T00:28:19.4460899Z 2b4c5cb24bd0
2025-08-15T00:28:27.3395042Z Deleted Containers:
2025-08-15T00:28:27.3395824Z 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:27.3396362Z 
2025-08-15T00:28:27.3396571Z Total reclaimed space: 9.369GB
2025-08-15T00:28:27.3460053Z Prepare all required actions
2025-08-15T00:28:27.3489282Z ##[group]Run ./.github/actions/diskspace-cleanup
2025-08-15T00:28:27.3489610Z with:
2025-08-15T00:28:27.3489859Z   diskspace-cutoff: 70
2025-08-15T00:28:27.3490123Z env:
2025-08-15T00:28:27.3490348Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:27.3490759Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:27.3491336Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:27.3491904Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:27.3493336Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:27.3494130Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:27.3494430Z   AWS_REGION: us-east-1
2025-08-15T00:28:27.3494763Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:27.3495127Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:27.3499969Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:27.3500395Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:27.3500837Z   DEVICE_NAME: rocm
2025-08-15T00:28:27.3501101Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:27.3501403Z ##[endgroup]
2025-08-15T00:28:27.3516886Z ##[group]Run set -ex
2025-08-15T00:28:27.3517177Z set -ex
2025-08-15T00:28:27.3517423Z diskspace_cutoff=70
2025-08-15T00:28:27.3517800Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-08-15T00:28:27.3518208Z if [ ! -d "$docker_root_dir" ]; then
2025-08-15T00:28:27.3518720Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-08-15T00:28:27.3519209Z     exit 0
2025-08-15T00:28:27.3519455Z fi
2025-08-15T00:28:27.3519883Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-15T00:28:27.3520705Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-15T00:28:27.3521399Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-08-15T00:28:27.3521773Z     docker system prune -af
2025-08-15T00:28:27.3522396Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-15T00:28:27.3522949Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-08-15T00:28:27.3523531Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-08-15T00:28:27.3524039Z         echo "$msg"
2025-08-15T00:28:27.3524314Z         exit 1
2025-08-15T00:28:27.3524578Z     else
2025-08-15T00:28:27.3524870Z         difference=$((diskspace - diskspace_new))
2025-08-15T00:28:27.3525266Z         echo "Diskspace saved: $difference percent"
2025-08-15T00:28:27.3525604Z     fi
2025-08-15T00:28:27.3525823Z fi
2025-08-15T00:28:27.3548520Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-15T00:28:27.3548908Z env:
2025-08-15T00:28:27.3549134Z   GIT_DEFAULT_BRANCH: main
2025-08-15T00:28:27.3549546Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-15T00:28:27.3550131Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-15T00:28:27.3550677Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-15T00:28:27.3551555Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-15T00:28:27.3552346Z   AWS_DEFAULT_REGION: us-east-1
2025-08-15T00:28:27.3552634Z   AWS_REGION: us-east-1
2025-08-15T00:28:27.3552945Z   AWS_ACCESS_KEY_ID: ***
2025-08-15T00:28:27.3553317Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-15T00:28:27.3558153Z   AWS_SESSION_TOKEN: ***
2025-08-15T00:28:27.3558566Z   CONTAINER_NAME: 2b4c5cb24bd03d6053bb698f81e773e0b10942205cb7a78eb6657c1a2308dfac
2025-08-15T00:28:27.3559014Z   DEVICE_NAME: rocm
2025-08-15T00:28:27.3559281Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-15T00:28:27.3559591Z ##[endgroup]
2025-08-15T00:28:27.3607873Z + diskspace_cutoff=70
2025-08-15T00:28:27.3614276Z ++ docker info -f '{{.DockerRootDir}}'
2025-08-15T00:28:27.4178048Z + docker_root_dir=/media/4TB/docker-rootless
2025-08-15T00:28:27.4179389Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-08-15T00:28:27.4189470Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-08-15T00:28:27.4191021Z ++ sed -n 2p
2025-08-15T00:28:27.4193612Z ++ sed s/%//
2025-08-15T00:28:27.4194834Z ++ sed 's/ //'
2025-08-15T00:28:27.4220593Z + diskspace=15
2025-08-15T00:28:27.4221595Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-08-15T00:28:27.4222667Z + [[ 15 -ge 70 ]]
2025-08-15T00:28:27.4267691Z Post job cleanup.
2025-08-15T00:28:27.4314184Z Post job cleanup.
2025-08-15T00:28:27.5607452Z Post job cleanup.
2025-08-15T00:28:27.5995514Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-15T00:28:27.6379451Z Post job cleanup.
2025-08-15T00:28:27.7735391Z Post job cleanup.
2025-08-15T00:28:27.7778874Z Post job cleanup.
2025-08-15T00:28:27.8793323Z [command]/usr/bin/git version
2025-08-15T00:28:27.8835788Z git version 2.34.1
2025-08-15T00:28:27.8872085Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/38eaeec3-5452-4110-a9e7-fc246606c14d/.gitconfig'
2025-08-15T00:28:27.8881698Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/38eaeec3-5452-4110-a9e7-fc246606c14d' before making global git config changes
2025-08-15T00:28:27.8882692Z Adding repository directory to the temporary git global config as a safe directory
2025-08-15T00:28:27.8886465Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-15T00:28:27.8934609Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-08-15T00:28:27.8976342Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-08-15T00:28:27.9361218Z Entering 'android/libs/fbjni'
2025-08-15T00:28:27.9445389Z Entering 'third_party/FP16'
2025-08-15T00:28:27.9515998Z Entering 'third_party/FXdiv'
2025-08-15T00:28:27.9587903Z Entering 'third_party/NNPACK'
2025-08-15T00:28:27.9658858Z Entering 'third_party/NVTX'
2025-08-15T00:28:27.9733959Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-15T00:28:27.9808267Z Entering 'third_party/XNNPACK'
2025-08-15T00:28:27.9890683Z Entering 'third_party/aiter'
2025-08-15T00:28:27.9956496Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-15T00:28:28.0047539Z Entering 'third_party/benchmark'
2025-08-15T00:28:28.0116324Z Entering 'third_party/composable_kernel'
2025-08-15T00:28:28.0195598Z Entering 'third_party/cpp-httplib'
2025-08-15T00:28:28.0266676Z Entering 'third_party/cpuinfo'
2025-08-15T00:28:28.0338072Z Entering 'third_party/cudnn_frontend'
2025-08-15T00:28:28.0413231Z Entering 'third_party/cutlass'
2025-08-15T00:28:28.0488998Z Entering 'third_party/fbgemm'
2025-08-15T00:28:28.0558442Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-15T00:28:28.0633219Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-15T00:28:28.0715468Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-15T00:28:28.0783362Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-15T00:28:28.0869159Z Entering 'third_party/fbgemm/external/googletest'
2025-08-15T00:28:28.0936814Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-15T00:28:28.0996182Z Entering 'third_party/fbgemm/external/json'
2025-08-15T00:28:28.1070109Z Entering 'third_party/flash-attention'
2025-08-15T00:28:28.1136978Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-15T00:28:28.1214285Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-15T00:28:28.1291049Z Entering 'third_party/flatbuffers'
2025-08-15T00:28:28.1360247Z Entering 'third_party/fmt'
2025-08-15T00:28:28.1431345Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-15T00:28:28.1498915Z Entering 'third_party/gloo'
2025-08-15T00:28:28.1575914Z Entering 'third_party/googletest'
2025-08-15T00:28:28.1635872Z Entering 'third_party/ideep'
2025-08-15T00:28:28.1703624Z Entering 'third_party/ideep/mkl-dnn'
2025-08-15T00:28:28.1787671Z Entering 'third_party/ittapi'
2025-08-15T00:28:28.1859647Z Entering 'third_party/kineto'
2025-08-15T00:28:28.1935500Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-15T00:28:28.2001659Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-15T00:28:28.2076963Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-15T00:28:28.2145434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-15T00:28:28.2219351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-15T00:28:28.2276636Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-15T00:28:28.2343562Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-15T00:28:28.2411036Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-15T00:28:28.2472312Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-15T00:28:28.2539857Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-15T00:28:28.2602725Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-15T00:28:28.2670268Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-15T00:28:28.2735562Z Entering 'third_party/kleidiai'
2025-08-15T00:28:28.2801532Z Entering 'third_party/mimalloc'
2025-08-15T00:28:28.2874108Z Entering 'third_party/nlohmann'
2025-08-15T00:28:28.2944444Z Entering 'third_party/onnx'
2025-08-15T00:28:28.3037064Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-15T00:28:28.3107385Z Entering 'third_party/opentelemetry-cpp'
2025-08-15T00:28:28.3181560Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-15T00:28:28.3248979Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-15T00:28:28.3313996Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-15T00:28:28.3378276Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-15T00:28:28.3439826Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-15T00:28:28.3501985Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-15T00:28:28.3566659Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-15T00:28:28.3622054Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-15T00:28:28.3684906Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-15T00:28:28.3749101Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-15T00:28:28.3837459Z Entering 'third_party/pocketfft'
2025-08-15T00:28:28.3906598Z Entering 'third_party/protobuf'
2025-08-15T00:28:28.3980894Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-15T00:28:28.4049474Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-15T00:28:28.4116346Z Entering 'third_party/psimd'
2025-08-15T00:28:28.4186993Z Entering 'third_party/pthreadpool'
2025-08-15T00:28:28.4259856Z Entering 'third_party/pybind11'
2025-08-15T00:28:28.4330223Z Entering 'third_party/python-peachpy'
2025-08-15T00:28:28.4398763Z Entering 'third_party/sleef'
2025-08-15T00:28:28.4466476Z Entering 'third_party/tensorpipe'
2025-08-15T00:28:28.4533327Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-15T00:28:28.4603655Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-15T00:28:28.4666249Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-15T00:28:28.4732061Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-15T00:28:28.4790999Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-15T00:28:28.4884922Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-08-15T00:28:28.4911787Z http.https://github.com/.extraheader
2025-08-15T00:28:28.4922566Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-08-15T00:28:28.4962433Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-08-15T00:28:28.5333965Z Entering 'android/libs/fbjni'
2025-08-15T00:28:28.5377941Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5431615Z Entering 'third_party/FP16'
2025-08-15T00:28:28.5468920Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5523172Z Entering 'third_party/FXdiv'
2025-08-15T00:28:28.5565150Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5616604Z Entering 'third_party/NNPACK'
2025-08-15T00:28:28.5657793Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5710679Z Entering 'third_party/NVTX'
2025-08-15T00:28:28.5748087Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5804985Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-15T00:28:28.5842663Z http.https://github.com/.extraheader
2025-08-15T00:28:28.5896131Z Entering 'third_party/XNNPACK'
2025-08-15T00:28:28.5939629Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6010149Z Entering 'third_party/aiter'
2025-08-15T00:28:28.6049728Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6103294Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-15T00:28:28.6140821Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6205753Z Entering 'third_party/benchmark'
2025-08-15T00:28:28.6248682Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6296504Z Entering 'third_party/composable_kernel'
2025-08-15T00:28:28.6339630Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6403469Z Entering 'third_party/cpp-httplib'
2025-08-15T00:28:28.6441202Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6496304Z Entering 'third_party/cpuinfo'
2025-08-15T00:28:28.6540120Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6593306Z Entering 'third_party/cudnn_frontend'
2025-08-15T00:28:28.6628849Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6683059Z Entering 'third_party/cutlass'
2025-08-15T00:28:28.6722799Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6788620Z Entering 'third_party/fbgemm'
2025-08-15T00:28:28.6828513Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6877235Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-15T00:28:28.6910531Z http.https://github.com/.extraheader
2025-08-15T00:28:28.6957858Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-15T00:28:28.6997570Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7058811Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-15T00:28:28.7100433Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7153158Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-15T00:28:28.7189661Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7248284Z Entering 'third_party/fbgemm/external/googletest'
2025-08-15T00:28:28.7279043Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7331889Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-15T00:28:28.7368320Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7416852Z Entering 'third_party/fbgemm/external/json'
2025-08-15T00:28:28.7453122Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7505181Z Entering 'third_party/flash-attention'
2025-08-15T00:28:28.7543940Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7595387Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-15T00:28:28.7627628Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7684397Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-15T00:28:28.7721786Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7786230Z Entering 'third_party/flatbuffers'
2025-08-15T00:28:28.7823880Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7878813Z Entering 'third_party/fmt'
2025-08-15T00:28:28.7920023Z http.https://github.com/.extraheader
2025-08-15T00:28:28.7972492Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-15T00:28:28.8012515Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8060389Z Entering 'third_party/gloo'
2025-08-15T00:28:28.8103541Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8155154Z Entering 'third_party/googletest'
2025-08-15T00:28:28.8192422Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8242260Z Entering 'third_party/ideep'
2025-08-15T00:28:28.8277713Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8327047Z Entering 'third_party/ideep/mkl-dnn'
2025-08-15T00:28:28.8364103Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8431006Z Entering 'third_party/ittapi'
2025-08-15T00:28:28.8470947Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8521394Z Entering 'third_party/kineto'
2025-08-15T00:28:28.8561723Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8617920Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-15T00:28:28.8652689Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8699703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-15T00:28:28.8734029Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8783411Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-15T00:28:28.8821068Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8869276Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-15T00:28:28.8904029Z http.https://github.com/.extraheader
2025-08-15T00:28:28.8952385Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-15T00:28:28.8982209Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9028938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-15T00:28:28.9063855Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9115605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-15T00:28:28.9153407Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9209446Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-15T00:28:28.9238716Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9287414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-15T00:28:28.9316542Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9371903Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-15T00:28:28.9406851Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9457406Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-15T00:28:28.9496874Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9546644Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-15T00:28:28.9583115Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9635041Z Entering 'third_party/kleidiai'
2025-08-15T00:28:28.9672029Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9725623Z Entering 'third_party/mimalloc'
2025-08-15T00:28:28.9757381Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9812946Z Entering 'third_party/nlohmann'
2025-08-15T00:28:28.9856529Z http.https://github.com/.extraheader
2025-08-15T00:28:28.9911412Z Entering 'third_party/onnx'
2025-08-15T00:28:28.9946374Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0012889Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-15T00:28:29.0048389Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0100898Z Entering 'third_party/opentelemetry-cpp'
2025-08-15T00:28:29.0138494Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0191062Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-15T00:28:29.0225746Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0271543Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-15T00:28:29.0305630Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0353895Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-15T00:28:29.0386405Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0433420Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-15T00:28:29.0466787Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0520567Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-15T00:28:29.0553716Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0605872Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-15T00:28:29.0638255Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0688223Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-15T00:28:29.0724899Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0773338Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-15T00:28:29.0809069Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0858515Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-15T00:28:29.0897462Z http.https://github.com/.extraheader
2025-08-15T00:28:29.0953388Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-15T00:28:29.0987669Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1057288Z Entering 'third_party/pocketfft'
2025-08-15T00:28:29.1099433Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1150797Z Entering 'third_party/protobuf'
2025-08-15T00:28:29.1186421Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1238362Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-15T00:28:29.1276598Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1332706Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-15T00:28:29.1367192Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1419477Z Entering 'third_party/psimd'
2025-08-15T00:28:29.1460403Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1512200Z Entering 'third_party/pthreadpool'
2025-08-15T00:28:29.1548333Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1606947Z Entering 'third_party/pybind11'
2025-08-15T00:28:29.1641915Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1697052Z Entering 'third_party/python-peachpy'
2025-08-15T00:28:29.1740107Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1792273Z Entering 'third_party/sleef'
2025-08-15T00:28:29.1829524Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1883250Z Entering 'third_party/tensorpipe'
2025-08-15T00:28:29.1922664Z http.https://github.com/.extraheader
2025-08-15T00:28:29.1975627Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-15T00:28:29.2009349Z http.https://github.com/.extraheader
2025-08-15T00:28:29.2057534Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-15T00:28:29.2092727Z http.https://github.com/.extraheader
2025-08-15T00:28:29.2139520Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-15T00:28:29.2180386Z http.https://github.com/.extraheader
2025-08-15T00:28:29.2231460Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-15T00:28:29.2268332Z http.https://github.com/.extraheader
2025-08-15T00:28:29.2315343Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-15T00:28:29.2349987Z http.https://github.com/.extraheader
2025-08-15T00:28:29.2582564Z Cleaning up orphan processes