2025-08-14T21:19:20.5503586Z Current runner version: '2.327.1'
2025-08-14T21:19:20.5509995Z Runner name: 'gpud535'
2025-08-14T21:19:20.5510758Z Runner group name: 'linux.rocm.gpu.group'
2025-08-14T21:19:20.5511849Z Machine name: 'gpud535'
2025-08-14T21:19:20.5518201Z ##[group]GITHUB_TOKEN Permissions
2025-08-14T21:19:20.5522338Z Contents: read
2025-08-14T21:19:20.5523532Z Metadata: read
2025-08-14T21:19:20.5524835Z ##[endgroup]
2025-08-14T21:19:20.5529318Z Secret source: Actions
2025-08-14T21:19:20.5530708Z Prepare workflow directory
2025-08-14T21:19:21.2498309Z Prepare all required actions
2025-08-14T21:19:21.2542019Z Getting action download info
2025-08-14T21:19:21.6433805Z Download action repository 'pytorch/pytorch@main' (SHA:3be70dc30e893b552fc0f23ca06cd8f7949b6d08)
2025-08-14T21:19:27.0077265Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722)
2025-08-14T21:19:27.7616971Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076)
2025-08-14T21:19:28.4242188Z Download action repository 'pytorch/test-infra@main' (SHA:83f58f391e939c10dcb8cb6d745e4cefa3b98a84)
2025-08-14T21:19:29.6888995Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-08-14T21:19:30.6803501Z Getting action download info
2025-08-14T21:19:30.8242902Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955)
2025-08-14T21:19:31.5897259Z Getting action download info
2025-08-14T21:19:31.7375977Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e)
2025-08-14T21:19:32.4506065Z Getting action download info
2025-08-14T21:19:32.6333392Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/main (1fc683cf17c8c673044538d10266c00f92987be2)
2025-08-14T21:19:32.6337509Z ##[group] Inputs
2025-08-14T21:19:32.6337834Z   build-environment: linux-jammy-rocm-py3.10
2025-08-14T21:19:32.6339022Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:19:32.6340384Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:32.6340952Z   sync-tag: 
2025-08-14T21:19:32.6341637Z   timeout-minutes: 300
2025-08-14T21:19:32.6341869Z   tests-to-include: 
2025-08-14T21:19:32.6342071Z   dashboard-tag: 
2025-08-14T21:19:32.6342528Z   disable-monitor: true
2025-08-14T21:19:32.6342784Z   monitor-log-interval: 5
2025-08-14T21:19:32.6343039Z   monitor-data-collect-interval: 1
2025-08-14T21:19:32.6343279Z ##[endgroup]
2025-08-14T21:19:32.6343679Z Complete job name: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:19:32.8036726Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main
2025-08-14T21:19:32.8037342Z with:
2025-08-14T21:19:32.8037508Z   no-sudo: true
2025-08-14T21:19:32.8037703Z   submodules: recursive
2025-08-14T21:19:32.8037906Z   fetch-depth: 0
2025-08-14T21:19:32.8038223Z env:
2025-08-14T21:19:32.8038384Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:32.8038582Z ##[endgroup]
2025-08-14T21:19:32.8110912Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-08-14T21:19:32.8111653Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT"
2025-08-14T21:19:32.8148580Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:32.8149194Z env:
2025-08-14T21:19:32.8149350Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:32.8149544Z ##[endgroup]
2025-08-14T21:19:32.8500602Z ##[group]Run # Use all available CPUs for fetching
2025-08-14T21:19:32.8501426Z # Use all available CPUs for fetching
2025-08-14T21:19:32.8502048Z cd "${GITHUB_WORKSPACE}"
2025-08-14T21:19:32.8502664Z git config --global fetch.parallel 0
2025-08-14T21:19:32.8503346Z git config --global submodule.fetchJobs 0
2025-08-14T21:19:32.8503947Z 
2025-08-14T21:19:32.8504569Z # Clean workspace. The default checkout action should also do this, but
2025-08-14T21:19:32.8505411Z # do it here as well just in case
2025-08-14T21:19:32.8505969Z if [[ -d .git ]]; then
2025-08-14T21:19:32.8506482Z   if [ -z "${NO_SUDO}" ]; then
2025-08-14T21:19:32.8507023Z     sudo git clean -ffdx
2025-08-14T21:19:32.8507519Z   else
2025-08-14T21:19:32.8507952Z     git clean -ffdx
2025-08-14T21:19:32.8508392Z   fi
2025-08-14T21:19:32.8508763Z fi
2025-08-14T21:19:32.8581234Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:32.8581841Z env:
2025-08-14T21:19:32.8582160Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:32.8582547Z   NO_SUDO: true
2025-08-14T21:19:32.8583009Z ##[endgroup]
2025-08-14T21:19:33.3921825Z Removing .additional_ci_files/
2025-08-14T21:19:33.3922577Z Removing build/
2025-08-14T21:19:33.3922976Z Removing dist/
2025-08-14T21:19:33.3923403Z Removing test/test-reports/
2025-08-14T21:19:33.4049321Z ##[group]Run actions/checkout@v4
2025-08-14T21:19:33.4049835Z with:
2025-08-14T21:19:33.4050259Z   ref: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:19:33.4050806Z   fetch-depth: 0
2025-08-14T21:19:33.4051203Z   submodules: recursive
2025-08-14T21:19:33.4051625Z   show-progress: false
2025-08-14T21:19:33.4052068Z   repository: pytorch/pytorch
2025-08-14T21:19:33.4052820Z   token: ***
2025-08-14T21:19:33.4053191Z   ssh-strict: true
2025-08-14T21:19:33.4053572Z   ssh-user: git
2025-08-14T21:19:33.4053970Z   persist-credentials: true
2025-08-14T21:19:33.4054444Z   clean: true
2025-08-14T21:19:33.4054948Z   sparse-checkout-cone-mode: true
2025-08-14T21:19:33.4055528Z   fetch-tags: false
2025-08-14T21:19:33.4055955Z   lfs: false
2025-08-14T21:19:33.4056382Z   set-safe-directory: true
2025-08-14T21:19:33.4056863Z env:
2025-08-14T21:19:33.4057264Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:33.4057757Z ##[endgroup]
2025-08-14T21:19:33.5496961Z Syncing repository: pytorch/pytorch
2025-08-14T21:19:33.5500086Z ##[group]Getting Git version info
2025-08-14T21:19:33.5501099Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch'
2025-08-14T21:19:33.5502390Z [command]/usr/bin/git version
2025-08-14T21:19:33.5502885Z git version 2.34.1
2025-08-14T21:19:33.5504421Z ##[endgroup]
2025-08-14T21:19:33.5511182Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/6bf47f37-b52b-4fab-ae0c-4a8c13f29488/.gitconfig'
2025-08-14T21:19:33.5513672Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/6bf47f37-b52b-4fab-ae0c-4a8c13f29488' before making global git config changes
2025-08-14T21:19:33.5515178Z Adding repository directory to the temporary git global config as a safe directory
2025-08-14T21:19:33.5516383Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:19:33.5518062Z [command]/usr/bin/git config --local --get remote.origin.url
2025-08-14T21:19:33.5567524Z https://github.com/pytorch/pytorch
2025-08-14T21:19:33.5591294Z ##[group]Removing previously created refs, to avoid conflicts
2025-08-14T21:19:33.5593920Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD
2025-08-14T21:19:33.5644105Z HEAD
2025-08-14T21:19:33.5727963Z ##[endgroup]
2025-08-14T21:19:33.5734530Z [command]/usr/bin/git submodule status
2025-08-14T21:19:33.6549344Z  7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe)
2025-08-14T21:19:33.6771937Z  4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081)
2025-08-14T21:19:33.6999915Z  b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327)
2025-08-14T21:19:33.7254821Z  c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0)
2025-08-14T21:19:33.7370184Z  2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07 third_party/NVTX (v3.1.0-263-g2942f16)
2025-08-14T21:19:33.7579630Z  1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600)
2025-08-14T21:19:33.8512776Z  51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656)
2025-08-14T21:19:33.8604242Z  01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101)
2025-08-14T21:19:33.8673911Z  299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3)
2025-08-14T21:19:33.8908469Z  7fe50dc3da2069d6645d9deb8c017a876472a977 third_party/composable_kernel (rocm-6.4.2-459-g7fe50dc3d)
2025-08-14T21:19:33.9180204Z  3af7f2c16147f3fbc6e4d717032daf505dc1652c third_party/cpp-httplib (v0.20.1)
2025-08-14T21:19:33.9485274Z  5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244)
2025-08-14T21:19:33.9581356Z  f937055efc6d414d11f4c6577e3977fe74f35fb6 third_party/cudnn_frontend (v0.5-52-gf937055)
2025-08-14T21:19:33.9820766Z  e51efbfe18fe4f4cbb66ab814c55bf4aa0185491 third_party/cutlass (v4.1.0)
2025-08-14T21:19:33.9952297Z  21c7d30c526c0f1ad873ecc632dca6cfa8a69067 third_party/fbgemm (v1.3.0-rc1-165-g21c7d30c)
2025-08-14T21:19:34.0177458Z  979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4)
2025-08-14T21:19:34.0248419Z  a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23)
2025-08-14T21:19:34.0992382Z  40626af88bd7df9a5fb80be7b25ac85b122d6c21 third_party/fmt (11.2.0)
2025-08-14T21:19:34.1291388Z  3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17)
2025-08-14T21:19:34.1608973Z  c7b7b022c124d9643957d9bd55f57ac59fce8fa2 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-33-gc7b7b02)
2025-08-14T21:19:34.2087102Z  52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108)
2025-08-14T21:19:34.2292912Z  719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1)
2025-08-14T21:19:34.2459835Z  dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5)
2025-08-14T21:19:34.3050270Z  5e7501833f1021ce6f618572d3baf657b6319658 third_party/kineto (remotes/origin/sraikund/test-98-g5e75018)
2025-08-14T21:19:34.3124632Z  cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7 third_party/kleidiai (v1.8.0)
2025-08-14T21:19:34.3194580Z  fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4)
2025-08-14T21:19:34.3268055Z  55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0)
2025-08-14T21:19:34.3944611Z  e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0)
2025-08-14T21:19:34.4014551Z  a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2)
2025-08-14T21:19:34.4089404Z  0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5)
2025-08-14T21:19:34.4760574Z  d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b)
2025-08-14T21:19:34.4979805Z  072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master)
2025-08-14T21:19:34.5130646Z  4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e)
2025-08-14T21:19:34.5262047Z  a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e)
2025-08-14T21:19:34.5482419Z  f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated)
2025-08-14T21:19:34.5678378Z  5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8)
2025-08-14T21:19:34.5902631Z  dacda0567d9f23d4bc503e1c4f84aa65f33ac38a third_party/tensorpipe (remotes/origin/HEAD)
2025-08-14T21:19:34.5935452Z ##[group]Cleaning the repository
2025-08-14T21:19:34.5941792Z [command]/usr/bin/git clean -ffdx
2025-08-14T21:19:34.6568398Z [command]/usr/bin/git reset --hard HEAD
2025-08-14T21:19:34.8341423Z HEAD is now at 9faad049c4e Update on "[while_loop] support gen_schema for while_loop"
2025-08-14T21:19:34.8400700Z ##[endgroup]
2025-08-14T21:19:34.8402917Z ##[group]Disabling automatic garbage collection
2025-08-14T21:19:34.8413419Z [command]/usr/bin/git config --local gc.auto 0
2025-08-14T21:19:34.8489718Z ##[endgroup]
2025-08-14T21:19:34.8490546Z ##[group]Setting up auth
2025-08-14T21:19:34.8503322Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-08-14T21:19:34.8597573Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-08-14T21:19:34.9285039Z Entering 'android/libs/fbjni'
2025-08-14T21:19:34.9408262Z Entering 'third_party/FP16'
2025-08-14T21:19:34.9536178Z Entering 'third_party/FXdiv'
2025-08-14T21:19:34.9661372Z Entering 'third_party/NNPACK'
2025-08-14T21:19:34.9788427Z Entering 'third_party/NVTX'
2025-08-14T21:19:34.9913757Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:35.0038668Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:35.0195282Z Entering 'third_party/aiter'
2025-08-14T21:19:35.0318789Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:35.0460957Z Entering 'third_party/benchmark'
2025-08-14T21:19:35.0583331Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:35.0727323Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:35.0846941Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:35.0967914Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:35.1088293Z Entering 'third_party/cutlass'
2025-08-14T21:19:35.1232884Z Entering 'third_party/fbgemm'
2025-08-14T21:19:35.1356366Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:35.1469740Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:35.1601204Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:35.1719394Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:35.1857362Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:35.1975598Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:35.2090712Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:35.2221602Z Entering 'third_party/flash-attention'
2025-08-14T21:19:35.2344508Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:35.2474214Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:35.2617905Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:35.2747159Z Entering 'third_party/fmt'
2025-08-14T21:19:35.2871152Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:35.2994100Z Entering 'third_party/gloo'
2025-08-14T21:19:35.3118129Z Entering 'third_party/googletest'
2025-08-14T21:19:35.3240911Z Entering 'third_party/ideep'
2025-08-14T21:19:35.3358611Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:35.3493969Z Entering 'third_party/ittapi'
2025-08-14T21:19:35.3616535Z Entering 'third_party/kineto'
2025-08-14T21:19:35.3734054Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:35.3843325Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:35.3963636Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:35.4079140Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:35.4194477Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:35.4306557Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:35.4428313Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:35.4540873Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:35.4655049Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:35.4768548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:35.4887244Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:35.5000564Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:35.5122691Z Entering 'third_party/kleidiai'
2025-08-14T21:19:35.5243330Z Entering 'third_party/mimalloc'
2025-08-14T21:19:35.5365722Z Entering 'third_party/nlohmann'
2025-08-14T21:19:35.5489793Z Entering 'third_party/onnx'
2025-08-14T21:19:35.5645890Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:35.5776095Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:35.5901795Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:35.6014959Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:35.6127127Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:35.6238293Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:35.6354449Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:35.6466311Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:35.6575698Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:35.6681764Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:35.6798992Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:35.6917899Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:35.7077280Z Entering 'third_party/pocketfft'
2025-08-14T21:19:35.7197033Z Entering 'third_party/protobuf'
2025-08-14T21:19:35.7323871Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:35.7436657Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:35.7559197Z Entering 'third_party/psimd'
2025-08-14T21:19:35.7680833Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:35.7804500Z Entering 'third_party/pybind11'
2025-08-14T21:19:35.7926548Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:35.8045206Z Entering 'third_party/sleef'
2025-08-14T21:19:35.8179166Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:35.8299111Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:35.8414743Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:35.8523926Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:35.8637279Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:35.8748619Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:35.8915303Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-08-14T21:19:35.8991933Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-08-14T21:19:35.9695216Z Entering 'android/libs/fbjni'
2025-08-14T21:19:35.9823286Z Entering 'third_party/FP16'
2025-08-14T21:19:35.9953273Z Entering 'third_party/FXdiv'
2025-08-14T21:19:36.0080562Z Entering 'third_party/NNPACK'
2025-08-14T21:19:36.0212083Z Entering 'third_party/NVTX'
2025-08-14T21:19:36.0336057Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:36.0463272Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:36.0627345Z Entering 'third_party/aiter'
2025-08-14T21:19:36.0752932Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:36.0894382Z Entering 'third_party/benchmark'
2025-08-14T21:19:36.1018780Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:36.1161994Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:36.1282266Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:36.1405266Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:36.1530386Z Entering 'third_party/cutlass'
2025-08-14T21:19:36.1675399Z Entering 'third_party/fbgemm'
2025-08-14T21:19:36.1796772Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:36.1916736Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:36.2049851Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:36.2168531Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:36.2300025Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:36.2414284Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:36.2529386Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:36.2654654Z Entering 'third_party/flash-attention'
2025-08-14T21:19:36.2777232Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:36.2905770Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:36.3050807Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:36.3179808Z Entering 'third_party/fmt'
2025-08-14T21:19:36.3304676Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:36.3431669Z Entering 'third_party/gloo'
2025-08-14T21:19:36.3555024Z Entering 'third_party/googletest'
2025-08-14T21:19:36.3677400Z Entering 'third_party/ideep'
2025-08-14T21:19:36.3794125Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:36.3928481Z Entering 'third_party/ittapi'
2025-08-14T21:19:36.4048296Z Entering 'third_party/kineto'
2025-08-14T21:19:36.4168657Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:36.4275733Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:36.4395294Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:36.4514894Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:36.4633904Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:36.4749428Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:36.4875653Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:36.4990057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:36.5108068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:36.5223801Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:36.5343035Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:36.5457858Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:36.5580180Z Entering 'third_party/kleidiai'
2025-08-14T21:19:36.5703461Z Entering 'third_party/mimalloc'
2025-08-14T21:19:36.5827695Z Entering 'third_party/nlohmann'
2025-08-14T21:19:36.5950773Z Entering 'third_party/onnx'
2025-08-14T21:19:36.6105577Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:36.6234551Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:36.6357379Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:36.6470065Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:36.6583820Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:36.6698275Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:36.6813246Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:36.6923740Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:36.7033316Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:36.7140692Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:36.7255791Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:36.7375930Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:36.7536505Z Entering 'third_party/pocketfft'
2025-08-14T21:19:36.7658601Z Entering 'third_party/protobuf'
2025-08-14T21:19:36.7783410Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:36.7895246Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:36.8015863Z Entering 'third_party/psimd'
2025-08-14T21:19:36.8136308Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:36.8256583Z Entering 'third_party/pybind11'
2025-08-14T21:19:36.8375969Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:36.8495413Z Entering 'third_party/sleef'
2025-08-14T21:19:36.8620843Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:36.8738984Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:36.8851579Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:36.8965699Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:36.9080948Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:36.9191692Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:36.9369633Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-08-14T21:19:36.9473277Z ##[endgroup]
2025-08-14T21:19:36.9474081Z ##[group]Fetching the repository
2025-08-14T21:19:36.9489164Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2025-08-14T21:19:37.9406773Z From https://github.com/pytorch/pytorch
2025-08-14T21:19:37.9407620Z  - [deleted]                 (none)     -> ciflow/inductor/159530
2025-08-14T21:19:38.0087689Z  - [deleted]                 (none)     -> ciflow/inductor/159875
2025-08-14T21:19:38.0088579Z  - [deleted]                 (none)     -> ciflow/inductor/160000
2025-08-14T21:19:38.0089338Z  - [deleted]                 (none)     -> ciflow/trunk/159530
2025-08-14T21:19:38.0090179Z  - [deleted]                 (none)     -> ciflow/trunk/160000
2025-08-14T21:19:39.4871560Z  + 81069eac44e...9e777070cd7 gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig  (forced update)
2025-08-14T21:19:39.4883074Z  + 83d85c0e02b...65bcf0cf22c gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig  (forced update)
2025-08-14T21:19:39.4889949Z    13d33dae859..d6b43a6cd4e  gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head
2025-08-14T21:19:39.4897701Z  + 7a68526b8bf...0768ffd59f3 gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig  (forced update)
2025-08-14T21:19:39.4904993Z    22ef8f43f77..87732b2c98e  gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base
2025-08-14T21:19:39.4910048Z    81ef0f46d54..5556c80b778  gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head
2025-08-14T21:19:39.4912875Z  + 9c9dea5ab2a...c6b3f286785 gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig  (forced update)
2025-08-14T21:19:39.4920180Z    8d6f73986c7..e4dcf56a427  gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base
2025-08-14T21:19:39.4926467Z    f6602a59ed3..2f44712cf54  gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head
2025-08-14T21:19:39.4933144Z  + 5ce995d9848...77fbab074fc gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig  (forced update)
2025-08-14T21:19:39.4938782Z    472b4f2b3f0..3d434480221  gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base
2025-08-14T21:19:39.4946068Z    6d088669e7c..347333b3995  gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head
2025-08-14T21:19:39.4949427Z  + d9e5647addb...9ee0a5f1e67 gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig  (forced update)
2025-08-14T21:19:39.4953914Z    f12b09aa2fc..72cea7eea1f  gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base
2025-08-14T21:19:39.4960622Z    6c4dfda1327..64b4dccb023  gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head
2025-08-14T21:19:39.4967095Z  + 965276ff7a3...03b5add0bd3 gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig  (forced update)
2025-08-14T21:19:39.4972056Z  + 9fbca79be0b...dbe36858fef gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig  (forced update)
2025-08-14T21:19:39.5056785Z    ea5b50f2a44..42e61c43417  gh/xmfan/274/base           -> origin/gh/xmfan/274/base
2025-08-14T21:19:39.5064290Z    b87cd6dc372..bf07dcc2698  gh/xmfan/274/head           -> origin/gh/xmfan/274/head
2025-08-14T21:19:39.5069031Z  + 16c7cc80222...35ce16d727d gh/xmfan/274/orig           -> origin/gh/xmfan/274/orig  (forced update)
2025-08-14T21:19:39.5073990Z  * [new branch]              gh/xmfan/277/base           -> origin/gh/xmfan/277/base
2025-08-14T21:19:39.5079316Z  * [new branch]              gh/xmfan/277/head           -> origin/gh/xmfan/277/head
2025-08-14T21:19:39.5082316Z  * [new branch]              gh/xmfan/277/orig           -> origin/gh/xmfan/277/orig
2025-08-14T21:19:39.5111529Z    eac2d9d695a..3be70dc30e8  main                        -> origin/main
2025-08-14T21:19:39.5148835Z  t [tag update]              ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562
2025-08-14T21:19:39.5153672Z  t [tag update]              ciflow/inductor/159367      -> ciflow/inductor/159367
2025-08-14T21:19:39.5157356Z  t [tag update]              ciflow/inductor/159368      -> ciflow/inductor/159368
2025-08-14T21:19:39.5160841Z  t [tag update]              ciflow/inductor/159483      -> ciflow/inductor/159483
2025-08-14T21:19:39.5164841Z  t [tag update]              ciflow/inductor/159864      -> ciflow/inductor/159864
2025-08-14T21:19:39.5168591Z  t [tag update]              ciflow/inductor/159865      -> ciflow/inductor/159865
2025-08-14T21:19:39.5177555Z  * [new tag]                 ciflow/inductor/160679      -> ciflow/inductor/160679
2025-08-14T21:19:39.5183806Z  * [new tag]                 ciflow/trunk/157152         -> ciflow/trunk/157152
2025-08-14T21:19:39.5187875Z  t [tag update]              ciflow/trunk/159562         -> ciflow/trunk/159562
2025-08-14T21:19:39.5193395Z  t [tag update]              ciflow/win-arm64/159562     -> ciflow/win-arm64/159562
2025-08-14T21:19:39.5205822Z  * [new tag]                 trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 -> trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08
2025-08-14T21:19:39.5209617Z  * [new tag]                 trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 -> trunk/47a1db823dfcdacdb99f317428fc3791a18c5812
2025-08-14T21:19:39.6252504Z [command]/usr/bin/git rev-parse --verify --quiet 1fc683cf17c8c673044538d10266c00f92987be2^{object}
2025-08-14T21:19:39.6373202Z 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:19:39.6386577Z ##[endgroup]
2025-08-14T21:19:39.6387391Z ##[group]Determining the checkout info
2025-08-14T21:19:39.6388226Z ##[endgroup]
2025-08-14T21:19:39.6397181Z [command]/usr/bin/git sparse-checkout disable
2025-08-14T21:19:39.6831431Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig
2025-08-14T21:19:39.6902068Z ##[group]Checking out the ref
2025-08-14T21:19:39.6910151Z [command]/usr/bin/git checkout --progress --force 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:19:40.0044516Z Previous HEAD position was 9faad049c4e Update on "[while_loop] support gen_schema for while_loop"
2025-08-14T21:19:40.0063618Z HEAD is now at 1fc683cf17c [Inductor] Allow indexing a flexible layout for extract_input_node_reduction_ranges (#160645)
2025-08-14T21:19:40.0177191Z ##[endgroup]
2025-08-14T21:19:40.0177998Z ##[group]Setting up auth for fetching submodules
2025-08-14T21:19:40.0188466Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2025-08-14T21:19:40.0280469Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2025-08-14T21:19:40.0355393Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2025-08-14T21:19:40.0433812Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2025-08-14T21:19:40.0505755Z ##[endgroup]
2025-08-14T21:19:40.0506499Z ##[group]Fetching submodules
2025-08-14T21:19:40.0512796Z [command]/usr/bin/git submodule sync --recursive
2025-08-14T21:19:40.1197557Z Synchronizing submodule url for 'android/libs/fbjni'
2025-08-14T21:19:40.1310071Z Synchronizing submodule url for 'third_party/FP16'
2025-08-14T21:19:40.1421795Z Synchronizing submodule url for 'third_party/FXdiv'
2025-08-14T21:19:40.1531034Z Synchronizing submodule url for 'third_party/NNPACK'
2025-08-14T21:19:40.1639461Z Synchronizing submodule url for 'third_party/NVTX'
2025-08-14T21:19:40.1750276Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:40.1861944Z Synchronizing submodule url for 'third_party/XNNPACK'
2025-08-14T21:19:40.1999491Z Synchronizing submodule url for 'third_party/aiter'
2025-08-14T21:19:40.2098387Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:40.2222046Z Synchronizing submodule url for 'third_party/benchmark'
2025-08-14T21:19:40.2328685Z Synchronizing submodule url for 'third_party/composable_kernel'
2025-08-14T21:19:40.2457852Z Synchronizing submodule url for 'third_party/cpp-httplib'
2025-08-14T21:19:40.2564267Z Synchronizing submodule url for 'third_party/cpuinfo'
2025-08-14T21:19:40.2673416Z Synchronizing submodule url for 'third_party/cudnn_frontend'
2025-08-14T21:19:40.2781112Z Synchronizing submodule url for 'third_party/cutlass'
2025-08-14T21:19:40.2903813Z Synchronizing submodule url for 'third_party/fbgemm'
2025-08-14T21:19:40.3006513Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:40.3101953Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:40.3211889Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:40.3305553Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:40.3422028Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:40.3518079Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:40.3610202Z Synchronizing submodule url for 'third_party/fbgemm/external/json'
2025-08-14T21:19:40.3721558Z Synchronizing submodule url for 'third_party/flash-attention'
2025-08-14T21:19:40.3823080Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:40.3938948Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:40.4054242Z Synchronizing submodule url for 'third_party/flatbuffers'
2025-08-14T21:19:40.4174297Z Synchronizing submodule url for 'third_party/fmt'
2025-08-14T21:19:40.4271339Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:40.4377432Z Synchronizing submodule url for 'third_party/gloo'
2025-08-14T21:19:40.4482331Z Synchronizing submodule url for 'third_party/googletest'
2025-08-14T21:19:40.4574229Z Synchronizing submodule url for 'third_party/ideep'
2025-08-14T21:19:40.4673837Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:40.4798508Z Synchronizing submodule url for 'third_party/ittapi'
2025-08-14T21:19:40.4913824Z Synchronizing submodule url for 'third_party/kineto'
2025-08-14T21:19:40.5012075Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:40.5101714Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:40.5202992Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:40.5302214Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:40.5401473Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:40.5493910Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:40.5594775Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:40.5691152Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:40.5792536Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:40.5892897Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:40.5997525Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:40.6092140Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:40.6195427Z Synchronizing submodule url for 'third_party/kleidiai'
2025-08-14T21:19:40.6307514Z Synchronizing submodule url for 'third_party/mimalloc'
2025-08-14T21:19:40.6416530Z Synchronizing submodule url for 'third_party/nlohmann'
2025-08-14T21:19:40.6526680Z Synchronizing submodule url for 'third_party/onnx'
2025-08-14T21:19:40.6665840Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:40.6778437Z Synchronizing submodule url for 'third_party/opentelemetry-cpp'
2025-08-14T21:19:40.6881411Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:40.6979763Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:40.7074545Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:40.7171425Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:40.7267387Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:40.7358831Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:40.7453186Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:40.7542292Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:40.7642945Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:40.7745489Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:40.7893239Z Synchronizing submodule url for 'third_party/pocketfft'
2025-08-14T21:19:40.7995969Z Synchronizing submodule url for 'third_party/protobuf'
2025-08-14T21:19:40.8114881Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:40.8211841Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:40.8319407Z Synchronizing submodule url for 'third_party/psimd'
2025-08-14T21:19:40.8427918Z Synchronizing submodule url for 'third_party/pthreadpool'
2025-08-14T21:19:40.8535766Z Synchronizing submodule url for 'third_party/pybind11'
2025-08-14T21:19:40.8643510Z Synchronizing submodule url for 'third_party/python-peachpy'
2025-08-14T21:19:40.8750968Z Synchronizing submodule url for 'third_party/sleef'
2025-08-14T21:19:40.8862886Z Synchronizing submodule url for 'third_party/tensorpipe'
2025-08-14T21:19:40.8960414Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:40.9053224Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:40.9147033Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:40.9239781Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:40.9328620Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:40.9475116Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2025-08-14T21:19:41.0772131Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2025-08-14T21:19:41.1395112Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2025-08-14T21:19:41.2016427Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2025-08-14T21:19:41.2651398Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2025-08-14T21:19:41.3312162Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07'
2025-08-14T21:19:41.3957787Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1'
2025-08-14T21:19:41.4983678Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883'
2025-08-14T21:19:41.5835134Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150'
2025-08-14T21:19:41.6796938Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf'
2025-08-14T21:19:41.7512923Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f'
2025-08-14T21:19:41.8684272Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977'
2025-08-14T21:19:41.9398303Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c'
2025-08-14T21:19:42.0058850Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d'
2025-08-14T21:19:42.0787788Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6'
2025-08-14T21:19:42.1669134Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491'
2025-08-14T21:19:42.2598591Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067'
2025-08-14T21:19:42.3172664Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea'
2025-08-14T21:19:42.3903618Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9'
2025-08-14T21:19:42.4544621Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349'
2025-08-14T21:19:42.5321180Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad'
2025-08-14T21:19:42.5981354Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-08-14T21:19:42.6570443Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c'
2025-08-14T21:19:42.7264746Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03'
2025-08-14T21:19:42.8011711Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5'
2025-08-14T21:19:42.8947992Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33'
2025-08-14T21:19:42.9746608Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420'
2025-08-14T21:19:43.0540059Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757'
2025-08-14T21:19:43.1227042Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21'
2025-08-14T21:19:43.1874200Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2025-08-14T21:19:43.2536406Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2'
2025-08-14T21:19:43.3211746Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723'
2025-08-14T21:19:43.3886783Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3'
2025-08-14T21:19:43.4941826Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d'
2025-08-14T21:19:43.5645989Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959'
2025-08-14T21:19:43.6357736Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658'
2025-08-14T21:19:43.7010139Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e'
2025-08-14T21:19:43.7641479Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9'
2025-08-14T21:19:43.8262706Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400'
2025-08-14T21:19:43.8886009Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05'
2025-08-14T21:19:43.9492002Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067'
2025-08-14T21:19:44.0071061Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4'
2025-08-14T21:19:44.0685641Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446'
2025-08-14T21:19:44.1300274Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850'
2025-08-14T21:19:44.2013865Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5'
2025-08-14T21:19:44.2612744Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150'
2025-08-14T21:19:44.3235510Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164'
2025-08-14T21:19:44.3833704Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2025-08-14T21:19:44.4528923Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7'
2025-08-14T21:19:44.5232824Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e'
2025-08-14T21:19:44.6007075Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72'
2025-08-14T21:19:44.7046988Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83'
2025-08-14T21:19:44.7785303Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-08-14T21:19:44.8575881Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878'
2025-08-14T21:19:44.9170179Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2'
2025-08-14T21:19:44.9773104Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1'
2025-08-14T21:19:45.0362863Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa'
2025-08-14T21:19:45.1057484Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d'
2025-08-14T21:19:45.1656037Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce'
2025-08-14T21:19:45.2236247Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5'
2025-08-14T21:19:45.2851908Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d'
2025-08-14T21:19:45.3484794Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4'
2025-08-14T21:19:45.4103349Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2025-08-14T21:19:45.5094595Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50'
2025-08-14T21:19:45.5820888Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa'
2025-08-14T21:19:45.6904617Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2025-08-14T21:19:45.7503512Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2025-08-14T21:19:45.8123510Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2025-08-14T21:19:45.8784960Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2025-08-14T21:19:45.9420291Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8'
2025-08-14T21:19:46.0122864Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4'
2025-08-14T21:19:46.0754983Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2025-08-14T21:19:46.1412353Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68'
2025-08-14T21:19:46.2093469Z Submodule path 'third_party/tensorpipe': checked out 'dacda0567d9f23d4bc503e1c4f84aa65f33ac38a'
2025-08-14T21:19:46.2673534Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2025-08-14T21:19:46.3247282Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2025-08-14T21:19:46.4174672Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b'
2025-08-14T21:19:46.4833959Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2025-08-14T21:19:46.5390711Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2025-08-14T21:19:46.5627788Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2025-08-14T21:19:46.6302553Z Entering 'android/libs/fbjni'
2025-08-14T21:19:46.6423799Z Entering 'third_party/FP16'
2025-08-14T21:19:46.6540114Z Entering 'third_party/FXdiv'
2025-08-14T21:19:46.6651752Z Entering 'third_party/NNPACK'
2025-08-14T21:19:46.6757453Z Entering 'third_party/NVTX'
2025-08-14T21:19:46.6869619Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:46.6980025Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:46.7120326Z Entering 'third_party/aiter'
2025-08-14T21:19:46.7229587Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:46.7356802Z Entering 'third_party/benchmark'
2025-08-14T21:19:46.7464165Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:46.7592178Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:46.7698729Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:46.7813690Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:46.7914124Z Entering 'third_party/cutlass'
2025-08-14T21:19:46.8040448Z Entering 'third_party/fbgemm'
2025-08-14T21:19:46.8151167Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:46.8250965Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:46.8365231Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:46.8467968Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:46.8588425Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:46.8689915Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:46.8789745Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:46.8903690Z Entering 'third_party/flash-attention'
2025-08-14T21:19:46.9012556Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:46.9128310Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:46.9252018Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:46.9366473Z Entering 'third_party/fmt'
2025-08-14T21:19:46.9475204Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:46.9584794Z Entering 'third_party/gloo'
2025-08-14T21:19:46.9696071Z Entering 'third_party/googletest'
2025-08-14T21:19:46.9802013Z Entering 'third_party/ideep'
2025-08-14T21:19:46.9904991Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:47.0024241Z Entering 'third_party/ittapi'
2025-08-14T21:19:47.0131548Z Entering 'third_party/kineto'
2025-08-14T21:19:47.0235598Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:47.0331438Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:47.0435215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:47.0537973Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:47.0639470Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:47.0734225Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:47.0842007Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:47.0943087Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:47.1045740Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:47.1149323Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:47.1267220Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:47.1368128Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:47.1476431Z Entering 'third_party/kleidiai'
2025-08-14T21:19:47.1585417Z Entering 'third_party/mimalloc'
2025-08-14T21:19:47.1694516Z Entering 'third_party/nlohmann'
2025-08-14T21:19:47.1805771Z Entering 'third_party/onnx'
2025-08-14T21:19:47.1946837Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:47.2065480Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:47.2178180Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:47.2278582Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:47.2378923Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:47.2477706Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:47.2579596Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:47.2679137Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:47.2779005Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:47.2874317Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:47.2980680Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:47.3088458Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:47.3235951Z Entering 'third_party/pocketfft'
2025-08-14T21:19:47.3343247Z Entering 'third_party/protobuf'
2025-08-14T21:19:47.3452720Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:47.3554673Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:47.3662810Z Entering 'third_party/psimd'
2025-08-14T21:19:47.3775041Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:47.3882602Z Entering 'third_party/pybind11'
2025-08-14T21:19:47.3991776Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:47.4103608Z Entering 'third_party/sleef'
2025-08-14T21:19:47.4215329Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:47.4323299Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:47.4426243Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:47.4531909Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:47.4630698Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:47.4732474Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:47.4876805Z ##[endgroup]
2025-08-14T21:19:47.4877561Z ##[group]Persisting credentials for submodules
2025-08-14T21:19:47.4888109Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :"
2025-08-14T21:19:47.5577299Z Entering 'android/libs/fbjni'
2025-08-14T21:19:47.5644125Z url.https://github.com/.insteadof
2025-08-14T21:19:47.5644805Z url.https://github.com/.insteadof
2025-08-14T21:19:47.5747430Z Entering 'third_party/FP16'
2025-08-14T21:19:47.5815127Z url.https://github.com/.insteadof
2025-08-14T21:19:47.5815794Z url.https://github.com/.insteadof
2025-08-14T21:19:47.5916040Z Entering 'third_party/FXdiv'
2025-08-14T21:19:47.5982783Z url.https://github.com/.insteadof
2025-08-14T21:19:47.5983533Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6079267Z Entering 'third_party/NNPACK'
2025-08-14T21:19:47.6146478Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6147138Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6241241Z Entering 'third_party/NVTX'
2025-08-14T21:19:47.6307600Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6308257Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6404924Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:47.6472147Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6472796Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6571147Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:47.6637847Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6638494Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6768218Z Entering 'third_party/aiter'
2025-08-14T21:19:47.6834799Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6835458Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6931058Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:47.6993826Z url.https://github.com/.insteadof
2025-08-14T21:19:47.6994490Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7110903Z Entering 'third_party/benchmark'
2025-08-14T21:19:47.7178499Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7179157Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7276061Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:47.7342974Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7343625Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7461735Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:47.7528403Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7529183Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7627408Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:47.7695459Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7696143Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7795947Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:47.7862930Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7863609Z url.https://github.com/.insteadof
2025-08-14T21:19:47.7961940Z Entering 'third_party/cutlass'
2025-08-14T21:19:47.8029392Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8030042Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8148405Z Entering 'third_party/fbgemm'
2025-08-14T21:19:47.8215987Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8216668Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8318069Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:47.8381038Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8382240Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8473340Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:47.8536169Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8536852Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8644816Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:47.8707454Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8708118Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8802170Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:47.8862517Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8863191Z url.https://github.com/.insteadof
2025-08-14T21:19:47.8973591Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:47.9036117Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9036787Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9129024Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:47.9189693Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9190374Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9280823Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:47.9341731Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9342430Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9446016Z Entering 'third_party/flash-attention'
2025-08-14T21:19:47.9512325Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9512977Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9603741Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:47.9665004Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9665668Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9772071Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:47.9832781Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9833436Z url.https://github.com/.insteadof
2025-08-14T21:19:47.9952084Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:48.0016666Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0017375Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0116486Z Entering 'third_party/fmt'
2025-08-14T21:19:48.0180672Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0181321Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0275332Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:48.0340638Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0341297Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0437932Z Entering 'third_party/gloo'
2025-08-14T21:19:48.0500808Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0501450Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0596304Z Entering 'third_party/googletest'
2025-08-14T21:19:48.0660982Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0661635Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0755725Z Entering 'third_party/ideep'
2025-08-14T21:19:48.0819499Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0820169Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0909190Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:48.0968094Z url.https://github.com/.insteadof
2025-08-14T21:19:48.0969019Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1083242Z Entering 'third_party/ittapi'
2025-08-14T21:19:48.1148861Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1149553Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1243267Z Entering 'third_party/kineto'
2025-08-14T21:19:48.1307910Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1308567Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1400696Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:48.1462258Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1462918Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1553306Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:48.1617435Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1618290Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1712135Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:48.1772923Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1773578Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1865012Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:48.1926032Z url.https://github.com/.insteadof
2025-08-14T21:19:48.1926695Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2016876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:48.2076172Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2076938Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2166033Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:48.2227535Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2228205Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2327045Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:48.2389341Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2390244Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2481635Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:48.2542733Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2543479Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2632030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:48.2691975Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2692716Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2789168Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:48.2849937Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2850701Z url.https://github.com/.insteadof
2025-08-14T21:19:48.2947218Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:48.3008994Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3044831Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3100839Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:48.3162910Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3163677Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3266087Z Entering 'third_party/kleidiai'
2025-08-14T21:19:48.3330813Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3331564Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3426127Z Entering 'third_party/mimalloc'
2025-08-14T21:19:48.3496363Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3497062Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3594201Z Entering 'third_party/nlohmann'
2025-08-14T21:19:48.3659582Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3660328Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3757390Z Entering 'third_party/onnx'
2025-08-14T21:19:48.3822951Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3823756Z url.https://github.com/.insteadof
2025-08-14T21:19:48.3954600Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:48.4018985Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4019788Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4125391Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:48.4194745Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4195431Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4289456Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:48.4350597Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4351373Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4441585Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:48.4502975Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4503797Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4595681Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:48.4655006Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4655779Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4745181Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:48.4804830Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4805756Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4898658Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:48.4957805Z url.https://github.com/.insteadof
2025-08-14T21:19:48.4958580Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5049807Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:48.5109200Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5109924Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5200701Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:48.5260928Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5261711Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5348228Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:48.5408331Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5409157Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5500331Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:48.5559473Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5560236Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5661867Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:48.5721978Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5722760Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5864174Z Entering 'third_party/pocketfft'
2025-08-14T21:19:48.5929674Z url.https://github.com/.insteadof
2025-08-14T21:19:48.5930440Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6027993Z Entering 'third_party/protobuf'
2025-08-14T21:19:48.6095455Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6096223Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6194937Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:48.6256798Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6257629Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6350579Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:48.6412723Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6413540Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6513492Z Entering 'third_party/psimd'
2025-08-14T21:19:48.6579069Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6579839Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6675159Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:48.6742530Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6743251Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6838925Z Entering 'third_party/pybind11'
2025-08-14T21:19:48.6905917Z url.https://github.com/.insteadof
2025-08-14T21:19:48.6906675Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7004555Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:48.7071575Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7072359Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7170979Z Entering 'third_party/sleef'
2025-08-14T21:19:48.7237134Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7237924Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7331782Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:48.7395197Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7395941Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7484912Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:48.7544392Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7545133Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7637647Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:48.7699947Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7700730Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7791419Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:48.7853126Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7853896Z url.https://github.com/.insteadof
2025-08-14T21:19:48.7944437Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:48.8005038Z url.https://github.com/.insteadof
2025-08-14T21:19:48.8005947Z url.https://github.com/.insteadof
2025-08-14T21:19:48.8091705Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:48.8152376Z url.https://github.com/.insteadof
2025-08-14T21:19:48.8153114Z url.https://github.com/.insteadof
2025-08-14T21:19:48.8306627Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url"
2025-08-14T21:19:48.8986126Z Entering 'android/libs/fbjni'
2025-08-14T21:19:48.9092255Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2025-08-14T21:19:48.9154290Z Entering 'third_party/FP16'
2025-08-14T21:19:48.9264616Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2025-08-14T21:19:48.9322620Z Entering 'third_party/FXdiv'
2025-08-14T21:19:48.9425807Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2025-08-14T21:19:48.9484444Z Entering 'third_party/NNPACK'
2025-08-14T21:19:48.9586682Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2025-08-14T21:19:48.9644152Z Entering 'third_party/NVTX'
2025-08-14T21:19:48.9746740Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config	remote.origin.url
2025-08-14T21:19:48.9811424Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:48.9907571Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2025-08-14T21:19:48.9966108Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:49.0068891Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2025-08-14T21:19:49.0160640Z Entering 'third_party/aiter'
2025-08-14T21:19:49.0261446Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config	remote.origin.url
2025-08-14T21:19:49.0316905Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:49.0419604Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config	remote.origin.url
2025-08-14T21:19:49.0497035Z Entering 'third_party/benchmark'
2025-08-14T21:19:49.0597533Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:19:49.0655485Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:49.0751629Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config	remote.origin.url
2025-08-14T21:19:49.0827003Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:49.0925615Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config	remote.origin.url
2025-08-14T21:19:49.0983325Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:49.1081520Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2025-08-14T21:19:49.1140652Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:49.1239833Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2025-08-14T21:19:49.1297329Z Entering 'third_party/cutlass'
2025-08-14T21:19:49.1394753Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2025-08-14T21:19:49.1474828Z Entering 'third_party/fbgemm'
2025-08-14T21:19:49.1583000Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2025-08-14T21:19:49.1645719Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:49.1745525Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config	remote.origin.url
2025-08-14T21:19:49.1799162Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:49.1901007Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config	remote.origin.url
2025-08-14T21:19:49.1974237Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:49.2072522Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config	remote.origin.url
2025-08-14T21:19:49.2130039Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:49.2229809Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config	remote.origin.url
2025-08-14T21:19:49.2305948Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:49.2406810Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config	remote.origin.url
2025-08-14T21:19:49.2463363Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:49.2564020Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config	remote.origin.url
2025-08-14T21:19:49.2618663Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:49.2718870Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config	remote.origin.url
2025-08-14T21:19:49.2785676Z Entering 'third_party/flash-attention'
2025-08-14T21:19:49.2907096Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config	remote.origin.url
2025-08-14T21:19:49.2961695Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:49.3059759Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config	remote.origin.url
2025-08-14T21:19:49.3130017Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:49.3229006Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config	remote.origin.url
2025-08-14T21:19:49.3309775Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:49.3411935Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2025-08-14T21:19:49.3474907Z Entering 'third_party/fmt'
2025-08-14T21:19:49.3575620Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2025-08-14T21:19:49.3633309Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:49.3733595Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2025-08-14T21:19:49.3791755Z Entering 'third_party/gloo'
2025-08-14T21:19:49.3893809Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2025-08-14T21:19:49.3950882Z Entering 'third_party/googletest'
2025-08-14T21:19:49.4050579Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:49.4112257Z Entering 'third_party/ideep'
2025-08-14T21:19:49.4219168Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2025-08-14T21:19:49.4272729Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:49.4374487Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2025-08-14T21:19:49.4450674Z Entering 'third_party/ittapi'
2025-08-14T21:19:49.4552067Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2025-08-14T21:19:49.4609998Z Entering 'third_party/kineto'
2025-08-14T21:19:49.4712199Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2025-08-14T21:19:49.4765826Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:49.4869398Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config	remote.origin.url
2025-08-14T21:19:49.4920426Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:49.5022701Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config	remote.origin.url
2025-08-14T21:19:49.5080554Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:49.5185236Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config	remote.origin.url
2025-08-14T21:19:49.5242068Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:49.5345476Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config	remote.origin.url
2025-08-14T21:19:49.5399084Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:49.5506040Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config	remote.origin.url
2025-08-14T21:19:49.5557171Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:49.5659492Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config	remote.origin.url
2025-08-14T21:19:49.5720979Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:49.5821814Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config	remote.origin.url
2025-08-14T21:19:49.5875930Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:49.5980744Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:49.6037346Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:49.6137752Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config	remote.origin.url
2025-08-14T21:19:49.6196629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:49.6295520Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config	remote.origin.url
2025-08-14T21:19:49.6358125Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:49.6455935Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2025-08-14T21:19:49.6510988Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:49.6617927Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:49.6682183Z Entering 'third_party/kleidiai'
2025-08-14T21:19:49.6784480Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config	remote.origin.url
2025-08-14T21:19:49.6845060Z Entering 'third_party/mimalloc'
2025-08-14T21:19:49.6946008Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config	remote.origin.url
2025-08-14T21:19:49.7003987Z Entering 'third_party/nlohmann'
2025-08-14T21:19:49.7103325Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2025-08-14T21:19:49.7163495Z Entering 'third_party/onnx'
2025-08-14T21:19:49.7261117Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2025-08-14T21:19:49.7351869Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:49.7451257Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:19:49.7518820Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:49.7620046Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config	remote.origin.url
2025-08-14T21:19:49.7676959Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:49.7780555Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:19:49.7834571Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:49.7934192Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:49.7988767Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:49.8087143Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config	remote.origin.url
2025-08-14T21:19:49.8140733Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:49.8237551Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config	remote.origin.url
2025-08-14T21:19:49.8293391Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:49.8392015Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config	remote.origin.url
2025-08-14T21:19:49.8443918Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:49.8542495Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config	remote.origin.url
2025-08-14T21:19:49.8595469Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:49.8693376Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config	remote.origin.url
2025-08-14T21:19:49.8742747Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:49.8841607Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config	remote.origin.url
2025-08-14T21:19:49.8899462Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:49.9000180Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config	remote.origin.url
2025-08-14T21:19:49.9062968Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:49.9161953Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config	remote.origin.url
2025-08-14T21:19:49.9264878Z Entering 'third_party/pocketfft'
2025-08-14T21:19:49.9364608Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2025-08-14T21:19:49.9421532Z Entering 'third_party/protobuf'
2025-08-14T21:19:49.9521903Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2025-08-14T21:19:49.9583330Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:49.9682700Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2025-08-14T21:19:49.9738258Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:49.9837585Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:49.9901740Z Entering 'third_party/psimd'
2025-08-14T21:19:50.0002684Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2025-08-14T21:19:50.0063511Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:50.0171869Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2025-08-14T21:19:50.0226601Z Entering 'third_party/pybind11'
2025-08-14T21:19:50.0333702Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:19:50.0389299Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:50.0491271Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2025-08-14T21:19:50.0548249Z Entering 'third_party/sleef'
2025-08-14T21:19:50.0652665Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2025-08-14T21:19:50.0714261Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:50.0819574Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2025-08-14T21:19:50.0876055Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:50.0974222Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2025-08-14T21:19:50.1029455Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:50.1127117Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2025-08-14T21:19:50.1182668Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:50.1281188Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2025-08-14T21:19:50.1335619Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:50.1438539Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2025-08-14T21:19:50.1486803Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:50.1585583Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2025-08-14T21:19:50.2213316Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2025-08-14T21:19:50.2894182Z Entering 'android/libs/fbjni'
2025-08-14T21:19:50.3005139Z Entering 'third_party/FP16'
2025-08-14T21:19:50.3114726Z Entering 'third_party/FXdiv'
2025-08-14T21:19:50.3224290Z Entering 'third_party/NNPACK'
2025-08-14T21:19:50.3335834Z Entering 'third_party/NVTX'
2025-08-14T21:19:50.3448468Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:50.3555896Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:50.3701958Z Entering 'third_party/aiter'
2025-08-14T21:19:50.3809610Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:50.3937221Z Entering 'third_party/benchmark'
2025-08-14T21:19:50.4045739Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:50.4173966Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:50.4284366Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:50.4395676Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:50.4504897Z Entering 'third_party/cutlass'
2025-08-14T21:19:50.4636303Z Entering 'third_party/fbgemm'
2025-08-14T21:19:50.4748713Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:50.4850082Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:50.4969731Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:50.5071375Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:50.5191936Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:50.5293646Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:50.5397777Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:50.5513007Z Entering 'third_party/flash-attention'
2025-08-14T21:19:50.5622029Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:50.5741029Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:50.5869049Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:50.5983643Z Entering 'third_party/fmt'
2025-08-14T21:19:50.6093280Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:50.6202540Z Entering 'third_party/gloo'
2025-08-14T21:19:50.6315796Z Entering 'third_party/googletest'
2025-08-14T21:19:50.6423773Z Entering 'third_party/ideep'
2025-08-14T21:19:50.6532940Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:50.6655824Z Entering 'third_party/ittapi'
2025-08-14T21:19:50.6772185Z Entering 'third_party/kineto'
2025-08-14T21:19:50.6877830Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:50.6979398Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:50.7086353Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:50.7192941Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:50.7295365Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:50.7396759Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:50.7511445Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:50.7615694Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:50.7718977Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:50.7824685Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:50.7935708Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:50.8036919Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:50.8150149Z Entering 'third_party/kleidiai'
2025-08-14T21:19:50.8260497Z Entering 'third_party/mimalloc'
2025-08-14T21:19:50.8372712Z Entering 'third_party/nlohmann'
2025-08-14T21:19:50.8489355Z Entering 'third_party/onnx'
2025-08-14T21:19:50.8638164Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:50.8758717Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:50.8872560Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:50.8974666Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:50.9077461Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:50.9177459Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:50.9283021Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:50.9382584Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:50.9483937Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:50.9582673Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:50.9690107Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:50.9800446Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:50.9950779Z Entering 'third_party/pocketfft'
2025-08-14T21:19:51.0061374Z Entering 'third_party/protobuf'
2025-08-14T21:19:51.0174795Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:51.0278180Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:51.0391602Z Entering 'third_party/psimd'
2025-08-14T21:19:51.0502688Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:51.0612986Z Entering 'third_party/pybind11'
2025-08-14T21:19:51.0727123Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:51.0838504Z Entering 'third_party/sleef'
2025-08-14T21:19:51.0948924Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:51.1053258Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:51.1153568Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:51.1254483Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:51.1356295Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:51.1449332Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:51.1611834Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2025-08-14T21:19:51.2295749Z Entering 'android/libs/fbjni'
2025-08-14T21:19:51.2412898Z Entering 'third_party/FP16'
2025-08-14T21:19:51.2518088Z Entering 'third_party/FXdiv'
2025-08-14T21:19:51.2629891Z Entering 'third_party/NNPACK'
2025-08-14T21:19:51.2739795Z Entering 'third_party/NVTX'
2025-08-14T21:19:51.2850903Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T21:19:51.2961832Z Entering 'third_party/XNNPACK'
2025-08-14T21:19:51.3103942Z Entering 'third_party/aiter'
2025-08-14T21:19:51.3210871Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T21:19:51.3341450Z Entering 'third_party/benchmark'
2025-08-14T21:19:51.3451451Z Entering 'third_party/composable_kernel'
2025-08-14T21:19:51.3582420Z Entering 'third_party/cpp-httplib'
2025-08-14T21:19:51.3691590Z Entering 'third_party/cpuinfo'
2025-08-14T21:19:51.3800266Z Entering 'third_party/cudnn_frontend'
2025-08-14T21:19:51.3910402Z Entering 'third_party/cutlass'
2025-08-14T21:19:51.4038889Z Entering 'third_party/fbgemm'
2025-08-14T21:19:51.4155014Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T21:19:51.4261551Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T21:19:51.4383704Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T21:19:51.4489303Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T21:19:51.4612377Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T21:19:51.4714666Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T21:19:51.4818604Z Entering 'third_party/fbgemm/external/json'
2025-08-14T21:19:51.4935846Z Entering 'third_party/flash-attention'
2025-08-14T21:19:51.5042691Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T21:19:51.5162404Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T21:19:51.5292801Z Entering 'third_party/flatbuffers'
2025-08-14T21:19:51.5415015Z Entering 'third_party/fmt'
2025-08-14T21:19:51.5523453Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T21:19:51.5638791Z Entering 'third_party/gloo'
2025-08-14T21:19:51.5745374Z Entering 'third_party/googletest'
2025-08-14T21:19:51.5856243Z Entering 'third_party/ideep'
2025-08-14T21:19:51.5958968Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T21:19:51.6083145Z Entering 'third_party/ittapi'
2025-08-14T21:19:51.6189355Z Entering 'third_party/kineto'
2025-08-14T21:19:51.6294622Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T21:19:51.6394755Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T21:19:51.6500749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T21:19:51.6603793Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T21:19:51.6708215Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T21:19:51.6810249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T21:19:51.6922649Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T21:19:51.7026778Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T21:19:51.7143388Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T21:19:51.7248502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T21:19:51.7359700Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T21:19:51.7462485Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T21:19:51.7575019Z Entering 'third_party/kleidiai'
2025-08-14T21:19:51.7685137Z Entering 'third_party/mimalloc'
2025-08-14T21:19:51.7794597Z Entering 'third_party/nlohmann'
2025-08-14T21:19:51.7909550Z Entering 'third_party/onnx'
2025-08-14T21:19:51.8054269Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T21:19:51.8179077Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T21:19:51.8293028Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T21:19:51.8392763Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T21:19:51.8499031Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T21:19:51.8602080Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T21:19:51.8710252Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T21:19:51.8811919Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T21:19:51.8911427Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T21:19:51.9008512Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T21:19:51.9117379Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T21:19:51.9225747Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T21:19:51.9377670Z Entering 'third_party/pocketfft'
2025-08-14T21:19:51.9490900Z Entering 'third_party/protobuf'
2025-08-14T21:19:51.9600754Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T21:19:51.9705785Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T21:19:51.9813805Z Entering 'third_party/psimd'
2025-08-14T21:19:51.9920796Z Entering 'third_party/pthreadpool'
2025-08-14T21:19:52.0030849Z Entering 'third_party/pybind11'
2025-08-14T21:19:52.0137314Z Entering 'third_party/python-peachpy'
2025-08-14T21:19:52.0245388Z Entering 'third_party/sleef'
2025-08-14T21:19:52.0356197Z Entering 'third_party/tensorpipe'
2025-08-14T21:19:52.0462398Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T21:19:52.0569245Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T21:19:52.0673787Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T21:19:52.0780513Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T21:19:52.0881028Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T21:19:52.1031991Z ##[endgroup]
2025-08-14T21:19:52.1180087Z [command]/usr/bin/git log -1 --format=%H
2025-08-14T21:19:52.1301744Z 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:19:52.1711661Z Prepare all required actions
2025-08-14T21:19:52.1713070Z Getting action download info
2025-08-14T21:19:52.3557828Z ##[group]Run ./.github/actions/setup-rocm
2025-08-14T21:19:52.3558366Z env:
2025-08-14T21:19:52.3558716Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.3559153Z ##[endgroup]
2025-08-14T21:19:52.3600523Z ##[group]Run dpkg -l | grep -E "  rocm"
2025-08-14T21:19:52.3601032Z dpkg -l | grep -E "  rocm"
2025-08-14T21:19:52.3661596Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.3662164Z env:
2025-08-14T21:19:52.3662487Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.3662860Z ##[endgroup]
2025-08-14T21:19:52.4095229Z ii  rocm                                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) software stack meta package
2025-08-14T21:19:52.4096589Z ii  rocm-cmake                             0.14.0.60303-74~22.04                   amd64        rocm-cmake built using CMake
2025-08-14T21:19:52.4097791Z ii  rocm-core                              6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4098949Z ii  rocm-dbgapi                            0.77.0.60303-74~22.04                   amd64        Library to provide AMD GPU debugger API
2025-08-14T21:19:52.4100163Z ii  rocm-debug-agent                       2.0.3.60303-74~22.04                    amd64        Radeon Open Compute Debug Agent (ROCdebug-agent)
2025-08-14T21:19:52.4101454Z ii  rocm-developer-tools                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4102681Z ii  rocm-device-libs                       1.0.0.60303-74~22.04                    amd64        Radeon Open Compute - device libraries
2025-08-14T21:19:52.4103701Z ii  rocm-gdb                               15.2.60303-74~22.04                     amd64        ROCgdb
2025-08-14T21:19:52.4104914Z ii  rocm-hip-libraries                     6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4106197Z ii  rocm-hip-runtime                       6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4107451Z ii  rocm-hip-runtime-dev                   6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4108698Z ii  rocm-hip-sdk                           6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4109928Z ii  rocm-language-runtime                  6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4162922Z ii  rocm-llvm                              18.0.0.25012.60303-74~22.04             amd64        ROCm core compiler
2025-08-14T21:19:52.4164263Z ii  rocm-ml-libraries                      6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4165676Z ii  rocm-ml-sdk                            6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4166902Z ii  rocm-opencl                            2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-08-14T21:19:52.4168138Z ii  rocm-opencl-dev                        2.0.0.60303-74~22.04                    amd64        clr built using CMake
2025-08-14T21:19:52.4169484Z ii  rocm-opencl-runtime                    6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4170742Z ii  rocm-opencl-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4172668Z ii  rocm-openmp-sdk                        6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) OpenMP Software development Kit.
2025-08-14T21:19:52.4174325Z ii  rocm-smi-lib                           7.4.0.60303-74~22.04                    amd64        AMD System Management libraries
2025-08-14T21:19:52.4175453Z ii  rocm-utils                             6.3.3.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime software stack
2025-08-14T21:19:52.4176604Z ii  rocminfo                               1.0.0.60303-74~22.04                    amd64        Radeon Open Compute (ROCm) Runtime rocminfo tool
2025-08-14T21:19:52.4214745Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T21:19:52.4215773Z # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T21:19:52.4216512Z # shellcheck disable=SC2046
2025-08-14T21:19:52.4217096Z docker stop $(docker ps -q) || true
2025-08-14T21:19:52.4217704Z # Prune all stopped containers.
2025-08-14T21:19:52.4218281Z docker container prune -f
2025-08-14T21:19:52.4284244Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.4284923Z env:
2025-08-14T21:19:52.4285319Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.4286114Z ##[endgroup]
2025-08-14T21:19:52.5029238Z "docker stop" requires at least 1 argument.
2025-08-14T21:19:52.5029778Z See 'docker stop --help'.
2025-08-14T21:19:52.5030067Z 
2025-08-14T21:19:52.5030333Z Usage:  docker stop [OPTIONS] CONTAINER [CONTAINER...]
2025-08-14T21:19:52.5030810Z 
2025-08-14T21:19:52.5031016Z Stop one or more running containers
2025-08-14T21:19:52.5388681Z Total reclaimed space: 0B
2025-08-14T21:19:52.5488511Z ##[group]Run cat /etc/os-release || true
2025-08-14T21:19:52.5489151Z cat /etc/os-release || true
2025-08-14T21:19:52.5489756Z cat /etc/apt/sources.list.d/rocm.list || true
2025-08-14T21:19:52.5490410Z cat /opt/rocm/.info/version || true
2025-08-14T21:19:52.5490940Z whoami
2025-08-14T21:19:52.5561544Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.5562234Z env:
2025-08-14T21:19:52.5562598Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.5563054Z ##[endgroup]
2025-08-14T21:19:52.5692757Z PRETTY_NAME="Ubuntu 22.04.4 LTS"
2025-08-14T21:19:52.5693283Z NAME="Ubuntu"
2025-08-14T21:19:52.5693616Z VERSION_ID="22.04"
2025-08-14T21:19:52.5693995Z VERSION="22.04.4 LTS (Jammy Jellyfish)"
2025-08-14T21:19:52.5694459Z VERSION_CODENAME=jammy
2025-08-14T21:19:52.5694811Z ID=ubuntu
2025-08-14T21:19:52.5695117Z ID_LIKE=debian
2025-08-14T21:19:52.5695498Z HOME_URL="https://www.ubuntu.com/"
2025-08-14T21:19:52.5696064Z SUPPORT_URL="https://help.ubuntu.com/"
2025-08-14T21:19:52.5696750Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
2025-08-14T21:19:52.5697737Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
2025-08-14T21:19:52.5698602Z UBUNTU_CODENAME=jammy
2025-08-14T21:19:52.5721994Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main
2025-08-14T21:19:52.5749317Z 6.3.3-74
2025-08-14T21:19:52.5793427Z pytorchci
2025-08-14T21:19:52.5843150Z ##[group]Run dpkg -l | grep -E "  amdgpu"
2025-08-14T21:19:52.5843777Z dpkg -l | grep -E "  amdgpu"
2025-08-14T21:19:52.5913059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.5913796Z env:
2025-08-14T21:19:52.5914188Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.5914622Z ##[endgroup]
2025-08-14T21:19:52.6365719Z ii  amdgpu-core                            1:6.3.60303-2119913.22.04               all          Core meta package for unified amdgpu driver.
2025-08-14T21:19:52.6367001Z ii  amdgpu-dkms                            1:6.10.5.60303-2119913.22.04            all          amdgpu driver in DKMS format.
2025-08-14T21:19:52.6368283Z ii  amdgpu-dkms-firmware                   1:6.10.5.60303-2119913.22.04            all          firmware blobs used by amdgpu driver in DKMS format
2025-08-14T21:19:52.6370221Z ii  amdgpu-install                         6.3.60303-2119913.22.04                 all          AMDGPU driver repository and installer
2025-08-14T21:19:52.6417771Z ##[group]Run rocm-smi
2025-08-14T21:19:52.6418254Z rocm-smi
2025-08-14T21:19:52.6489912Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.6490596Z env:
2025-08-14T21:19:52.6490979Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.6491422Z ##[endgroup]
2025-08-14T21:19:52.8499739Z 
2025-08-14T21:19:52.8499868Z 
2025-08-14T21:19:52.8500458Z ========================================= ROCm System Management Interface =========================================
2025-08-14T21:19:52.8501442Z =================================================== Concise Info ===================================================
2025-08-14T21:19:52.8502514Z Device  Node  IDs              Temp    Power  Partitions          SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2025-08-14T21:19:52.8504096Z               (DID,     GUID)  (Edge)  (Avg)  (Mem, Compute, ID)                                                   
2025-08-14T21:19:52.8504968Z ====================================================================================================================
2025-08-14T21:19:52.8506189Z 0       4     0x740c,   57586  46.0°C  87.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:19:52.8507431Z 1       5     0x740c,   45873  44.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:19:52.8508565Z 2       2     0x740c,   51627  39.0°C  92.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:19:52.8509688Z 3       3     0x740c,   64489  41.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:19:52.8510804Z 4       8     0x740c,   30939  39.0°C  96.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:19:52.8511964Z 5       9     0x740c,   8466   42.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:19:52.8513088Z 6       6     0x740c,   41154  37.0°C  92.0W  N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  560.0W  0%     0%    
2025-08-14T21:19:52.8514213Z 7       7     0x740c,   63755  31.0°C  N/A    N/A, N/A, 0         800Mhz  1600Mhz  0%   auto  0.0W    0%     0%    
2025-08-14T21:19:52.8515023Z ====================================================================================================================
2025-08-14T21:19:52.8515735Z =============================================== End of ROCm SMI Log ================================================
2025-08-14T21:19:52.8660874Z ##[group]Run rocminfo
2025-08-14T21:19:52.8661137Z rocminfo
2025-08-14T21:19:52.8723529Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:52.8724180Z env:
2025-08-14T21:19:52.8724554Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:52.8724980Z ##[endgroup]
2025-08-14T21:19:53.0143825Z ROCk module version 6.10.5 is loaded
2025-08-14T21:19:53.0144637Z =====================    
2025-08-14T21:19:53.0145192Z HSA System Attributes    
2025-08-14T21:19:53.0145686Z =====================    
2025-08-14T21:19:53.0146133Z Runtime Version:         1.14
2025-08-14T21:19:53.0146596Z Runtime Ext Version:     1.6
2025-08-14T21:19:53.0147060Z System Timestamp Freq.:  1000.000000MHz
2025-08-14T21:19:53.0147955Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-08-14T21:19:53.0148870Z Machine Model:           LARGE                              
2025-08-14T21:19:53.0149617Z System Endianness:       LITTLE                             
2025-08-14T21:19:53.0150207Z Mwaitx:                  DISABLED
2025-08-14T21:19:53.0150675Z DMAbuf Support:          YES
2025-08-14T21:19:53.0150954Z 
2025-08-14T21:19:53.0151094Z ==========               
2025-08-14T21:19:53.0151500Z HSA Agents               
2025-08-14T21:19:53.0151886Z ==========               
2025-08-14T21:19:53.0152784Z *******                  
2025-08-14T21:19:53.0153527Z Agent 1                  
2025-08-14T21:19:53.0153893Z *******                  
2025-08-14T21:19:53.0154413Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:19:53.0155080Z   Uuid:                    CPU-XX                             
2025-08-14T21:19:53.0155747Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:19:53.0156451Z   Vendor Name:             CPU                                
2025-08-14T21:19:53.0157095Z   Feature:                 None specified                     
2025-08-14T21:19:53.0157735Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:19:53.0158401Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0159060Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:19:53.0159689Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:19:53.0160324Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:19:53.0160952Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0161530Z   Node:                    0                                  
2025-08-14T21:19:53.0162119Z   Device Type:             CPU                                
2025-08-14T21:19:53.0162663Z   Cache Info:              
2025-08-14T21:19:53.0163127Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:19:53.0163687Z   Chip ID:                 0(0x0)                             
2025-08-14T21:19:53.0164397Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:19:53.0165079Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:19:53.0165940Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:19:53.0166555Z   BDFID:                   0                                  
2025-08-14T21:19:53.0167181Z   Internal Node ID:        0                                  
2025-08-14T21:19:53.0167826Z   Compute Unit:            64                                 
2025-08-14T21:19:53.0168467Z   SIMDs per CU:            0                                  
2025-08-14T21:19:53.0169098Z   Shader Engines:          0                                  
2025-08-14T21:19:53.0169752Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:19:53.0170444Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:19:53.0171042Z   Memory Properties:       
2025-08-14T21:19:53.0171498Z   Features:                None
2025-08-14T21:19:53.0171933Z   Pool Info:               
2025-08-14T21:19:53.0172357Z     Pool 1                   
2025-08-14T21:19:53.0172885Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0173569Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:19:53.0174234Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0174911Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0175632Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0176340Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0177020Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0177601Z     Pool 2                   
2025-08-14T21:19:53.0178145Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0178801Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:19:53.0179413Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0180076Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0181079Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0181780Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0182783Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0183648Z     Pool 3                   
2025-08-14T21:19:53.0184171Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:19:53.0184804Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:19:53.0185418Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0186065Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0186745Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0187428Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0188113Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0188690Z     Pool 4                   
2025-08-14T21:19:53.0189202Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0189828Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:19:53.0190447Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0191121Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0191798Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0192497Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0193168Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0193744Z   ISA Info:                
2025-08-14T21:19:53.0194164Z *******                  
2025-08-14T21:19:53.0194552Z Agent 2                  
2025-08-14T21:19:53.0194938Z *******                  
2025-08-14T21:19:53.0195406Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:19:53.0196032Z   Uuid:                    CPU-XX                             
2025-08-14T21:19:53.0196680Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:19:53.0197381Z   Vendor Name:             CPU                                
2025-08-14T21:19:53.0198061Z   Feature:                 None specified                     
2025-08-14T21:19:53.0198698Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:19:53.0199357Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0200013Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:19:53.0200664Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:19:53.0201291Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:19:53.0201936Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0202530Z   Node:                    1                                  
2025-08-14T21:19:53.0203118Z   Device Type:             CPU                                
2025-08-14T21:19:53.0203682Z   Cache Info:              
2025-08-14T21:19:53.0204137Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:19:53.0204733Z   Chip ID:                 0(0x0)                             
2025-08-14T21:19:53.0205351Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:19:53.0206127Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:19:53.0206772Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:19:53.0207385Z   BDFID:                   0                                  
2025-08-14T21:19:53.0208010Z   Internal Node ID:        1                                  
2025-08-14T21:19:53.0208638Z   Compute Unit:            64                                 
2025-08-14T21:19:53.0209265Z   SIMDs per CU:            0                                  
2025-08-14T21:19:53.0209894Z   Shader Engines:          0                                  
2025-08-14T21:19:53.0210552Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:19:53.0211231Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:19:53.0212153Z   Memory Properties:       
2025-08-14T21:19:53.0212875Z   Features:                None
2025-08-14T21:19:53.0213303Z   Pool Info:               
2025-08-14T21:19:53.0213719Z     Pool 1                   
2025-08-14T21:19:53.0214236Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0214878Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:19:53.0215498Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0216164Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0216848Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0217559Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0218244Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0218823Z     Pool 2                   
2025-08-14T21:19:53.0219380Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0220009Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:19:53.0220617Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0221260Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0221960Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0222657Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0223322Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0223899Z     Pool 3                   
2025-08-14T21:19:53.0224473Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:19:53.0225095Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:19:53.0225710Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0226392Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0227082Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0227778Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0228443Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0229019Z     Pool 4                   
2025-08-14T21:19:53.0229518Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0230131Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:19:53.0230737Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0231415Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0232088Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:19:53.0232768Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0233448Z       Accessible by all:       TRUE                               
2025-08-14T21:19:53.0234031Z   ISA Info:                
2025-08-14T21:19:53.0234440Z *******                  
2025-08-14T21:19:53.0234823Z Agent 3                  
2025-08-14T21:19:53.0235212Z *******                  
2025-08-14T21:19:53.0235654Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0236242Z   Uuid:                    GPU-d8bf612a1b3d9d46               
2025-08-14T21:19:53.0236902Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0237566Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0238204Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0238832Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0239482Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0240134Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0241037Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0241962Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0242576Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0243170Z   Node:                    2                                  
2025-08-14T21:19:53.0243751Z   Device Type:             GPU                                
2025-08-14T21:19:53.0244313Z   Cache Info:              
2025-08-14T21:19:53.0244755Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0245309Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0246179Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0246786Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0247444Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0248122Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0248735Z   BDFID:                   12800                              
2025-08-14T21:19:53.0249352Z   Internal Node ID:        2                                  
2025-08-14T21:19:53.0249994Z   Compute Unit:            104                                
2025-08-14T21:19:53.0250610Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0251236Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0251902Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0252572Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0253270Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0253866Z   Memory Properties:       
2025-08-14T21:19:53.0254345Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0254961Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0255637Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0256320Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0256925Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0257431Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0257956Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0258489Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0259072Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0259740Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0260398Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0260979Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0261457Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0261995Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0262548Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0263170Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0270080Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0270929Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0271635Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0272239Z   Pool Info:               
2025-08-14T21:19:53.0272664Z     Pool 1                   
2025-08-14T21:19:53.0273217Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0273869Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0274497Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0275148Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0276286Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0277008Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0277975Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0278556Z     Pool 2                   
2025-08-14T21:19:53.0279084Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0279747Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0280406Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0281057Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0281750Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0282449Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0283127Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0283692Z     Pool 3                   
2025-08-14T21:19:53.0284233Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0284865Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0285607Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0286268Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0286944Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0287646Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0288310Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0288892Z     Pool 4                   
2025-08-14T21:19:53.0289396Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0289982Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0290596Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0291253Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0291954Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0292626Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0293291Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0293860Z   ISA Info:                
2025-08-14T21:19:53.0294275Z     ISA 1                    
2025-08-14T21:19:53.0294819Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0295547Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0296260Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0296936Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0297641Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0298286Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0298939Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0299560Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0300097Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0300664Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0301186Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0301786Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0302375Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0302884Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0303424Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0303950Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0304883Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0305452Z *******                  
2025-08-14T21:19:53.0306143Z Agent 4                  
2025-08-14T21:19:53.0306522Z *******                  
2025-08-14T21:19:53.0306971Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0307580Z   Uuid:                    GPU-ecc83b03e3da2c15               
2025-08-14T21:19:53.0308237Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0308899Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0309544Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0310187Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0310831Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0311495Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0312128Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0312770Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0313410Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0313991Z   Node:                    3                                  
2025-08-14T21:19:53.0314596Z   Device Type:             GPU                                
2025-08-14T21:19:53.0315149Z   Cache Info:              
2025-08-14T21:19:53.0315616Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0316159Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0316723Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0317342Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0317986Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0318634Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0319239Z   BDFID:                   13568                              
2025-08-14T21:19:53.0319858Z   Internal Node ID:        3                                  
2025-08-14T21:19:53.0320490Z   Compute Unit:            104                                
2025-08-14T21:19:53.0321113Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0321745Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0322395Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0323080Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0323767Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0324389Z   Memory Properties:       
2025-08-14T21:19:53.0324879Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0325599Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0326285Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0326976Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0327593Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0328102Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0328642Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0329151Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0329739Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0330392Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0331046Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0331634Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0332089Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0332637Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0333163Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0334115Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0335108Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0335798Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0336457Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0337016Z   Pool Info:               
2025-08-14T21:19:53.0337437Z     Pool 1                   
2025-08-14T21:19:53.0337966Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0338611Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0339240Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0339900Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0340595Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0341301Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0341995Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0342570Z     Pool 2                   
2025-08-14T21:19:53.0343102Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0343728Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0344346Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0344995Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0345684Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0346378Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0347041Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0347621Z     Pool 3                   
2025-08-14T21:19:53.0348260Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0348915Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0349542Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0350193Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0350881Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0351560Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0352250Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0352828Z     Pool 4                   
2025-08-14T21:19:53.0353333Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0353935Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0354540Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0355208Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0355893Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0356660Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0357329Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0357923Z   ISA Info:                
2025-08-14T21:19:53.0358338Z     ISA 1                    
2025-08-14T21:19:53.0358871Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0359595Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0360278Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0360963Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0361663Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0362325Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0363310Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0364183Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0364725Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0365259Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0366134Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0366722Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0367314Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0367795Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0368338Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0368876Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0369471Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0370044Z *******                  
2025-08-14T21:19:53.0370447Z Agent 5                  
2025-08-14T21:19:53.0370849Z *******                  
2025-08-14T21:19:53.0371286Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0371896Z   Uuid:                    GPU-17d29e8be4a2b908               
2025-08-14T21:19:53.0372547Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0373199Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0373845Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0374470Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0375140Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0375723Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0376290Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0376847Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0377400Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0377932Z   Node:                    4                                  
2025-08-14T21:19:53.0378448Z   Device Type:             GPU                                
2025-08-14T21:19:53.0378948Z   Cache Info:              
2025-08-14T21:19:53.0379344Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0379839Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0380329Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0380896Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0381496Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0382076Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0382618Z   BDFID:                   4352                               
2025-08-14T21:19:53.0383163Z   Internal Node ID:        4                                  
2025-08-14T21:19:53.0383742Z   Compute Unit:            104                                
2025-08-14T21:19:53.0384291Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0384857Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0385450Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0386054Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0386666Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0387199Z   Memory Properties:       
2025-08-14T21:19:53.0387629Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0388163Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0388816Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0389419Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0390255Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0390973Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0391449Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0391919Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0392435Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0393028Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0393612Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0394126Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0394545Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0395008Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0395496Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0396046Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0396689Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0397318Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0397902Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0398415Z   Pool Info:               
2025-08-14T21:19:53.0398791Z     Pool 1                   
2025-08-14T21:19:53.0399269Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0399835Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0400403Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0400977Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0401591Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0402219Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0402822Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0403395Z     Pool 2                   
2025-08-14T21:19:53.0403874Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0404454Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0405013Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0405699Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0406314Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0406932Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0407543Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0408052Z     Pool 3                   
2025-08-14T21:19:53.0408521Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0409077Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0409638Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0410235Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0410841Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0411467Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0412060Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0412575Z     Pool 4                   
2025-08-14T21:19:53.0413001Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0413533Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0434869Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0435571Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0436608Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0437265Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0438140Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0438657Z   ISA Info:                
2025-08-14T21:19:53.0439031Z     ISA 1                    
2025-08-14T21:19:53.0439525Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0440184Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0440789Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0441389Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0441994Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0442553Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0443108Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0443669Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0444160Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0444627Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0445092Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0445721Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0446241Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0446663Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0447133Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0447592Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0448131Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0448626Z *******                  
2025-08-14T21:19:53.0448978Z Agent 6                  
2025-08-14T21:19:53.0449331Z *******                  
2025-08-14T21:19:53.0449738Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0450290Z   Uuid:                    GPU-4fe0daed09e32bd4               
2025-08-14T21:19:53.0450867Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0451441Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0452001Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0452555Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0453125Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0453702Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0454265Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0454822Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0455389Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0455909Z   Node:                    5                                  
2025-08-14T21:19:53.0456435Z   Device Type:             GPU                                
2025-08-14T21:19:53.0456927Z   Cache Info:              
2025-08-14T21:19:53.0457337Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0457832Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0458326Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0458875Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0459451Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0460023Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0460560Z   BDFID:                   5120                               
2025-08-14T21:19:53.0461092Z   Internal Node ID:        5                                  
2025-08-14T21:19:53.0461955Z   Compute Unit:            104                                
2025-08-14T21:19:53.0462753Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0463312Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0463890Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0464479Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0465090Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0465620Z   Memory Properties:       
2025-08-14T21:19:53.0466039Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0466560Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0467145Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0467723Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0468258Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0468716Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0469188Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0469644Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0470143Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0470717Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0471281Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0471787Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0472196Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0472663Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0473131Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0473667Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0474316Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0474923Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0475501Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0475995Z   Pool Info:               
2025-08-14T21:19:53.0476356Z     Pool 1                   
2025-08-14T21:19:53.0476829Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0477393Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0477950Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0478523Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0479137Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0479753Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0480335Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0480852Z     Pool 2                   
2025-08-14T21:19:53.0481324Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0481890Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0482448Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0483025Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0483634Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0484244Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0484832Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0485337Z     Pool 3                   
2025-08-14T21:19:53.0486064Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0486603Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0487487Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0488331Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0488943Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0489556Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0490141Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0490648Z     Pool 4                   
2025-08-14T21:19:53.0491083Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0491607Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0492141Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0492723Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0493321Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0493934Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0494534Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0495030Z   ISA Info:                
2025-08-14T21:19:53.0495388Z     ISA 1                    
2025-08-14T21:19:53.0495855Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0496495Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0497097Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0497687Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0498301Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0498862Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0499423Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0499955Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0500442Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0500928Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0501383Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0501903Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0502417Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0502845Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0503310Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0503780Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0504315Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0504805Z *******                  
2025-08-14T21:19:53.0505159Z Agent 7                  
2025-08-14T21:19:53.0505501Z *******                  
2025-08-14T21:19:53.0505897Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0506439Z   Uuid:                    GPU-acdc18a5a79ca32e               
2025-08-14T21:19:53.0507036Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0507630Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0508191Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0508750Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0509306Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0509874Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0510432Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0510979Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0511524Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0512289Z   Node:                    6                                  
2025-08-14T21:19:53.0513026Z   Device Type:             GPU                                
2025-08-14T21:19:53.0513514Z   Cache Info:              
2025-08-14T21:19:53.0513920Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0514396Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0514892Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0515421Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0515986Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0516551Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0517082Z   BDFID:                   44544                              
2025-08-14T21:19:53.0517620Z   Internal Node ID:        6                                  
2025-08-14T21:19:53.0518174Z   Compute Unit:            104                                
2025-08-14T21:19:53.0518726Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0519283Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0519865Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0520455Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0521053Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0521578Z   Memory Properties:       
2025-08-14T21:19:53.0521983Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0522510Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0523087Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0523674Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0524199Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0524641Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0525116Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0525663Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0526173Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0526741Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0527307Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0527803Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0528214Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0528681Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0529145Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0529694Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0530313Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0530924Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0531511Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0532007Z   Pool Info:               
2025-08-14T21:19:53.0532374Z     Pool 1                   
2025-08-14T21:19:53.0532841Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0533402Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0533944Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0534519Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0535124Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0535747Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0536326Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0536833Z     Pool 2                   
2025-08-14T21:19:53.0537601Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0538411Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0538949Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0539514Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0540112Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0540713Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0541303Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0541805Z     Pool 3                   
2025-08-14T21:19:53.0542241Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0542784Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0543312Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0543918Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0544526Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0545145Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0545741Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0546240Z     Pool 4                   
2025-08-14T21:19:53.0546677Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0547195Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0547734Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0548307Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0548913Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0549530Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0550119Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0550643Z   ISA Info:                
2025-08-14T21:19:53.0550999Z     ISA 1                    
2025-08-14T21:19:53.0551482Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0552109Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0552712Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0553307Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0553927Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0554496Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0555055Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0555602Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0556070Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0556567Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0557032Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0557553Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0558065Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0558489Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0558966Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0559430Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0559967Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0560462Z *******                  
2025-08-14T21:19:53.0560814Z Agent 8                  
2025-08-14T21:19:53.0561159Z *******                  
2025-08-14T21:19:53.0561546Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0562324Z   Uuid:                    GPU-a46db4ad480ce7bc               
2025-08-14T21:19:53.0563152Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0563740Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0564289Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0564853Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0565413Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0566090Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0566663Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0567210Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0567766Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0568276Z   Node:                    7                                  
2025-08-14T21:19:53.0568812Z   Device Type:             GPU                                
2025-08-14T21:19:53.0569310Z   Cache Info:              
2025-08-14T21:19:53.0569715Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0570200Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0570686Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0571229Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0571794Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0572369Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0572892Z   BDFID:                   45824                              
2025-08-14T21:19:53.0573438Z   Internal Node ID:        7                                  
2025-08-14T21:19:53.0574005Z   Compute Unit:            104                                
2025-08-14T21:19:53.0574563Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0575127Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0575717Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0576327Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0576925Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0577453Z   Memory Properties:       
2025-08-14T21:19:53.0577868Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0578399Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0578995Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0579580Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0580116Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0580557Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0581035Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0581486Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0582016Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0582595Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0583166Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0583678Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0584077Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0584552Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0585012Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0585563Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0586194Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0586792Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0587681Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0588443Z   Pool Info:               
2025-08-14T21:19:53.0588807Z     Pool 1                   
2025-08-14T21:19:53.0589273Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0589843Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0590406Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0590983Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0591593Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0592206Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0592804Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0593308Z     Pool 2                   
2025-08-14T21:19:53.0593768Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0594338Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0594886Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0595464Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0596070Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0596683Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0597270Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0597780Z     Pool 3                   
2025-08-14T21:19:53.0598225Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0598766Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0599311Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0599880Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0600489Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0601097Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0601684Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0602181Z     Pool 4                   
2025-08-14T21:19:53.0602608Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0603130Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0603660Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0604238Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0604830Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0605439Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0606262Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0606774Z   ISA Info:                
2025-08-14T21:19:53.0607136Z     ISA 1                    
2025-08-14T21:19:53.0607609Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0608238Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0608837Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0609435Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0610047Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0610612Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0611179Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0611713Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0612180Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0612953Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0613460Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0614237Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0614765Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0615198Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0615669Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0616140Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0616665Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0617164Z *******                  
2025-08-14T21:19:53.0617505Z Agent 9                  
2025-08-14T21:19:53.0617846Z *******                  
2025-08-14T21:19:53.0618235Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0618765Z   Uuid:                    GPU-6250e8145e0cb8ff               
2025-08-14T21:19:53.0619348Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0619939Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0620504Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0621055Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0621624Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0622195Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0622762Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0623319Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0623864Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0624380Z   Node:                    8                                  
2025-08-14T21:19:53.0624896Z   Device Type:             GPU                                
2025-08-14T21:19:53.0625397Z   Cache Info:              
2025-08-14T21:19:53.0625803Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0626298Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0626796Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0627337Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0627917Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0628480Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0629020Z   BDFID:                   36352                              
2025-08-14T21:19:53.0629552Z   Internal Node ID:        8                                  
2025-08-14T21:19:53.0630113Z   Compute Unit:            104                                
2025-08-14T21:19:53.0630651Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0631227Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0631818Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0632410Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0633017Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0633535Z   Memory Properties:       
2025-08-14T21:19:53.0633944Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0634465Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0635053Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0635643Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0636169Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0636615Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0637074Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0637775Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0638484Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0639060Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0639638Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0640145Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0640557Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0641027Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0641495Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0642025Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0642658Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0643265Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0643844Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0644358Z   Pool Info:               
2025-08-14T21:19:53.0644724Z     Pool 1                   
2025-08-14T21:19:53.0645197Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0645847Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0646407Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0646985Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0647603Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0648223Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0648815Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0649327Z     Pool 2                   
2025-08-14T21:19:53.0649786Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0650350Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0650894Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0651502Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0652130Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0652748Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0653357Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0653867Z     Pool 3                   
2025-08-14T21:19:53.0654335Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0654875Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0655434Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0656025Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0656630Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0657265Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0657869Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0658390Z     Pool 4                   
2025-08-14T21:19:53.0658820Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0659349Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0659901Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0660474Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0661089Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0661703Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0662312Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0662817Z   ISA Info:                
2025-08-14T21:19:53.0663470Z     ISA 1                    
2025-08-14T21:19:53.0664240Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0664888Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0665501Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0666098Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0666738Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0667302Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0667877Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0668414Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0668901Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0669396Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0669869Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0670411Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0670934Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0671381Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0671863Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0672344Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0672892Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0673432Z *******                  
2025-08-14T21:19:53.0673787Z Agent 10                 
2025-08-14T21:19:53.0674130Z *******                  
2025-08-14T21:19:53.0674531Z   Name:                    gfx90a                             
2025-08-14T21:19:53.0675069Z   Uuid:                    GPU-bef9ac6669b03ca0               
2025-08-14T21:19:53.0675672Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:19:53.0676281Z   Vendor Name:             AMD                                
2025-08-14T21:19:53.0676851Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:19:53.0677421Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:19:53.0677989Z   Float Round Mode:        NEAR                               
2025-08-14T21:19:53.0678578Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:19:53.0679148Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:19:53.0679719Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:19:53.0680282Z   Queue Type:              MULTI                              
2025-08-14T21:19:53.0680832Z   Node:                    9                                  
2025-08-14T21:19:53.0681379Z   Device Type:             GPU                                
2025-08-14T21:19:53.0681879Z   Cache Info:              
2025-08-14T21:19:53.0682300Z     L1:                      16(0x10) KB                        
2025-08-14T21:19:53.0682790Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:19:53.0683307Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:19:53.0683846Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:19:53.0684442Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:19:53.0685039Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:19:53.0685656Z   BDFID:                   37632                              
2025-08-14T21:19:53.0686221Z   Internal Node ID:        9                                  
2025-08-14T21:19:53.0686793Z   Compute Unit:            104                                
2025-08-14T21:19:53.0687353Z   SIMDs per CU:            4                                  
2025-08-14T21:19:53.0687907Z   Shader Engines:          8                                  
2025-08-14T21:19:53.0688784Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:19:53.0689672Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:19:53.0690296Z   Coherent Host Access:    FALSE                              
2025-08-14T21:19:53.0690982Z   Memory Properties:       
2025-08-14T21:19:53.0691397Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:19:53.0691938Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:19:53.0692523Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:19:53.0693113Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0693645Z   Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0694091Z     x                        1024(0x400)                        
2025-08-14T21:19:53.0694572Z     y                        1024(0x400)                        
2025-08-14T21:19:53.0695029Z     z                        1024(0x400)                        
2025-08-14T21:19:53.0695561Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:19:53.0696147Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:19:53.0696758Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0697293Z   Grid Max Size per Dimension:
2025-08-14T21:19:53.0697733Z     x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0698260Z     y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0698747Z     z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0699310Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:19:53.0699949Z   Packet Processor uCode:: 92                                 
2025-08-14T21:19:53.0700553Z   SDMA engine uCode::      9                                  
2025-08-14T21:19:53.0701144Z   IOMMU Support::          None                               
2025-08-14T21:19:53.0701658Z   Pool Info:               
2025-08-14T21:19:53.0702061Z     Pool 1                   
2025-08-14T21:19:53.0702566Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:19:53.0703144Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0703712Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0704351Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0704980Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0705630Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0706233Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0706771Z     Pool 2                   
2025-08-14T21:19:53.0707239Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:19:53.0707813Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0708369Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0708983Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0709643Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0710283Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0710934Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0711449Z     Pool 3                   
2025-08-14T21:19:53.0711931Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:19:53.0712480Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:19:53.0713038Z       Allocatable:             TRUE                               
2025-08-14T21:19:53.0713630Z       Alloc Granule:           4KB                                
2025-08-14T21:19:53.0714245Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:19:53.0715116Z       Alloc Alignment:         4KB                                
2025-08-14T21:19:53.0715928Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0716445Z     Pool 4                   
2025-08-14T21:19:53.0716877Z       Segment:                 GROUP                              
2025-08-14T21:19:53.0717420Z       Size:                    64(0x40) KB                        
2025-08-14T21:19:53.0717979Z       Allocatable:             FALSE                              
2025-08-14T21:19:53.0718564Z       Alloc Granule:           0KB                                
2025-08-14T21:19:53.0719178Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:19:53.0719795Z       Alloc Alignment:         0KB                                
2025-08-14T21:19:53.0720398Z       Accessible by all:       FALSE                              
2025-08-14T21:19:53.0720916Z   ISA Info:                
2025-08-14T21:19:53.0721299Z     ISA 1                    
2025-08-14T21:19:53.0721796Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:19:53.0722440Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:19:53.0723059Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:19:53.0723665Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0724303Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:19:53.0724877Z       Fast f16:                TRUE                               
2025-08-14T21:19:53.0725466Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:19:53.0726327Z       Workgroup Max Size per Dimension:
2025-08-14T21:19:53.0726804Z         x                        1024(0x400)                        
2025-08-14T21:19:53.0727302Z         y                        1024(0x400)                        
2025-08-14T21:19:53.0727770Z         z                        1024(0x400)                        
2025-08-14T21:19:53.0728320Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:19:53.0728851Z       Grid Max Size per Dimension:
2025-08-14T21:19:53.0729296Z         x                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0729777Z         y                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0730273Z         z                        4294967295(0xffffffff)             
2025-08-14T21:19:53.0730822Z       FBarrier Max Size:       32                                 
2025-08-14T21:19:53.0731325Z *** Done ***             
2025-08-14T21:19:53.0753413Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:19:53.0753733Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:19:53.0754276Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-14T21:19:53.0754767Z if [[ $ngpu -eq 0 ]]; then
2025-08-14T21:19:53.0755039Z     echo "Error: Failed to detect any GPUs on the runner"
2025-08-14T21:19:53.0755320Z     echo "$msg"
2025-08-14T21:19:53.0755497Z     exit 1
2025-08-14T21:19:53.0755645Z fi
2025-08-14T21:19:53.0786728Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.0786993Z env:
2025-08-14T21:19:53.0787147Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.0787324Z ##[endgroup]
2025-08-14T21:19:53.2543896Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main
2025-08-14T21:19:53.2544629Z with:
2025-08-14T21:19:53.2545021Z   diskspace-cutoff: 70
2025-08-14T21:19:53.2545429Z env:
2025-08-14T21:19:53.2545802Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.2546229Z ##[endgroup]
2025-08-14T21:19:53.2617579Z ##[group]Run set -ex
2025-08-14T21:19:53.2618069Z set -ex
2025-08-14T21:19:53.2618457Z diskspace_cutoff=70
2025-08-14T21:19:53.2619169Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-08-14T21:19:53.2620455Z if [ ! -d "$docker_root_dir" ]; then
2025-08-14T21:19:53.2621906Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-08-14T21:19:53.2622905Z     exit 0
2025-08-14T21:19:53.2623349Z fi
2025-08-14T21:19:53.2624204Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T21:19:53.2625922Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-14T21:19:53.2627209Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-08-14T21:19:53.2627835Z     docker system prune -af
2025-08-14T21:19:53.2628708Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T21:19:53.2629681Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-08-14T21:19:53.2630705Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-08-14T21:19:53.2631782Z         echo "$msg"
2025-08-14T21:19:53.2632289Z         exit 1
2025-08-14T21:19:53.2632762Z     else
2025-08-14T21:19:53.2633297Z         difference=$((diskspace - diskspace_new))
2025-08-14T21:19:53.2634100Z         echo "Diskspace saved: $difference percent"
2025-08-14T21:19:53.2634673Z     fi
2025-08-14T21:19:53.2635018Z fi
2025-08-14T21:19:53.2697888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.2698473Z env:
2025-08-14T21:19:53.2698825Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.2699274Z ##[endgroup]
2025-08-14T21:19:53.2795806Z + diskspace_cutoff=70
2025-08-14T21:19:53.2803960Z ++ docker info -f '{{.DockerRootDir}}'
2025-08-14T21:19:53.3781026Z + docker_root_dir=/media/4TB/docker-rootless
2025-08-14T21:19:53.3781967Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-08-14T21:19:53.3796742Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-08-14T21:19:53.3799237Z ++ sed -n 2p
2025-08-14T21:19:53.3807251Z ++ sed s/%//
2025-08-14T21:19:53.3808102Z ++ sed 's/ //'
2025-08-14T21:19:53.3846909Z + diskspace=14
2025-08-14T21:19:53.3848314Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-08-14T21:19:53.3849623Z + [[ 14 -ge 70 ]]
2025-08-14T21:19:53.3903394Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-08-14T21:19:53.3904204Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts"
2025-08-14T21:19:53.3904856Z rm -rf "${RUNNER_ARTIFACT_DIR}"
2025-08-14T21:19:53.3905450Z mkdir -p "${RUNNER_ARTIFACT_DIR}"
2025-08-14T21:19:53.3924989Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:19:53.3925885Z 
2025-08-14T21:19:53.3926563Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results"
2025-08-14T21:19:53.3927432Z rm -rf "${RUNNER_TEST_RESULTS_DIR}"
2025-08-14T21:19:53.3928085Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}"
2025-08-14T21:19:53.3928948Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:19:53.3929719Z 
2025-08-14T21:19:53.3930210Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs"
2025-08-14T21:19:53.3930821Z rm -rf "${RUNNER_DOCS_DIR}"
2025-08-14T21:19:53.3931360Z mkdir -p "${RUNNER_DOCS_DIR}"
2025-08-14T21:19:53.3932034Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}"
2025-08-14T21:19:53.3994349Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.3994999Z env:
2025-08-14T21:19:53.3995352Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.3995779Z ##[endgroup]
2025-08-14T21:19:53.4336060Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:19:53.4337429Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:19:53.4338644Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}"
2025-08-14T21:19:53.4407280Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.4408045Z env:
2025-08-14T21:19:53.4408481Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.4409309Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:53.4410426Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:53.4411406Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:53.4412082Z ##[endgroup]
2025-08-14T21:19:53.4644812Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-08-14T21:19:53.4646066Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py.
2025-08-14T21:19:53.4646873Z # Add render group for container creation.
2025-08-14T21:19:53.4647645Z render_gid=`cat /etc/group | grep render | cut -d: -f3`
2025-08-14T21:19:53.4648567Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG.
2025-08-14T21:19:53.4649464Z if [ -f "/etc/podinfo/gha-render-devices" ]; then
2025-08-14T21:19:53.4650112Z   DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices)
2025-08-14T21:19:53.4650665Z else
2025-08-14T21:19:53.4651054Z   DEVICE_FLAG="--device /dev/dri"
2025-08-14T21:19:53.4651487Z fi
2025-08-14T21:19:53.4652196Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively.
2025-08-14T21:19:53.4653306Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal.
2025-08-14T21:19:53.4654359Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries.
2025-08-14T21:19:53.4655456Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary.
2025-08-14T21:19:53.4657321Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}"
2025-08-14T21:19:53.4713914Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.4714493Z env:
2025-08-14T21:19:53.4714813Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.4715461Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:53.4716411Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:53.4717292Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:53.4717888Z ##[endgroup]
2025-08-14T21:19:53.4938981Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:19:53.4939848Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx')
2025-08-14T21:19:53.4940443Z if [[ $ngpu -lt 4 ]]; then
2025-08-14T21:19:53.4941264Z   echo "Error: only $ngpu GPU(s) detected, at least 4 GPUs are needed for distributed jobs"
2025-08-14T21:19:53.4942069Z   exit 1
2025-08-14T21:19:53.4942431Z fi
2025-08-14T21:19:53.5005643Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:53.5006292Z env:
2025-08-14T21:19:53.5006653Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.5007452Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:53.5008726Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:53.5009773Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:53.5011493Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:53.5013357Z ##[endgroup]
2025-08-14T21:19:53.6784831Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-08-14T21:19:53.6785773Z with:
2025-08-14T21:19:53.6786392Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only
2025-08-14T21:19:53.6787137Z   aws-region: us-east-1
2025-08-14T21:19:53.6787581Z   role-duration-seconds: 18000
2025-08-14T21:19:53.6788060Z   audience: sts.amazonaws.com
2025-08-14T21:19:53.6788506Z env:
2025-08-14T21:19:53.6788896Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:53.6789743Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:53.6790997Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:53.6792108Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:53.6793839Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:53.6795311Z ##[endgroup]
2025-08-14T21:19:54.0340012Z Assuming role with OIDC
2025-08-14T21:19:54.2641331Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions
2025-08-14T21:19:54.3442023Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076
2025-08-14T21:19:54.3442890Z with:
2025-08-14T21:19:54.3443285Z   mask-password: true
2025-08-14T21:19:54.3443735Z   registry-type: private
2025-08-14T21:19:54.3444184Z   skip-logout: false
2025-08-14T21:19:54.3444594Z env:
2025-08-14T21:19:54.3444966Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:54.3446114Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:54.3447237Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:54.3448486Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:54.3450414Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:54.3451941Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:54.3452480Z   AWS_REGION: us-east-1
2025-08-14T21:19:54.3453481Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:54.3454166Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:54.3464600Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:54.3465047Z ##[endgroup]
2025-08-14T21:19:54.7614609Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.2344288Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main
2025-08-14T21:19:55.2345168Z with:
2025-08-14T21:19:55.2346562Z   docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2348085Z   use-custom-docker-registry: true
2025-08-14T21:19:55.2348669Z   docker-build-dir: .ci/docker
2025-08-14T21:19:55.2349206Z   docker-build-script: ./build.sh
2025-08-14T21:19:55.2349718Z   working-directory: .
2025-08-14T21:19:55.2350329Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.2351027Z   force-push: false
2025-08-14T21:19:55.2351428Z env:
2025-08-14T21:19:55.2351813Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:55.2352567Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:55.2353708Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:55.2354801Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:55.2356509Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:55.2358520Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:55.2359023Z   AWS_REGION: us-east-1
2025-08-14T21:19:55.2359652Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:55.2360335Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:55.2371473Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:55.2371964Z ##[endgroup]
2025-08-14T21:19:55.2411349Z ##[group]Run set -ex
2025-08-14T21:19:55.2411878Z set -ex
2025-08-14T21:19:55.2412269Z 
2025-08-14T21:19:55.2412963Z # If the docker build directory or the build script doesn't exist, the action will
2025-08-14T21:19:55.2414113Z # gracefully return the docker image name as it is.  Pulling docker image in Linux
2025-08-14T21:19:55.2415098Z # job could then download the pre-built image as usual
2025-08-14T21:19:55.2416266Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then
2025-08-14T21:19:55.2417372Z   echo "skip=false" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2417944Z else
2025-08-14T21:19:55.2418406Z   echo "skip=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2419177Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2419878Z 
2025-08-14T21:19:55.2420831Z   echo "Not using custom ECR registry.  Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..."
2025-08-14T21:19:55.2421907Z   exit 0
2025-08-14T21:19:55.2422283Z fi
2025-08-14T21:19:55.2422648Z 
2025-08-14T21:19:55.2423236Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then
2025-08-14T21:19:55.2424269Z   # The docker image name already includes the ECR prefix and tag, so we can just
2025-08-14T21:19:55.2425181Z   # use it as it is, but first let's extract the tag
2025-08-14T21:19:55.2426014Z   DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}')
2025-08-14T21:19:55.2426930Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2427780Z   echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2428477Z else
2025-08-14T21:19:55.2428945Z   if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then
2025-08-14T21:19:55.2429625Z     CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:}
2025-08-14T21:19:55.2430303Z     DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*}
2025-08-14T21:19:55.2430877Z   fi
2025-08-14T21:19:55.2432127Z   DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}")
2025-08-14T21:19:55.2433213Z   echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2434312Z   echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2435511Z   echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2436269Z fi
2025-08-14T21:19:55.2504184Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:55.2504877Z env:
2025-08-14T21:19:55.2505290Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:55.2506052Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:55.2507171Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:55.2508200Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:55.2509924Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:55.2511446Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:55.2511947Z   AWS_REGION: us-east-1
2025-08-14T21:19:55.2512534Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:55.2541314Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:55.2546189Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:55.2546403Z   REPO_NAME: pytorch
2025-08-14T21:19:55.2546951Z   DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2547522Z   DOCKER_BUILD_DIR: .ci/docker
2025-08-14T21:19:55.2547738Z   DOCKER_BUILD_SCRIPT: ./build.sh
2025-08-14T21:19:55.2548018Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.2548323Z   USE_CUSTOM_DOCKER_REGISTRY: true
2025-08-14T21:19:55.2548544Z   CUSTOM_TAG_PREFIX: 
2025-08-14T21:19:55.2548745Z ##[endgroup]
2025-08-14T21:19:55.2613265Z + [[ -d .ci/docker ]]
2025-08-14T21:19:55.2613923Z + [[ -f .ci/docker/./build.sh ]]
2025-08-14T21:19:55.2614494Z + [[ true == \t\r\u\e ]]
2025-08-14T21:19:55.2614975Z + echo skip=false
2025-08-14T21:19:55.2616887Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]]
2025-08-14T21:19:55.2633946Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2635872Z ++ awk -F '[:,]' '{print $2}'
2025-08-14T21:19:55.2693920Z + DOCKER_TAG=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2695351Z + echo docker-tag=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2702299Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2766103Z ##[group]Run set +e
2025-08-14T21:19:55.2766646Z set +e
2025-08-14T21:19:55.2767078Z set -x
2025-08-14T21:19:55.2767485Z 
2025-08-14T21:19:55.2767881Z login() {
2025-08-14T21:19:55.2768740Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-08-14T21:19:55.2769636Z }
2025-08-14T21:19:55.2770007Z 
2025-08-14T21:19:55.2770368Z retry () {
2025-08-14T21:19:55.2770836Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-08-14T21:19:55.2771379Z }
2025-08-14T21:19:55.2771743Z 
2025-08-14T21:19:55.2772148Z retry login "${DOCKER_REGISTRY}"
2025-08-14T21:19:55.2772676Z 
2025-08-14T21:19:55.2773052Z START_TIME=$(date +%s)
2025-08-14T21:19:55.2774060Z # Wait up to 120 minutes
2025-08-14T21:19:55.2774699Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do
2025-08-14T21:19:55.2775548Z   # Check if image already exists, if it does then skip building it
2025-08-14T21:19:55.2776386Z   if docker manifest inspect "${DOCKER_IMAGE}"; then
2025-08-14T21:19:55.2777022Z     exit 0
2025-08-14T21:19:55.2777433Z   fi
2025-08-14T21:19:55.2777798Z 
2025-08-14T21:19:55.2778447Z   # NB: This flag is used by Docker build workflow to push the image to ECR, so we can
2025-08-14T21:19:55.2779566Z   # use this to differentiate between the Docker build and regular build jobs. For the
2025-08-14T21:19:55.2780693Z   # latter, it will wait for the Docker images to become available before continuing
2025-08-14T21:19:55.2781575Z   if [ "${DOCKER_PUSH:-false}" == "true" ]; then
2025-08-14T21:19:55.2782274Z     # It's a Docker build job, let's build the image
2025-08-14T21:19:55.2782874Z     break
2025-08-14T21:19:55.2783282Z   else
2025-08-14T21:19:55.2783879Z     # It's a regular build job, wait for the image to become available
2025-08-14T21:19:55.2784589Z     sleep 300
2025-08-14T21:19:55.2785006Z   fi
2025-08-14T21:19:55.2785800Z done
2025-08-14T21:19:55.2786172Z 
2025-08-14T21:19:55.2786804Z # NB: This part requires a full checkout. Otherwise, the merge base will
2025-08-14T21:19:55.2787816Z # be empty.  The default action would be to continue rebuild the image
2025-08-14T21:19:55.2788726Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then
2025-08-14T21:19:55.2789521Z   # if we're on the base branch then use the parent commit
2025-08-14T21:19:55.2790349Z   MERGE_BASE=$(git rev-parse HEAD~)
2025-08-14T21:19:55.2791067Z else
2025-08-14T21:19:55.2791808Z   # otherwise we're on a PR, so use the most recent base commit
2025-08-14T21:19:55.2792679Z   MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION")
2025-08-14T21:19:55.2793304Z fi
2025-08-14T21:19:55.2793813Z 
2025-08-14T21:19:55.2794258Z if [[ -z "${MERGE_BASE}" ]]; then
2025-08-14T21:19:55.2794998Z   echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2795611Z 
2025-08-14T21:19:55.2796413Z   echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..."
2025-08-14T21:19:55.2797416Z   exit 0
2025-08-14T21:19:55.2797803Z fi
2025-08-14T21:19:55.2798173Z 
2025-08-14T21:19:55.2798709Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then
2025-08-14T21:19:55.2799870Z   echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit"
2025-08-14T21:19:55.2800882Z   exit 1
2025-08-14T21:19:55.2801272Z fi
2025-08-14T21:19:55.2801652Z 
2025-08-14T21:19:55.2802282Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}")
2025-08-14T21:19:55.2803409Z # If no image exists but the hash is the same as the previous hash then we should error out here
2025-08-14T21:19:55.2804410Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then
2025-08-14T21:19:55.2805929Z   echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch"
2025-08-14T21:19:55.2807257Z   echo "         Will re-build docker image to store in local cache, TTS may be longer"
2025-08-14T21:19:55.2808032Z fi
2025-08-14T21:19:55.2808404Z 
2025-08-14T21:19:55.2808855Z echo "rebuild=true" >> "${GITHUB_OUTPUT}"
2025-08-14T21:19:55.2876553Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:55.2877241Z env:
2025-08-14T21:19:55.2877656Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:55.2878773Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:55.2879923Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:55.2880958Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:55.2882692Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:55.2884206Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:55.2884726Z   AWS_REGION: us-east-1
2025-08-14T21:19:55.2885352Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:55.2886497Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:55.2895769Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:55.2896186Z   DOCKER_BUILD_DIR: .ci/docker
2025-08-14T21:19:55.2896722Z   BASE_REVISION: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:19:55.2898013Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2899494Z   DOCKER_TAG: pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:55.2900421Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.2901362Z   DOCKER_PUSH: 
2025-08-14T21:19:55.2901718Z ##[endgroup]
2025-08-14T21:19:55.3001634Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.3002447Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:55.3010494Z + aws ecr get-login-password --region us-east-1
2025-08-14T21:19:55.3014078Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:56.5517661Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-08-14T21:19:56.5518818Z Configure a credential helper to remove this warning. See
2025-08-14T21:19:56.5519887Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-08-14T21:19:56.5520581Z 
2025-08-14T21:19:56.5526051Z Login Succeeded
2025-08-14T21:19:56.5587223Z ++ date +%s
2025-08-14T21:19:56.5613536Z + START_TIME=1755206396
2025-08-14T21:19:56.5623554Z ++ date +%s
2025-08-14T21:19:56.5652175Z + [[ 1755199196 -lt 1755206396 ]]
2025-08-14T21:19:56.5653756Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:57.4747137Z {
2025-08-14T21:19:57.4747612Z 	"schemaVersion": 2,
2025-08-14T21:19:57.4753202Z 	"mediaType": "application/vnd.docker.distribution.manifest.v2+json",
2025-08-14T21:19:57.4754157Z 	"config": {
2025-08-14T21:19:57.4754795Z 		"mediaType": "application/vnd.docker.container.image.v1+json",
2025-08-14T21:19:57.4755510Z 		"size": 28646,
2025-08-14T21:19:57.4756256Z 		"digest": "sha256:460acde6d8cbae0e961d7e3597d10525c881798e91edaf1ab0df7a07e0905a24"
2025-08-14T21:19:57.4757139Z 	},
2025-08-14T21:19:57.4757527Z 	"layers": [
2025-08-14T21:19:57.4757886Z 		{
2025-08-14T21:19:57.4758469Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4759185Z 			"size": 30448173,
2025-08-14T21:19:57.4759911Z 			"digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495"
2025-08-14T21:19:57.4760730Z 		},
2025-08-14T21:19:57.4761060Z 		{
2025-08-14T21:19:57.4761636Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4762342Z 			"size": 1554,
2025-08-14T21:19:57.4763023Z 			"digest": "sha256:c7b4a852a45516e27a9256df90878663d770f96d271d6155d43be78cc5225eef"
2025-08-14T21:19:57.4763791Z 		},
2025-08-14T21:19:57.4764123Z 		{
2025-08-14T21:19:57.4764698Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4765414Z 			"size": 313280717,
2025-08-14T21:19:57.4766355Z 			"digest": "sha256:c5628c7e7b33c6491651d9cce9ee596279d693cc7c3dcfa912aca94f75341cf7"
2025-08-14T21:19:57.4767730Z 		},
2025-08-14T21:19:57.4768071Z 		{
2025-08-14T21:19:57.4768647Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4769365Z 			"size": 704,
2025-08-14T21:19:57.4770073Z 			"digest": "sha256:836ab08052e8eb2bae68e69ae086fd23a5f04a8491c320718ab47f84f03aebb1"
2025-08-14T21:19:57.4770874Z 		},
2025-08-14T21:19:57.4771188Z 		{
2025-08-14T21:19:57.4771732Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4772421Z 			"size": 1218,
2025-08-14T21:19:57.4773119Z 			"digest": "sha256:0802c59adcde1d758d228271cc5f1bf6acbc597053ff8e8f3c1073d59a85bf9d"
2025-08-14T21:19:57.4773913Z 		},
2025-08-14T21:19:57.4774230Z 		{
2025-08-14T21:19:57.4774790Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4775596Z 			"size": 485,
2025-08-14T21:19:57.4776283Z 			"digest": "sha256:e97311a6a967664cbe10c5027a1ec60c514caa9a1160167d8363088fd1f9fe09"
2025-08-14T21:19:57.4777052Z 		},
2025-08-14T21:19:57.4777383Z 		{
2025-08-14T21:19:57.4777919Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4778606Z 			"size": 110343847,
2025-08-14T21:19:57.4779338Z 			"digest": "sha256:fe9f48a18b7ef6bdbd006ea4cb784b7e73e0daa2409a1a053c02bafed7993529"
2025-08-14T21:19:57.4780136Z 		},
2025-08-14T21:19:57.4780849Z 		{
2025-08-14T21:19:57.4781393Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4782083Z 			"size": 4242,
2025-08-14T21:19:57.4782793Z 			"digest": "sha256:4d3edeebe7da3872c026bd4e61d5f63aee7f760e1c8cf0eb226e1d95e528735b"
2025-08-14T21:19:57.4783591Z 		},
2025-08-14T21:19:57.4783908Z 		{
2025-08-14T21:19:57.4784447Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4785111Z 			"size": 1709,
2025-08-14T21:19:57.4785808Z 			"digest": "sha256:5a5cc76ada432cccf7d18e0eb79379afb95deaaa7afec482406267924d291ae4"
2025-08-14T21:19:57.4786600Z 		},
2025-08-14T21:19:57.4786913Z 		{
2025-08-14T21:19:57.4787451Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4788135Z 			"size": 724,
2025-08-14T21:19:57.4788804Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:19:57.4789569Z 		},
2025-08-14T21:19:57.4789885Z 		{
2025-08-14T21:19:57.4790426Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4791122Z 			"size": 3244135942,
2025-08-14T21:19:57.4791843Z 			"digest": "sha256:ee623dde94b52da337d3cc0e356115ccbc33b9b5ec03f9e06fce9f05ba3572ad"
2025-08-14T21:19:57.4792637Z 		},
2025-08-14T21:19:57.4792954Z 		{
2025-08-14T21:19:57.4793486Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4794167Z 			"size": 380,
2025-08-14T21:19:57.4794833Z 			"digest": "sha256:d6226eb61f823984003d5ac28f4d66fec9b27baf5d54a9513286483f5912cd88"
2025-08-14T21:19:57.4795607Z 		},
2025-08-14T21:19:57.4795921Z 		{
2025-08-14T21:19:57.4796461Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4797148Z 			"size": 234112,
2025-08-14T21:19:57.4797834Z 			"digest": "sha256:6de917b18d3a540adb3b7e446e9802f89a69d39f804f40d25751639b6cea7c33"
2025-08-14T21:19:57.4798596Z 		},
2025-08-14T21:19:57.4798915Z 		{
2025-08-14T21:19:57.4799445Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4800140Z 			"size": 793,
2025-08-14T21:19:57.4800815Z 			"digest": "sha256:76a69b57b6837bef07dbc1b481cf28a62dfd7c7063219d9f6e0d0d63067653c7"
2025-08-14T21:19:57.4801586Z 		},
2025-08-14T21:19:57.4801905Z 		{
2025-08-14T21:19:57.4802445Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4803121Z 			"size": 106,
2025-08-14T21:19:57.4803803Z 			"digest": "sha256:451d7a75db7137d6ca0ccb4e3635d4a16f3287db4cc2a3a4d3dbc2d3b0266f06"
2025-08-14T21:19:57.4804571Z 		},
2025-08-14T21:19:57.4804881Z 		{
2025-08-14T21:19:57.4805420Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4806576Z 			"size": 1496,
2025-08-14T21:19:57.4807268Z 			"digest": "sha256:acf5babd87f23aa905883eb434073e9a00ff41679134f2f4827dd86949f5a9d9"
2025-08-14T21:19:57.4808045Z 		},
2025-08-14T21:19:57.4808357Z 		{
2025-08-14T21:19:57.4808885Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4809582Z 			"size": 453553322,
2025-08-14T21:19:57.4810309Z 			"digest": "sha256:06e2fbb6cb2eeccd028bfaabb2e674bf25c9d84d3fa91c9e2f03bcf765f92599"
2025-08-14T21:19:57.4811113Z 		},
2025-08-14T21:19:57.4811415Z 		{
2025-08-14T21:19:57.4811934Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4812613Z 			"size": 163,
2025-08-14T21:19:57.4813332Z 			"digest": "sha256:87ddced081ff535d58b842d961844bf1b5979ddc8ad7f2d31f606cd6c73ec758"
2025-08-14T21:19:57.4814142Z 		},
2025-08-14T21:19:57.4814482Z 		{
2025-08-14T21:19:57.4815031Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4815765Z 			"size": 2484,
2025-08-14T21:19:57.4816498Z 			"digest": "sha256:1eb879bde5e5b9ce59b91eacb794a63e1180a55615094477886cba5016e993d7"
2025-08-14T21:19:57.4817321Z 		},
2025-08-14T21:19:57.4817662Z 		{
2025-08-14T21:19:57.4818215Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4818922Z 			"size": 8102124021,
2025-08-14T21:19:57.4819972Z 			"digest": "sha256:c3717fc4dc94ce7fdd93c87843ab97f72e05c0e87203ae15e48b87935b189a27"
2025-08-14T21:19:57.4820761Z 		},
2025-08-14T21:19:57.4821079Z 		{
2025-08-14T21:19:57.4821629Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4822308Z 			"size": 105,
2025-08-14T21:19:57.4823013Z 			"digest": "sha256:0c9ed178fd2ebc5a6630c12887a96b3f9884b85144d726435e77d33ab1ee11ab"
2025-08-14T21:19:57.4823799Z 		},
2025-08-14T21:19:57.4824120Z 		{
2025-08-14T21:19:57.4824658Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4825347Z 			"size": 612,
2025-08-14T21:19:57.4826032Z 			"digest": "sha256:b4626f6a341ca316ed5525f9c12e667b82f444b8cf673a0ad0bf8b78ccfd5bc9"
2025-08-14T21:19:57.4826819Z 		},
2025-08-14T21:19:57.4827145Z 		{
2025-08-14T21:19:57.4827687Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4828393Z 			"size": 677677354,
2025-08-14T21:19:57.4829117Z 			"digest": "sha256:4bf07717a6a2b5fd5a05d41b16dedebb413da5d50ae09457e112d5b42ecacbe1"
2025-08-14T21:19:57.4830119Z 		},
2025-08-14T21:19:57.4830464Z 		{
2025-08-14T21:19:57.4831021Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4831705Z 			"size": 111,
2025-08-14T21:19:57.4832374Z 			"digest": "sha256:f258db683e793ef4f283958f771de6376461ec4c2d2ac2947f2ad72a998f45ad"
2025-08-14T21:19:57.4833145Z 		},
2025-08-14T21:19:57.4833472Z 		{
2025-08-14T21:19:57.4834015Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4834711Z 			"size": 1555,
2025-08-14T21:19:57.4835419Z 			"digest": "sha256:2d70f6b6782f0d7c3342f35a89c20ca75e63acc6cffff00c2f5b0c4ae16aa20d"
2025-08-14T21:19:57.4836198Z 		},
2025-08-14T21:19:57.4836524Z 		{
2025-08-14T21:19:57.4837061Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4837751Z 			"size": 107,
2025-08-14T21:19:57.4838452Z 			"digest": "sha256:cfbe9750bae7f94f985be33742a90ac52adfdbe5870aa09b09ad6ce80db6541b"
2025-08-14T21:19:57.4839267Z 		},
2025-08-14T21:19:57.4839588Z 		{
2025-08-14T21:19:57.4840139Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4840838Z 			"size": 166,
2025-08-14T21:19:57.4841550Z 			"digest": "sha256:8d2ff9ef7f4c24e23debf7d02ee44f5acc6e3ed95168f970dfa061e270556036"
2025-08-14T21:19:57.4842350Z 		},
2025-08-14T21:19:57.4842668Z 		{
2025-08-14T21:19:57.4843238Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4843957Z 			"size": 2838566,
2025-08-14T21:19:57.4844672Z 			"digest": "sha256:59a73147541eccf1318db0147d0e40952ff983b3515bbc1b2586caa5d370c2e5"
2025-08-14T21:19:57.4846122Z 		},
2025-08-14T21:19:57.4846504Z 		{
2025-08-14T21:19:57.4847094Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4847812Z 			"size": 107,
2025-08-14T21:19:57.4848524Z 			"digest": "sha256:80e89acaf7bc105eb24090a7638ae51ec2ec270cf4e181e3002f3ef6cea7d2dc"
2025-08-14T21:19:57.4849340Z 		},
2025-08-14T21:19:57.4849675Z 		{
2025-08-14T21:19:57.4850221Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4850927Z 			"size": 802,
2025-08-14T21:19:57.4851618Z 			"digest": "sha256:c9c2b424b8e08d943dc259a3796d66eede3a1e93a6460df5db132c0036d3d6af"
2025-08-14T21:19:57.4852414Z 		},
2025-08-14T21:19:57.4852738Z 		{
2025-08-14T21:19:57.4853292Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4854007Z 			"size": 26113684,
2025-08-14T21:19:57.4854714Z 			"digest": "sha256:26060b8c841ee0f2c22485c0d51390219230b08ce81ce14c500e40be58488ac5"
2025-08-14T21:19:57.4855488Z 		},
2025-08-14T21:19:57.4855821Z 		{
2025-08-14T21:19:57.4856358Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4857058Z 			"size": 104,
2025-08-14T21:19:57.4857744Z 			"digest": "sha256:58b4bcac07b333b658450cb576e2bb832b11cd699d76f1f3846e739deb4a5738"
2025-08-14T21:19:57.4858527Z 		},
2025-08-14T21:19:57.4859186Z 		{
2025-08-14T21:19:57.4859732Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4860421Z 			"size": 425,
2025-08-14T21:19:57.4861113Z 			"digest": "sha256:18c4ed1ec491095788e352ae018afd84de0f251fbcfb8f74d5d893e1e9ab196d"
2025-08-14T21:19:57.4861900Z 		},
2025-08-14T21:19:57.4862221Z 		{
2025-08-14T21:19:57.4862758Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4863437Z 			"size": 19308715,
2025-08-14T21:19:57.4864144Z 			"digest": "sha256:4d7414e539c3ad1ab34e46032ca390dae2d25805222308bb02ecc1b35713b1a2"
2025-08-14T21:19:57.4864914Z 		},
2025-08-14T21:19:57.4865239Z 		{
2025-08-14T21:19:57.4865800Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4866495Z 			"size": 691,
2025-08-14T21:19:57.4867180Z 			"digest": "sha256:6738ba83282e002d92bff3d2b4951e3c1a67f5ec2c1bad2fd780c2f5d444748f"
2025-08-14T21:19:57.4867965Z 		},
2025-08-14T21:19:57.4868295Z 		{
2025-08-14T21:19:57.4868859Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4869557Z 			"size": 724,
2025-08-14T21:19:57.4870216Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:19:57.4870983Z 		},
2025-08-14T21:19:57.4871308Z 		{
2025-08-14T21:19:57.4871840Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4872526Z 			"size": 116,
2025-08-14T21:19:57.4873193Z 			"digest": "sha256:dfb0f24886393e1d394f1f433dc9346026679dafd7a60c3a93de17d94078c1ca"
2025-08-14T21:19:57.4873974Z 		},
2025-08-14T21:19:57.4874306Z 		{
2025-08-14T21:19:57.4874850Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4875544Z 			"size": 136,
2025-08-14T21:19:57.4876224Z 			"digest": "sha256:dc833b0762f2e144670a660f6b7ce62cec71a5fdd24df4e67b5c6173d5834451"
2025-08-14T21:19:57.4876992Z 		},
2025-08-14T21:19:57.4877311Z 		{
2025-08-14T21:19:57.4877842Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4878535Z 			"size": 139,
2025-08-14T21:19:57.4879210Z 			"digest": "sha256:8827df8ca2da347e0032d1bff3b0312437f711c5d0b5f2164f8a60c3368a9827"
2025-08-14T21:19:57.4879979Z 		},
2025-08-14T21:19:57.4880295Z 		{
2025-08-14T21:19:57.4880823Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4881503Z 			"size": 32,
2025-08-14T21:19:57.4882190Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-08-14T21:19:57.4882981Z 		},
2025-08-14T21:19:57.4883288Z 		{
2025-08-14T21:19:57.4883823Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4884782Z 			"size": 214,
2025-08-14T21:19:57.4885589Z 			"digest": "sha256:cd36d0dbfc741c82fee3a71ab009d02c22329a76463482a521b51f4e183248b3"
2025-08-14T21:19:57.4886386Z 		},
2025-08-14T21:19:57.4886721Z 		{
2025-08-14T21:19:57.4887256Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4887962Z 			"size": 347,
2025-08-14T21:19:57.4888628Z 			"digest": "sha256:4ba8e7a736c8199931fd7ff9931a5f17b7b931d0383a3e158f1b12b191a1d250"
2025-08-14T21:19:57.4889399Z 		},
2025-08-14T21:19:57.4889712Z 		{
2025-08-14T21:19:57.4890259Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4890950Z 			"size": 88301,
2025-08-14T21:19:57.4891661Z 			"digest": "sha256:d3f2353a1f26b127b1869d0eb92a134748c89b45f5defec4fcb2c356bc73c540"
2025-08-14T21:19:57.4892450Z 		},
2025-08-14T21:19:57.4892780Z 		{
2025-08-14T21:19:57.4893323Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4894019Z 			"size": 106,
2025-08-14T21:19:57.4894715Z 			"digest": "sha256:22c681dc84cd6e1f55b9874ead0fb9d31d471d5cb641391a3b3dd979c1502cc6"
2025-08-14T21:19:57.4895505Z 		},
2025-08-14T21:19:57.4895838Z 		{
2025-08-14T21:19:57.4896371Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4897063Z 			"size": 1666,
2025-08-14T21:19:57.4898145Z 			"digest": "sha256:519f3067ec6a99456e6fb280bd4b79e2478677684831299d601c9493523b67cc"
2025-08-14T21:19:57.4898914Z 		},
2025-08-14T21:19:57.4899233Z 		{
2025-08-14T21:19:57.4899770Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4900460Z 			"size": 724,
2025-08-14T21:19:57.4901131Z 			"digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9"
2025-08-14T21:19:57.4901902Z 		},
2025-08-14T21:19:57.4902215Z 		{
2025-08-14T21:19:57.4902747Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4903417Z 			"size": 139,
2025-08-14T21:19:57.4904100Z 			"digest": "sha256:6a8272f38d227d78a8f9b1ddf10bb73941fbd77e71ae3c582c27e08c0f6261c2"
2025-08-14T21:19:57.4904891Z 		},
2025-08-14T21:19:57.4905205Z 		{
2025-08-14T21:19:57.4905743Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4906425Z 			"size": 120,
2025-08-14T21:19:57.4907073Z 			"digest": "sha256:973e2484504fec2c1292e24122716507e8ca4717a35c142358174b60c0e401cf"
2025-08-14T21:19:57.4907841Z 		},
2025-08-14T21:19:57.4908156Z 		{
2025-08-14T21:19:57.4908690Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4909374Z 			"size": 5373253071,
2025-08-14T21:19:57.4910084Z 			"digest": "sha256:ac32a18ead25409bbcd67a8636c32d8af5852a347ea5503c7c9c6234cacf45a1"
2025-08-14T21:19:57.4910864Z 		},
2025-08-14T21:19:57.4911183Z 		{
2025-08-14T21:19:57.4911713Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4912392Z 			"size": 175,
2025-08-14T21:19:57.4913077Z 			"digest": "sha256:76846613d7067ca9de6a73db198902ca46b403bc78e2627f3f6fb805e1bbaa0a"
2025-08-14T21:19:57.4913833Z 		},
2025-08-14T21:19:57.4914151Z 		{
2025-08-14T21:19:57.4914684Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4915366Z 			"size": 1896,
2025-08-14T21:19:57.4916039Z 			"digest": "sha256:dcf1e01c98d6a6f72674d79a4e8e4047b54796576cd06ad682c225a92820a8f5"
2025-08-14T21:19:57.4916821Z 		},
2025-08-14T21:19:57.4917142Z 		{
2025-08-14T21:19:57.4917677Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4918361Z 			"size": 197509462,
2025-08-14T21:19:57.4919047Z 			"digest": "sha256:a88175fa027cfade948a7004100834f68828b13c0f21b314491c37084fbbe7e5"
2025-08-14T21:19:57.4919800Z 		},
2025-08-14T21:19:57.4920119Z 		{
2025-08-14T21:19:57.4920653Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4921346Z 			"size": 303,
2025-08-14T21:19:57.4922012Z 			"digest": "sha256:7f5277f691672469f431fd90a8c2bb702c6c68333f6be2cff868f00e416c5a1a"
2025-08-14T21:19:57.4923124Z 		},
2025-08-14T21:19:57.4923450Z 		{
2025-08-14T21:19:57.4923988Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4924677Z 			"size": 32,
2025-08-14T21:19:57.4925367Z 			"digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1"
2025-08-14T21:19:57.4926289Z 		},
2025-08-14T21:19:57.4926605Z 		{
2025-08-14T21:19:57.4927140Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4927820Z 			"size": 108,
2025-08-14T21:19:57.4928501Z 			"digest": "sha256:d0b078cebcc48c1f46a5d1286ae68a90c295cb6b81f033ba9a2f9c2498d6d7e5"
2025-08-14T21:19:57.4929280Z 		},
2025-08-14T21:19:57.4929587Z 		{
2025-08-14T21:19:57.4930133Z 			"mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
2025-08-14T21:19:57.4930818Z 			"size": 54145699,
2025-08-14T21:19:57.4931529Z 			"digest": "sha256:b01371e6ad0895119adb1d1dbfa20844d061be6aecadf9165128cebb28b9fd8e"
2025-08-14T21:19:57.4932320Z 		}
2025-08-14T21:19:57.4932649Z 	]
2025-08-14T21:19:57.4932974Z }
2025-08-14T21:19:57.4933324Z + exit 0
2025-08-14T21:19:57.4981641Z ##[group]Run set -eux
2025-08-14T21:19:57.4982197Z set -eux
2025-08-14T21:19:57.4983701Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin
2025-08-14T21:19:57.5051083Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:57.5051768Z env:
2025-08-14T21:19:57.5052158Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:57.5052919Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:57.5054068Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:57.5055088Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:57.5056816Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:57.5058390Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:57.5058877Z   AWS_REGION: us-east-1
2025-08-14T21:19:57.5059480Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:57.5060156Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:57.5070278Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:57.5070736Z ##[endgroup]
2025-08-14T21:19:57.5186553Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token
2025-08-14T21:19:57.5188339Z + jq --raw-output .SecretString
2025-08-14T21:19:57.5193209Z + jq -r .docker_hub_readonly_token
2025-08-14T21:19:57.5196194Z + docker login --username pytorchbot --password-stdin
2025-08-14T21:19:58.3032128Z 
2025-08-14T21:19:58.3035100Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action
2025-08-14T21:19:58.4090020Z Error: Cannot perform an interactive login from a non TTY device
2025-08-14T21:19:58.4168526Z ##[error]Process completed with exit code 1.
2025-08-14T21:19:58.4381412Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2025-08-14T21:19:58.4382217Z with:
2025-08-14T21:19:58.4383444Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:58.4384997Z   docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:58.4385673Z env:
2025-08-14T21:19:58.4386064Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:58.4386814Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:58.4387918Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:58.4388929Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:58.4390713Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:58.4392268Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:58.4392767Z   AWS_REGION: us-east-1
2025-08-14T21:19:58.4393342Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:58.4394009Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:58.4404094Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:58.4404552Z ##[endgroup]
2025-08-14T21:19:58.4434051Z ##[group]Run set -x
2025-08-14T21:19:58.4434580Z set -x
2025-08-14T21:19:58.4435005Z set +e
2025-08-14T21:19:58.4435396Z 
2025-08-14T21:19:58.4435785Z login() {
2025-08-14T21:19:58.4436653Z   aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1"
2025-08-14T21:19:58.4437568Z }
2025-08-14T21:19:58.4437952Z 
2025-08-14T21:19:58.4438349Z retry () {
2025-08-14T21:19:58.4438852Z   $*  || (sleep 1 && $*) || (sleep 2 && $*)
2025-08-14T21:19:58.4439425Z }
2025-08-14T21:19:58.4439828Z 
2025-08-14T21:19:58.4440683Z retry login "${DOCKER_REGISTRY}"
2025-08-14T21:19:58.4441252Z 
2025-08-14T21:19:58.4442101Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024')
2025-08-14T21:19:58.4443232Z echo "Compressed size of image in MB: ${IMAGE_SIZE}"
2025-08-14T21:19:58.4443886Z 
2025-08-14T21:19:58.4444280Z set -e
2025-08-14T21:19:58.4444890Z # ignore output since only exit code is used for conditional
2025-08-14T21:19:58.4446108Z # only pull docker image if it's not available locally
2025-08-14T21:19:58.4447063Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2025-08-14T21:19:58.4447928Z   retry docker pull "${DOCKER_IMAGE}"
2025-08-14T21:19:58.4448511Z fi
2025-08-14T21:19:58.4511426Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:19:58.4512113Z env:
2025-08-14T21:19:58.4512560Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:19:58.4513352Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:19:58.4514473Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:19:58.4515538Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:19:58.4517267Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:19:58.4518802Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:19:58.4519339Z   AWS_REGION: us-east-1
2025-08-14T21:19:58.4519939Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:19:58.4520626Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:19:58.4530793Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:19:58.4532545Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:58.4534075Z   DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:58.4534772Z ##[endgroup]
2025-08-14T21:19:58.4638925Z + set +e
2025-08-14T21:19:58.4639575Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:58.4640377Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:58.4647274Z + aws ecr get-login-password --region us-east-1
2025-08-14T21:19:58.4651753Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T21:19:59.7629732Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json.
2025-08-14T21:19:59.7630877Z Configure a credential helper to remove this warning. See
2025-08-14T21:19:59.7631904Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores
2025-08-14T21:19:59.7632601Z 
2025-08-14T21:19:59.7634438Z Login Succeeded
2025-08-14T21:19:59.7710296Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:19:59.7712310Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024'
2025-08-14T21:20:00.6971887Z + IMAGE_SIZE=17743.216945648193
2025-08-14T21:20:00.6972586Z Compressed size of image in MB: 17743.216945648193
2025-08-14T21:20:00.6973360Z + echo 'Compressed size of image in MB: 17743.216945648193'
2025-08-14T21:20:00.6973996Z + set -e
2025-08-14T21:20:00.6975393Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:20:00.7425993Z Prepare all required actions
2025-08-14T21:20:00.7481678Z ##[group]Run ./.github/actions/get-workflow-job-id
2025-08-14T21:20:00.7482236Z with:
2025-08-14T21:20:00.7482962Z   github-token: ***
2025-08-14T21:20:00.7483350Z env:
2025-08-14T21:20:00.7483700Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:20:00.7484376Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:20:00.7485919Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:20:00.7486831Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:20:00.7488630Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:20:00.7490089Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:20:00.7490542Z   AWS_REGION: us-east-1
2025-08-14T21:20:00.7491105Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:20:00.7491706Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:20:00.7500735Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:20:00.7501206Z ##[endgroup]
2025-08-14T21:20:00.7527686Z ##[group]Run set -eux
2025-08-14T21:20:00.7528212Z set -eux
2025-08-14T21:20:00.7529042Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-08-14T21:20:00.7591536Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:20:00.7592161Z env:
2025-08-14T21:20:00.7592548Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:20:00.7593224Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:20:00.7594203Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:20:00.7595118Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:20:00.7596639Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:20:00.7598017Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:20:00.7598486Z   AWS_REGION: us-east-1
2025-08-14T21:20:00.7599009Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:20:00.7599682Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:20:00.7609081Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:20:00.7609720Z   GITHUB_TOKEN: ***
2025-08-14T21:20:00.7610122Z ##[endgroup]
2025-08-14T21:20:00.7708765Z + python3 .github/scripts/get_workflow_job_id.py 16976370303 gpud535
2025-08-14T21:20:01.5302614Z Setting output job-id=48128182504
2025-08-14T21:20:01.5303837Z Setting output job-name=linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:20:01.5595058Z Prepare all required actions
2025-08-14T21:20:01.5595838Z Getting action download info
2025-08-14T21:20:01.7533664Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6)
2025-08-14T21:20:02.7479153Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093)
2025-08-14T21:20:03.5910973Z ##[group]Run ./.github/actions/download-build-artifacts
2025-08-14T21:20:03.5911255Z with:
2025-08-14T21:20:03.5911439Z   name: linux-jammy-rocm-py3.10
2025-08-14T21:20:03.5911683Z   s3-bucket: gha-artifacts
2025-08-14T21:20:03.5911879Z env:
2025-08-14T21:20:03.5912048Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:20:03.5912361Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:20:03.5912817Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:20:03.5913278Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:20:03.5914002Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:20:03.5914642Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:20:03.5914856Z   AWS_REGION: us-east-1
2025-08-14T21:20:03.5915096Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:20:03.5915385Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:20:03.5919497Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:20:03.5919694Z ##[endgroup]
2025-08-14T21:20:03.5947520Z ##[group]Run seemethere/download-artifact-s3@v4
2025-08-14T21:20:03.5947791Z with:
2025-08-14T21:20:03.5947968Z   name: linux-jammy-rocm-py3.10
2025-08-14T21:20:03.5948200Z   s3-bucket: gha-artifacts
2025-08-14T21:20:03.5948411Z   region: us-east-1
2025-08-14T21:20:03.5948589Z env:
2025-08-14T21:20:03.5948754Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:20:03.5949084Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:20:03.5949541Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:20:03.5949977Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:20:03.5950684Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:20:03.5951313Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:20:03.5951525Z   AWS_REGION: us-east-1
2025-08-14T21:20:03.5951766Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:20:03.5952050Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:20:03.5981892Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:20:03.5982218Z ##[endgroup]
2025-08-14T21:20:04.0488993Z (node:2765425) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-08-14T21:20:04.0490001Z 
2025-08-14T21:20:04.0490476Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-08-14T21:20:04.0491544Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-08-14T21:20:04.0492629Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-08-14T21:20:04.2597913Z Found 1 objects with prefix pytorch/pytorch/16976370303/linux-jammy-rocm-py3.10/
2025-08-14T21:20:04.2599360Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-08-14T21:21:15.1414793Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip
2025-08-14T21:21:15.1432298Z Artifact download has finished successfully
2025-08-14T21:21:15.1971268Z ##[group]Run unzip -o artifacts.zip
2025-08-14T21:21:15.1971988Z unzip -o artifacts.zip
2025-08-14T21:21:15.2042193Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:15.2042881Z env:
2025-08-14T21:21:15.2043272Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:15.2044581Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:15.2045843Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:15.2046862Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:15.2048554Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:15.2050061Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:15.2050576Z   AWS_REGION: us-east-1
2025-08-14T21:21:15.2051160Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:15.2051845Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:15.2062554Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:15.2063082Z ##[endgroup]
2025-08-14T21:21:15.2203563Z Archive:  artifacts.zip
2025-08-14T21:21:15.2204893Z    creating: dist/
2025-08-14T21:21:20.5114093Z   inflating: dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl  
2025-08-14T21:21:20.5241999Z   inflating: dist/.ninja_log         
2025-08-14T21:21:20.5242731Z    creating: build/custom_test_artifacts/
2025-08-14T21:21:20.5243518Z    creating: build/custom_test_artifacts/custom-op-build/
2025-08-14T21:21:20.5244493Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2025-08-14T21:21:20.5245745Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/
2025-08-14T21:21:20.5247162Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:21:20.5249273Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/
2025-08-14T21:21:20.5250457Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:21:20.5251721Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:21:20.5252982Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:21:20.5254405Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:21:20.5255824Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:21:20.5257164Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:21:20.5258440Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:21:20.5259675Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:21:20.5261148Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:21:20.5262635Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:21:20.5264008Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:21:20.5265471Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:21:20.5267046Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:21:20.5268475Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/
2025-08-14T21:21:20.5269566Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2025-08-14T21:21:20.5270705Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:21:20.5271903Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2025-08-14T21:21:20.5273201Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2025-08-14T21:21:20.5274694Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2025-08-14T21:21:20.5276558Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2025-08-14T21:21:20.5277904Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2025-08-14T21:21:20.5279263Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2025-08-14T21:21:20.5280660Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2025-08-14T21:21:20.5282041Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2025-08-14T21:21:20.5283413Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2025-08-14T21:21:20.5284751Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2025-08-14T21:21:20.5287313Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2025-08-14T21:21:20.5468418Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2025-08-14T21:21:20.5469897Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2025-08-14T21:21:20.5471381Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2025-08-14T21:21:20.5472971Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2025-08-14T21:21:20.5474480Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2025-08-14T21:21:20.5476469Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2025-08-14T21:21:20.5477936Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2025-08-14T21:21:20.5479461Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2025-08-14T21:21:20.5480937Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2025-08-14T21:21:20.5482381Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2025-08-14T21:21:20.5483821Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2025-08-14T21:21:20.5493843Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2025-08-14T21:21:20.5568936Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2025-08-14T21:21:20.5570719Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:21:20.5572178Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:21:20.5573464Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2025-08-14T21:21:20.5574585Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2025-08-14T21:21:20.5575692Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:21:20.5576842Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:21:20.5578001Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:21:20.5579098Z   inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:21:20.5580134Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2025-08-14T21:21:20.5581061Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2025-08-14T21:21:20.5582008Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2025-08-14T21:21:20.5726713Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2025-08-14T21:21:20.5778001Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2025-08-14T21:21:20.5779169Z    creating: build/custom_test_artifacts/jit-hook-build/
2025-08-14T21:21:20.5780038Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2025-08-14T21:21:20.5781025Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/
2025-08-14T21:21:20.5782209Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:21:20.5783355Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/
2025-08-14T21:21:20.5784457Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:21:20.5785626Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:21:20.5786857Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:21:20.5788213Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:21:20.5789549Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:21:20.5790793Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:21:20.5791981Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:21:20.5793135Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:21:20.5794514Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:21:20.5796296Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:21:20.5797569Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:21:20.5798951Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:21:20.5800436Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:21:20.5801702Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/
2025-08-14T21:21:20.5802712Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2025-08-14T21:21:20.5803785Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:21:20.5804897Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2025-08-14T21:21:20.5806554Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2025-08-14T21:21:20.5807985Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2025-08-14T21:21:20.5809359Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2025-08-14T21:21:20.5810635Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2025-08-14T21:21:20.5811947Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2025-08-14T21:21:20.5813279Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2025-08-14T21:21:20.5814617Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2025-08-14T21:21:20.5815953Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2025-08-14T21:21:20.5817282Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2025-08-14T21:21:20.5821870Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2025-08-14T21:21:20.5880730Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2025-08-14T21:21:20.5882273Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:21:20.5883637Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:21:20.5884845Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2025-08-14T21:21:20.5886482Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2025-08-14T21:21:20.5887575Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:21:20.5888727Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:21:20.5889883Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:21:20.5890970Z   inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:21:20.5891991Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2025-08-14T21:21:20.5892902Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2025-08-14T21:21:20.5893849Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2025-08-14T21:21:20.5922756Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2025-08-14T21:21:20.5923680Z    creating: build/custom_test_artifacts/custom-backend-build/
2025-08-14T21:21:20.5924581Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2025-08-14T21:21:20.5926405Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/
2025-08-14T21:21:20.5927718Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml  
2025-08-14T21:21:20.5928899Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/
2025-08-14T21:21:20.5930081Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake  
2025-08-14T21:21:20.5931360Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/
2025-08-14T21:21:20.5932591Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/
2025-08-14T21:21:20.5933987Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c  
2025-08-14T21:21:20.5935421Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out  
2025-08-14T21:21:20.5936762Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake  
2025-08-14T21:21:20.5938074Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/
2025-08-14T21:21:20.5939349Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/
2025-08-14T21:21:20.5940813Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2025-08-14T21:21:20.5942318Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out  
2025-08-14T21:21:20.5943707Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake  
2025-08-14T21:21:20.5945206Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin  
2025-08-14T21:21:20.5946789Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin  
2025-08-14T21:21:20.5948172Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/
2025-08-14T21:21:20.5949276Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2025-08-14T21:21:20.5950425Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2025-08-14T21:21:20.5951973Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2025-08-14T21:21:20.5953397Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2025-08-14T21:21:20.5954946Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2025-08-14T21:21:20.5956438Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2025-08-14T21:21:20.5957833Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2025-08-14T21:21:20.5959299Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2025-08-14T21:21:20.5960769Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2025-08-14T21:21:20.5962268Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2025-08-14T21:21:20.5963739Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2025-08-14T21:21:20.5965170Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2025-08-14T21:21:20.5966815Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2025-08-14T21:21:20.6063514Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2025-08-14T21:21:20.6065000Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2025-08-14T21:21:20.6066848Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2025-08-14T21:21:20.6068482Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2025-08-14T21:21:20.6070059Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2025-08-14T21:21:20.6071545Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2025-08-14T21:21:20.6073054Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2025-08-14T21:21:20.6074581Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2025-08-14T21:21:20.6076110Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2025-08-14T21:21:20.6077636Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2025-08-14T21:21:20.6079126Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2025-08-14T21:21:20.6088663Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2025-08-14T21:21:20.6139322Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2025-08-14T21:21:20.6140960Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2025-08-14T21:21:20.6142378Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2025-08-14T21:21:20.6143645Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2025-08-14T21:21:20.6144820Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2025-08-14T21:21:20.6145980Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2025-08-14T21:21:20.6147208Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json  
2025-08-14T21:21:20.6148808Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc  
2025-08-14T21:21:20.6150003Z   inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc  
2025-08-14T21:21:20.6151103Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2025-08-14T21:21:20.6152100Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2025-08-14T21:21:20.6153136Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2025-08-14T21:21:20.6239589Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2025-08-14T21:21:20.6275400Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2025-08-14T21:21:20.6276217Z    creating: build/lib/
2025-08-14T21:21:20.6351205Z   inflating: build/lib/libprotobuf-lite.a  
2025-08-14T21:21:20.6757315Z   inflating: build/lib/libprotobuf.a  
2025-08-14T21:21:20.7208922Z   inflating: build/lib/libprotoc.a   
2025-08-14T21:21:20.7217319Z   inflating: build/lib/libpthreadpool.a  
2025-08-14T21:21:20.7224810Z   inflating: build/lib/libcpuinfo.a  
2025-08-14T21:21:20.7231940Z   inflating: build/lib/libcpuinfo_internals.a  
2025-08-14T21:21:20.7232668Z   inflating: build/lib/libclog.a     
2025-08-14T21:21:20.7249946Z   inflating: build/lib/libpytorch_qnnpack.a  
2025-08-14T21:21:20.7250970Z   inflating: build/lib/libnnpack_reference_layers.a  
2025-08-14T21:21:20.7267969Z   inflating: build/lib/libnnpack.a   
2025-08-14T21:21:20.7437261Z   inflating: build/lib/libmicrokernels-prod.a  
2025-08-14T21:21:20.8230690Z   inflating: build/lib/libmicrokernels-all.a  
2025-08-14T21:21:20.8294473Z   inflating: build/lib/libgtest.a    
2025-08-14T21:21:20.8310005Z   inflating: build/lib/libgmock.a    
2025-08-14T21:21:20.8310691Z   inflating: build/lib/libgmock_main.a  
2025-08-14T21:21:20.8311278Z   inflating: build/lib/libgtest_main.a  
2025-08-14T21:21:20.8393379Z   inflating: build/lib/libXNNPACK.a  
2025-08-14T21:21:20.8461551Z   inflating: build/lib/libbenchmark.a  
2025-08-14T21:21:20.8462281Z   inflating: build/lib/libbenchmark_main.a  
2025-08-14T21:21:20.8469438Z   inflating: build/lib/libittnotify.a  
2025-08-14T21:21:20.8470154Z   inflating: build/lib/libjitprofiling.a  
2025-08-14T21:21:20.8529166Z   inflating: build/lib/libasmjit.a   
2025-08-14T21:21:20.9632045Z   inflating: build/lib/libfbgemm.a   
2025-08-14T21:21:20.9659363Z   inflating: build/lib/libtensorpipe_uv.a  
2025-08-14T21:21:21.0168980Z   inflating: build/lib/libtensorpipe.a  
2025-08-14T21:21:21.0278900Z   inflating: build/lib/libgloo.a     
2025-08-14T21:21:21.0322249Z   inflating: build/lib/libonnx_proto.a  
2025-08-14T21:21:21.0724252Z   inflating: build/lib/libgloo_hip.a  
2025-08-14T21:21:21.1384192Z   inflating: build/lib/libonnx.a     
2025-08-14T21:21:22.0845970Z   inflating: build/lib/libdnnl.a     
2025-08-14T21:21:22.0862233Z   inflating: build/lib/libfmt.a      
2025-08-14T21:21:22.1124081Z   inflating: build/lib/libkineto.a   
2025-08-14T21:21:22.1222536Z   inflating: build/lib/libc10.so     
2025-08-14T21:21:22.1223267Z   inflating: build/lib/libtorch_global_deps.so  
2025-08-14T21:21:22.1223985Z   inflating: build/lib/libcaffe2_nvrtc.so  
2025-08-14T21:21:22.1259408Z   inflating: build/lib/libc10_hip.so  
2025-08-14T21:21:24.8452307Z   inflating: build/lib/libtorch_cpu.so  
2025-08-14T21:21:24.8454865Z   inflating: build/lib/libshm.so     
2025-08-14T21:21:25.6404649Z   inflating: build/lib/libtorch_hip.so  
2025-08-14T21:21:25.6405337Z   inflating: build/lib/libtorch.so   
2025-08-14T21:21:25.6422307Z   inflating: build/lib/libjitbackend_test.so  
2025-08-14T21:21:25.6486559Z   inflating: build/lib/libtorchbind_test.so  
2025-08-14T21:21:25.6508023Z   inflating: build/lib/libbackend_with_compiler.so  
2025-08-14T21:21:25.6532217Z   inflating: build/lib/libaoti_custom_ops.so  
2025-08-14T21:21:25.8384986Z   inflating: build/lib/libtorch_python.so  
2025-08-14T21:21:25.8414851Z   inflating: build/lib/libnnapi_backend.so  
2025-08-14T21:21:25.8415512Z    creating: build/bin/
2025-08-14T21:21:25.8416739Z    creating: build/bin/CMakeFiles/
2025-08-14T21:21:25.8417415Z   inflating: build/bin/cmake_install.cmake  
2025-08-14T21:21:25.8418032Z   inflating: build/bin/CTestTestfile.cmake  
2025-08-14T21:21:25.8824007Z   inflating: build/bin/protoc-3.13.0.0  
2025-08-14T21:21:25.9230596Z   inflating: build/bin/protoc        
2025-08-14T21:21:25.9283063Z   inflating: build/bin/c10_AllocatorConfig_test  
2025-08-14T21:21:25.9332708Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2025-08-14T21:21:25.9383282Z   inflating: build/bin/c10_DeviceGuard_test  
2025-08-14T21:21:25.9434523Z   inflating: build/bin/c10_Device_test  
2025-08-14T21:21:25.9492669Z   inflating: build/bin/c10_DispatchKeySet_test  
2025-08-14T21:21:25.9546072Z   inflating: build/bin/c10_Scalar_test  
2025-08-14T21:21:25.9594420Z   inflating: build/bin/c10_StreamGuard_test  
2025-08-14T21:21:25.9645039Z   inflating: build/bin/c10_SymInt_test  
2025-08-14T21:21:25.9699017Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2025-08-14T21:21:25.9754144Z   inflating: build/bin/c10_InlineStreamGuard_test  
2025-08-14T21:21:25.9809147Z   inflating: build/bin/c10_SizesAndStrides_test  
2025-08-14T21:21:25.9877497Z   inflating: build/bin/c10_cow_test  
2025-08-14T21:21:25.9926509Z   inflating: build/bin/c10_ConstexprCrc_test  
2025-08-14T21:21:25.9974659Z   inflating: build/bin/c10_ArrayRef_test  
2025-08-14T21:21:26.0027578Z   inflating: build/bin/c10_Bitset_test  
2025-08-14T21:21:26.0076973Z   inflating: build/bin/c10_DeadlockDetection_test  
2025-08-14T21:21:26.0132908Z   inflating: build/bin/c10_Enumerate_test  
2025-08-14T21:21:26.0183260Z   inflating: build/bin/c10_Half_test  
2025-08-14T21:21:26.0235240Z   inflating: build/bin/c10_IntrusiveList_test  
2025-08-14T21:21:26.0290599Z   inflating: build/bin/c10_LeftRight_test  
2025-08-14T21:21:26.0345040Z   inflating: build/bin/c10_Metaprogramming_test  
2025-08-14T21:21:26.0397805Z   inflating: build/bin/c10_NetworkFlow_test  
2025-08-14T21:21:26.0446842Z   inflating: build/bin/c10_Semaphore_test  
2025-08-14T21:21:26.0501726Z   inflating: build/bin/c10_ThreadLocal_test  
2025-08-14T21:21:26.0550946Z   inflating: build/bin/c10_Synchronized_test  
2025-08-14T21:21:26.0602387Z   inflating: build/bin/c10_TypeIndex_test  
2025-08-14T21:21:26.0652630Z   inflating: build/bin/c10_TypeList_test  
2025-08-14T21:21:26.0701432Z   inflating: build/bin/c10_TypeTraits_test  
2025-08-14T21:21:26.0752261Z   inflating: build/bin/c10_accumulate_test  
2025-08-14T21:21:26.0807477Z   inflating: build/bin/c10_bfloat16_test  
2025-08-14T21:21:26.0863090Z   inflating: build/bin/c10_complex_math_test  
2025-08-14T21:21:26.0914114Z   inflating: build/bin/c10_bit_cast_test  
2025-08-14T21:21:26.0968306Z   inflating: build/bin/c10_complex_test  
2025-08-14T21:21:26.1017538Z   inflating: build/bin/c10_error_test  
2025-08-14T21:21:26.1067058Z   inflating: build/bin/c10_flags_test  
2025-08-14T21:21:26.1118885Z   inflating: build/bin/c10_exception_test  
2025-08-14T21:21:26.1169262Z   inflating: build/bin/c10_irange_test  
2025-08-14T21:21:26.1219224Z   inflating: build/bin/c10_generic_math_test  
2025-08-14T21:21:26.1377147Z   inflating: build/bin/c10_intrusive_ptr_test  
2025-08-14T21:21:26.1429612Z   inflating: build/bin/c10_lazy_test  
2025-08-14T21:21:26.1485949Z   inflating: build/bin/c10_logging_test  
2025-08-14T21:21:26.1558792Z   inflating: build/bin/c10_optional_test  
2025-08-14T21:21:26.1611591Z   inflating: build/bin/c10_registry_test  
2025-08-14T21:21:26.1672051Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2025-08-14T21:21:26.1722979Z   inflating: build/bin/c10_ssize_test  
2025-08-14T21:21:26.1872192Z   inflating: build/bin/c10_small_vector_test  
2025-08-14T21:21:26.1927763Z   inflating: build/bin/c10_string_util_test  
2025-08-14T21:21:26.1977519Z   inflating: build/bin/c10_tempfile_test  
2025-08-14T21:21:26.2026507Z   inflating: build/bin/c10_string_view_test  
2025-08-14T21:21:26.2082027Z   inflating: build/bin/c10_typeid_test  
2025-08-14T21:21:26.2125411Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2025-08-14T21:21:26.2173873Z   inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test  
2025-08-14T21:21:26.2222170Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream  
2025-08-14T21:21:26.2271027Z   inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device  
2025-08-14T21:21:26.2319211Z   inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes  
2025-08-14T21:21:26.2368202Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads  
2025-08-14T21:21:26.2416539Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks  
2025-08-14T21:21:26.2465231Z   inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block  
2025-08-14T21:21:26.2513565Z   inflating: build/bin/c10_hip_HIPTest  
2025-08-14T21:21:26.3068047Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2025-08-14T21:21:26.3638152Z   inflating: build/bin/vec_test_all_types_AVX512  
2025-08-14T21:21:26.4213744Z   inflating: build/bin/vec_test_all_types_AVX2  
2025-08-14T21:21:26.4265468Z   inflating: build/bin/BackoffTest   
2025-08-14T21:21:26.4318018Z   inflating: build/bin/FileStoreTest  
2025-08-14T21:21:26.4373830Z   inflating: build/bin/TCPStoreTest  
2025-08-14T21:21:26.4427102Z   inflating: build/bin/HashStoreTest  
2025-08-14T21:21:26.4491565Z   inflating: build/bin/ProcessGroupGlooTest  
2025-08-14T21:21:26.4493333Z   inflating: build/bin/example_allreduce  
2025-08-14T21:21:26.4497732Z   inflating: build/bin/torch_shm_manager  
2025-08-14T21:21:26.4550557Z   inflating: build/bin/static_runtime_bench  
2025-08-14T21:21:26.4788215Z   inflating: build/bin/static_runtime_test  
2025-08-14T21:21:26.4860117Z   inflating: build/bin/Dict_test     
2025-08-14T21:21:26.4912007Z   inflating: build/bin/Dimname_test  
2025-08-14T21:21:26.4975980Z   inflating: build/bin/MaybeOwned_test  
2025-08-14T21:21:26.5032167Z   inflating: build/bin/NamedTensor_test  
2025-08-14T21:21:26.5089599Z   inflating: build/bin/apply_utils_test  
2025-08-14T21:21:26.5147578Z   inflating: build/bin/atest         
2025-08-14T21:21:26.5210139Z   inflating: build/bin/basic         
2025-08-14T21:21:26.5265074Z   inflating: build/bin/broadcast_test  
2025-08-14T21:21:26.5315249Z   inflating: build/bin/cpu_allocator_test  
2025-08-14T21:21:26.5372451Z   inflating: build/bin/cpu_generator_test  
2025-08-14T21:21:26.5424624Z   inflating: build/bin/cpu_profiling_allocator_test  
2025-08-14T21:21:26.5513721Z   inflating: build/bin/cpu_rng_test  
2025-08-14T21:21:26.5564051Z   inflating: build/bin/dlconvertor_test  
2025-08-14T21:21:26.5620510Z   inflating: build/bin/extension_backend_test  
2025-08-14T21:21:26.5675030Z   inflating: build/bin/half_test     
2025-08-14T21:21:26.5767310Z   inflating: build/bin/ivalue_test   
2025-08-14T21:21:26.5816750Z   inflating: build/bin/lazy_tensor_test  
2025-08-14T21:21:26.5869682Z   inflating: build/bin/math_kernel_test  
2025-08-14T21:21:26.5922142Z   inflating: build/bin/memory_format_test  
2025-08-14T21:21:26.5975262Z   inflating: build/bin/memory_overlapping_test  
2025-08-14T21:21:26.6027378Z   inflating: build/bin/mobile_memory_cleanup  
2025-08-14T21:21:26.6082569Z   inflating: build/bin/native_test   
2025-08-14T21:21:26.6132866Z   inflating: build/bin/operator_name_test  
2025-08-14T21:21:26.6183306Z   inflating: build/bin/operators_test  
2025-08-14T21:21:26.6234434Z   inflating: build/bin/packedtensoraccessor_test  
2025-08-14T21:21:26.6290965Z   inflating: build/bin/quantized_test  
2025-08-14T21:21:26.6356328Z   inflating: build/bin/pow_test      
2025-08-14T21:21:26.6406231Z   inflating: build/bin/reduce_ops_test  
2025-08-14T21:21:26.6456857Z   inflating: build/bin/reportMemoryUsage_test  
2025-08-14T21:21:26.6512585Z   inflating: build/bin/scalar_tensor_test  
2025-08-14T21:21:26.6563105Z   inflating: build/bin/StorageUtils_test  
2025-08-14T21:21:26.6620934Z   inflating: build/bin/scalar_test   
2025-08-14T21:21:26.6672215Z   inflating: build/bin/stride_properties_test  
2025-08-14T21:21:26.6749899Z   inflating: build/bin/tensor_iterator_test  
2025-08-14T21:21:26.6803923Z   inflating: build/bin/type_ptr_test  
2025-08-14T21:21:26.6857443Z   inflating: build/bin/test_parallel  
2025-08-14T21:21:26.6907472Z   inflating: build/bin/thread_init_test  
2025-08-14T21:21:26.6965811Z   inflating: build/bin/type_test     
2025-08-14T21:21:26.7017652Z   inflating: build/bin/undefined_tensor_test  
2025-08-14T21:21:26.7066644Z   inflating: build/bin/verify_api_visibility  
2025-08-14T21:21:26.7117133Z   inflating: build/bin/weakref_test  
2025-08-14T21:21:26.7185140Z   inflating: build/bin/legacy_vmap_test  
2025-08-14T21:21:26.7236008Z   inflating: build/bin/wrapdim_test  
2025-08-14T21:21:26.7286959Z   inflating: build/bin/xla_tensor_test  
2025-08-14T21:21:26.7345298Z   inflating: build/bin/IListRef_test  
2025-08-14T21:21:26.7448175Z   inflating: build/bin/List_test     
2025-08-14T21:21:26.7562723Z   inflating: build/bin/kernel_function_legacy_test  
2025-08-14T21:21:26.7655552Z   inflating: build/bin/kernel_function_test  
2025-08-14T21:21:26.7776127Z   inflating: build/bin/kernel_lambda_legacy_test  
2025-08-14T21:21:26.7840604Z   inflating: build/bin/KernelFunction_test  
2025-08-14T21:21:26.7939223Z   inflating: build/bin/kernel_lambda_test  
2025-08-14T21:21:26.7998209Z   inflating: build/bin/kernel_stackbased_test  
2025-08-14T21:21:26.8090377Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2025-08-14T21:21:26.8140694Z   inflating: build/bin/CppSignature_test  
2025-08-14T21:21:26.8189443Z   inflating: build/bin/op_allowlist_test  
2025-08-14T21:21:26.8254560Z   inflating: build/bin/inline_container_test  
2025-08-14T21:21:26.8550676Z   inflating: build/bin/op_registration_test  
2025-08-14T21:21:26.8604847Z   inflating: build/bin/backend_fallback_test  
2025-08-14T21:21:26.8653705Z   inflating: build/bin/hip_complex_math_test  
2025-08-14T21:21:26.8705807Z   inflating: build/bin/hip_apply_test  
2025-08-14T21:21:26.8754827Z   inflating: build/bin/hip_complex_test  
2025-08-14T21:21:26.8803238Z   inflating: build/bin/hip_distributions_test  
2025-08-14T21:21:26.8851985Z   inflating: build/bin/hip_generator_test  
2025-08-14T21:21:26.8900906Z   inflating: build/bin/hip_half_test  
2025-08-14T21:21:26.8949720Z   inflating: build/bin/hip_integer_divider_test  
2025-08-14T21:21:26.9004212Z   inflating: build/bin/hip_optional_test  
2025-08-14T21:21:26.9053235Z   inflating: build/bin/hip_packedtensoraccessor_test  
2025-08-14T21:21:26.9101420Z   inflating: build/bin/hip_vectorized_test  
2025-08-14T21:21:26.9152606Z   inflating: build/bin/hip_dlconvertor_test  
2025-08-14T21:21:27.0167813Z   inflating: build/bin/test_jit      
2025-08-14T21:21:27.0459585Z   inflating: build/bin/test_nativert  
2025-08-14T21:21:27.0513384Z   inflating: build/bin/test_dist_autograd  
2025-08-14T21:21:27.0579676Z   inflating: build/bin/test_cpp_rpc  
2025-08-14T21:21:27.1645965Z   inflating: build/bin/test_api      
2025-08-14T21:21:27.1646908Z   inflating: build/bin/parallel_benchmark  
2025-08-14T21:21:27.1974506Z   inflating: build/bin/test_lazy     
2025-08-14T21:21:27.1975176Z    creating: .additional_ci_files/
2025-08-14T21:21:27.2046857Z   inflating: .additional_ci_files/test-times.json  
2025-08-14T21:21:27.2318889Z   inflating: .additional_ci_files/test-class-times.json  
2025-08-14T21:21:27.2385789Z ##[group]Run rm artifacts.zip
2025-08-14T21:21:27.2386365Z rm artifacts.zip
2025-08-14T21:21:27.2456877Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:27.2457690Z env:
2025-08-14T21:21:27.2458152Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:27.2459082Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:27.2460437Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:27.2461663Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:27.2463961Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:27.2465504Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:27.2465997Z   AWS_REGION: us-east-1
2025-08-14T21:21:27.2466601Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:27.2467273Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:27.2477887Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:27.2478329Z ##[endgroup]
2025-08-14T21:21:27.5965411Z ##[group]Run df -H
2025-08-14T21:21:27.5966066Z df -H
2025-08-14T21:21:27.6036949Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:27.6037652Z env:
2025-08-14T21:21:27.6038042Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:27.6038794Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:27.6039893Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:27.6040962Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:27.6042695Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:27.6044204Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:27.6044710Z   AWS_REGION: us-east-1
2025-08-14T21:21:27.6045293Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:27.6046123Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:27.6056231Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:27.6056682Z ##[endgroup]
2025-08-14T21:21:27.6234336Z Filesystem                                          Size  Used Avail Use% Mounted on
2025-08-14T21:21:27.6235230Z tmpfs                                               109G   33M  109G   1% /run
2025-08-14T21:21:27.6236001Z /dev/nvme0n1p2                                      944G   79G  817G   9% /
2025-08-14T21:21:27.6236757Z tmpfs                                               542G   33k  542G   1% /dev/shm
2025-08-14T21:21:27.6237469Z tmpfs                                               5.3M     0  5.3M   0% /run/lock
2025-08-14T21:21:27.6238186Z /dev/nvme0n1p1                                      536M  6.4M  530M   2% /boot/efi
2025-08-14T21:21:27.6238953Z /dev/nvme1n1p1                                      3.8T  473G  3.2T  14% /media/4TB
2025-08-14T21:21:27.6239717Z tmpfs                                               109G   33k  109G   1% /run/user/1307800118
2025-08-14T21:21:27.6240514Z 172.18.148.8:/export/amd2                           5.5T  278G  5.3T   6% /mnt
2025-08-14T21:21:27.6241532Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch     108G  348M  108G   1% /home/amd-pytorch
2025-08-14T21:21:27.6242640Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage          165T  137T   29T  83% /groups
2025-08-14T21:21:27.6243828Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch  5.5T  1.8T  3.8T  33% /scratch
2025-08-14T21:21:27.6244861Z pure1.jax.cs.cpe.ice.amd.com:/homes/jflecken        108G  840M  107G   1% /home/jflecken
2025-08-14T21:21:27.6246117Z pure1.jax.cs.cpe.ice.amd.com:/homes/khughes         108G  4.1G  104G   4% /home/khughes
2025-08-14T21:21:27.6247178Z pure1.jax.cs.cpe.ice.amd.com:/homes/jlay            108G   19G   89G  18% /home/jlay
2025-08-14T21:21:27.6310494Z Prepare all required actions
2025-08-14T21:21:27.6311239Z Getting action download info
2025-08-14T21:21:27.9624423Z ##[group]Run ./.github/actions/download-td-artifacts
2025-08-14T21:21:27.9625051Z with:
2025-08-14T21:21:27.9625399Z env:
2025-08-14T21:21:27.9625769Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:27.9626521Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:27.9627637Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:27.9628647Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:27.9630355Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:27.9631875Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:27.9632373Z   AWS_REGION: us-east-1
2025-08-14T21:21:27.9632991Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:27.9633659Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:27.9643698Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:27.9644143Z ##[endgroup]
2025-08-14T21:21:27.9697467Z ##[group]Run seemethere/download-artifact-s3@v4
2025-08-14T21:21:27.9698001Z with:
2025-08-14T21:21:27.9698323Z   name: td_results
2025-08-14T21:21:27.9698706Z   s3-bucket: gha-artifacts
2025-08-14T21:21:27.9699135Z   region: us-east-1
2025-08-14T21:21:27.9699480Z env:
2025-08-14T21:21:27.9699810Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:27.9700467Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:27.9701444Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:27.9702361Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:27.9703911Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:27.9705259Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:27.9705710Z   AWS_REGION: us-east-1
2025-08-14T21:21:27.9706204Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:27.9706801Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:27.9715832Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:27.9716604Z ##[endgroup]
2025-08-14T21:21:28.4333807Z (node:2765527) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023.
2025-08-14T21:21:28.4334781Z 
2025-08-14T21:21:28.4335156Z Please migrate your code to use AWS SDK for JavaScript (v3).
2025-08-14T21:21:28.4336095Z For more information, check the migration guide at https://a.co/7PzMCcy
2025-08-14T21:21:28.4337057Z (Use `node --trace-warnings ...` to show where the warning was created)
2025-08-14T21:21:28.6327070Z Found 1 objects with prefix pytorch/pytorch/16976370303/td_results/
2025-08-14T21:21:28.6328400Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-08-14T21:21:28.9680713Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json
2025-08-14T21:21:28.9690995Z Artifact download has finished successfully
2025-08-14T21:21:29.0179256Z ##[group]Run mkdir -p .additional_ci_files
2025-08-14T21:21:29.0179620Z mkdir -p .additional_ci_files
2025-08-14T21:21:29.0180050Z mv td_results.json .additional_ci_files/td_results.json || true
2025-08-14T21:21:29.0244213Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:29.0244893Z env:
2025-08-14T21:21:29.0245292Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:29.0246394Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:29.0247474Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:29.0248481Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:29.0250692Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:29.0252485Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:29.0253075Z   AWS_REGION: us-east-1
2025-08-14T21:21:29.0253755Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:29.0254557Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:29.0261098Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:29.0261334Z ##[endgroup]
2025-08-14T21:21:29.0488594Z ##[group]Run .github/scripts/parse_ref.py
2025-08-14T21:21:29.0489344Z .github/scripts/parse_ref.py
2025-08-14T21:21:29.0542524Z shell: /usr/bin/bash -e {0}
2025-08-14T21:21:29.0542785Z env:
2025-08-14T21:21:29.0542992Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:29.0543445Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:29.0544573Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:29.0545784Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:29.0547564Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:29.0549082Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:29.0549582Z   AWS_REGION: us-east-1
2025-08-14T21:21:29.0550136Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:29.0550851Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:29.0560891Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:29.0561337Z ##[endgroup]
2025-08-14T21:21:29.1012547Z Setting output branch=main
2025-08-14T21:21:29.1249570Z Prepare all required actions
2025-08-14T21:21:29.1250343Z Getting action download info
2025-08-14T21:21:29.3319120Z ##[group]Run ./.github/actions/filter-test-configs
2025-08-14T21:21:29.3319656Z with:
2025-08-14T21:21:29.3320268Z   github-token: ***
2025-08-14T21:21:29.3322430Z   test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:21:29.3325826Z   job-name: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:21:29.3326688Z env:
2025-08-14T21:21:29.3327018Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:29.3327744Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:29.3328697Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:29.3329594Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:29.3331093Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:29.3332425Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:29.3332869Z   AWS_REGION: us-east-1
2025-08-14T21:21:29.3333362Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:29.3333984Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:29.3342884Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:29.3343280Z ##[endgroup]
2025-08-14T21:21:29.3398793Z ##[group]Run nick-fields/retry@v3.0.0
2025-08-14T21:21:29.3399268Z with:
2025-08-14T21:21:29.3399582Z   shell: bash
2025-08-14T21:21:29.3399944Z   timeout_minutes: 10
2025-08-14T21:21:29.3400314Z   max_attempts: 5
2025-08-14T21:21:29.3400677Z   retry_wait_seconds: 30
2025-08-14T21:21:29.3401871Z   command: set -eux
# PyYAML 6.0 doesn't work with MacOS x86 anymore
# This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2
python3 -m pip install requests==2.27.1 pyyaml==6.0.2

2025-08-14T21:21:29.3403118Z   polling_interval_seconds: 1
2025-08-14T21:21:29.3403546Z   warning_on_retry: true
2025-08-14T21:21:29.3403935Z   continue_on_error: false
2025-08-14T21:21:29.3404326Z env:
2025-08-14T21:21:29.3404655Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:29.3405312Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:29.3406665Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:29.3407565Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:29.3409113Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:29.3410500Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:29.3410937Z   AWS_REGION: us-east-1
2025-08-14T21:21:29.3411411Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:29.3411997Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:29.3420976Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:29.3421569Z   GITHUB_TOKEN: ***
2025-08-14T21:21:29.3421961Z ##[endgroup]
2025-08-14T21:21:29.4382352Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2
2025-08-14T21:21:29.6932726Z Defaulting to user installation because normal site-packages is not writeable
2025-08-14T21:21:29.8562428Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1)
2025-08-14T21:21:29.8565959Z Requirement already satisfied: pyyaml==6.0.2 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.2)
2025-08-14T21:21:29.8659256Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3)
2025-08-14T21:21:29.8666141Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12)
2025-08-14T21:21:29.8671566Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20)
2025-08-14T21:21:29.8676057Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5)
2025-08-14T21:21:30.4378592Z Command completed after 1 attempt(s).
2025-08-14T21:21:30.4526881Z ##[group]Run set -x
2025-08-14T21:21:30.4527399Z set -x
2025-08-14T21:21:30.4527841Z 
2025-08-14T21:21:30.4528637Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-08-14T21:21:30.4529557Z # in runner workspace
2025-08-14T21:21:30.4530265Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py"
2025-08-14T21:21:30.4600661Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:30.4601348Z env:
2025-08-14T21:21:30.4601723Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:30.4602488Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:30.4603616Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:30.4604652Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:30.4606466Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:30.4608205Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:30.4608786Z   AWS_REGION: us-east-1
2025-08-14T21:21:30.4609430Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:30.4610084Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:30.4620174Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:30.4620605Z ##[endgroup]
2025-08-14T21:21:30.4721949Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py
2025-08-14T21:21:30.5056746Z Setting output branch=main
2025-08-14T21:21:30.5157787Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}"
2025-08-14T21:21:30.5158498Z echo "Workflow: ${GITHUB_WORKFLOW}"
2025-08-14T21:21:30.5159080Z echo "Job name: ${JOB_NAME}"
2025-08-14T21:21:30.5159579Z 
2025-08-14T21:21:30.5160237Z # Use relative path here as this could be checked out anywhere, not necessarily
2025-08-14T21:21:30.5161089Z # in runner workspace
2025-08-14T21:21:30.5161828Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \
2025-08-14T21:21:30.5162665Z   --workflow "${GITHUB_WORKFLOW}" \
2025-08-14T21:21:30.5163248Z   --job-name "${JOB_NAME}" \
2025-08-14T21:21:30.5166037Z   --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}" \
2025-08-14T21:21:30.5168703Z   --selected-test-configs "" \
2025-08-14T21:21:30.5169285Z   --pr-number "${PR_NUMBER}" \
2025-08-14T21:21:30.5169828Z   --tag "${TAG}" \
2025-08-14T21:21:30.5170357Z   --event-name "${EVENT_NAME}" \
2025-08-14T21:21:30.5170915Z   --schedule "${SCHEDULE}" \
2025-08-14T21:21:30.5171449Z   --branch "${HEAD_BRANCH}"
2025-08-14T21:21:30.5238957Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:30.5239629Z env:
2025-08-14T21:21:30.5240013Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:30.5241205Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:30.5242319Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:30.5243338Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:30.5245056Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:30.5246947Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:30.5247679Z   AWS_REGION: us-east-1
2025-08-14T21:21:30.5249158Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:30.5250084Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:30.5260718Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:30.5261699Z   GITHUB_TOKEN: ***
2025-08-14T21:21:30.5262904Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:21:30.5264240Z   PR_NUMBER: 
2025-08-14T21:21:30.5264852Z   TAG: 
2025-08-14T21:21:30.5265408Z   EVENT_NAME: schedule
2025-08-14T21:21:30.5266215Z   SCHEDULE: 45 4,12,20 * * 1-5
2025-08-14T21:21:30.5266911Z   HEAD_BRANCH: main
2025-08-14T21:21:30.5267547Z ##[endgroup]
2025-08-14T21:21:30.5378172Z Workflow: periodic
2025-08-14T21:21:30.5379320Z Job name: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:21:30.9984804Z Setting output keep-going=True
2025-08-14T21:21:30.9985744Z Setting output ci-verbose-test-logs=False
2025-08-14T21:21:30.9987149Z Setting output ci-test-showlocals=False
2025-08-14T21:21:30.9988027Z Setting output ci-no-test-timeout=False
2025-08-14T21:21:30.9988723Z Setting output ci-no-td=False
2025-08-14T21:21:30.9989751Z Setting output ci-td-distributed=False
2025-08-14T21:21:30.9990559Z Setting output is-unstable=False
2025-08-14T21:21:30.9991474Z Setting output reenabled-issues=
2025-08-14T21:21:30.9994449Z Setting output test-matrix={"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}
2025-08-14T21:21:30.9997414Z Setting output is-test-matrix-empty=False
2025-08-14T21:21:31.0245329Z ##[group]Run echo "Filtered matrix:"
2025-08-14T21:21:31.0246373Z echo "Filtered matrix:"
2025-08-14T21:21:31.0249272Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}"
2025-08-14T21:21:31.0252050Z 
2025-08-14T21:21:31.0252688Z echo
2025-08-14T21:21:31.0253539Z echo "Is the current job unstable? False"
2025-08-14T21:21:31.0254405Z 
2025-08-14T21:21:31.0254953Z echo
2025-08-14T21:21:31.0255798Z echo "Is keep-going label set? True"
2025-08-14T21:21:31.0256630Z 
2025-08-14T21:21:31.0257306Z echo
2025-08-14T21:21:31.0258084Z echo "Reenabled issues? "
2025-08-14T21:21:31.0326065Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:31.0327097Z env:
2025-08-14T21:21:31.0327692Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:31.0328926Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:31.0330334Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:31.0332170Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:31.0334229Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:31.0336019Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:31.0337076Z   AWS_REGION: us-east-1
2025-08-14T21:21:31.0337900Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:31.0338813Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:31.0349314Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:31.0350008Z ##[endgroup]
2025-08-14T21:21:31.0454442Z Filtered matrix:
2025-08-14T21:21:31.0457372Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}]}
2025-08-14T21:21:31.0459755Z 
2025-08-14T21:21:31.0460124Z Is the current job unstable? False
2025-08-14T21:21:31.0460660Z 
2025-08-14T21:21:31.0461066Z Is keep-going label set? True
2025-08-14T21:21:31.0461433Z 
2025-08-14T21:21:31.0461761Z Reenabled issues? 
2025-08-14T21:21:31.0533816Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-08-14T21:21:31.0534651Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}"
2025-08-14T21:21:31.0588839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T21:21:31.0589427Z env:
2025-08-14T21:21:31.0589764Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:31.0590437Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:31.0591405Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:31.0592292Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:31.0593774Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:31.0595100Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:31.0595609Z   AWS_REGION: us-east-1
2025-08-14T21:21:31.0596088Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:31.0596670Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:31.0606169Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:31.0606594Z   JOB_TIMEOUT: 300
2025-08-14T21:21:31.0606941Z ##[endgroup]
2025-08-14T21:21:31.0792068Z ##[group]Run set -x
2025-08-14T21:21:31.0792642Z set -x
2025-08-14T21:21:31.0793068Z 
2025-08-14T21:21:31.0793520Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2025-08-14T21:21:31.0794211Z   TEST_COMMAND=.ci/pytorch/multigpu-test.sh
2025-08-14T21:21:31.0794905Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2025-08-14T21:21:31.0795529Z   TEST_COMMAND=.ci/caffe2/test.sh
2025-08-14T21:21:31.0796068Z else
2025-08-14T21:21:31.0796536Z   TEST_COMMAND=.ci/pytorch/test.sh
2025-08-14T21:21:31.0797080Z fi
2025-08-14T21:21:31.0797428Z 
2025-08-14T21:21:31.0798012Z # detached container should get cleaned up by teardown_ec2_linux
2025-08-14T21:21:31.0798934Z # TODO: Stop building test binaries as part of the build phase
2025-08-14T21:21:31.0799732Z # Used for GPU_FLAG since that doesn't play nice
2025-08-14T21:21:31.0800439Z # shellcheck disable=SC2086,SC2090
2025-08-14T21:21:31.0801029Z container_name=$(docker run \
2025-08-14T21:21:31.0801584Z   ${GPU_FLAG:-} \
2025-08-14T21:21:31.0802076Z   -e BUILD_ENVIRONMENT \
2025-08-14T21:21:31.0802582Z   -e PR_NUMBER \
2025-08-14T21:21:31.0803050Z   -e GITHUB_ACTIONS \
2025-08-14T21:21:31.0803541Z   -e GITHUB_REPOSITORY \
2025-08-14T21:21:31.0804039Z   -e GITHUB_WORKFLOW \
2025-08-14T21:21:31.0804524Z   -e GITHUB_JOB \
2025-08-14T21:21:31.0804980Z   -e GITHUB_RUN_ID \
2025-08-14T21:21:31.0805460Z   -e GITHUB_RUN_NUMBER \
2025-08-14T21:21:31.0806135Z   -e GITHUB_RUN_ATTEMPT \
2025-08-14T21:21:31.0806671Z   -e JOB_ID \
2025-08-14T21:21:31.0807103Z   -e JOB_NAME \
2025-08-14T21:21:31.0807547Z   -e BRANCH \
2025-08-14T21:21:31.0807962Z   -e SHA1 \
2025-08-14T21:21:31.0808396Z   -e AWS_DEFAULT_REGION \
2025-08-14T21:21:31.0808916Z   -e IN_WHEEL_TEST \
2025-08-14T21:21:31.0809394Z   -e SHARD_NUMBER \
2025-08-14T21:21:31.0810294Z   -e TEST_CONFIG \
2025-08-14T21:21:31.0810769Z   -e NUM_TEST_SHARDS \
2025-08-14T21:21:31.0811261Z   -e REENABLED_ISSUES \
2025-08-14T21:21:31.0811778Z   -e CONTINUE_THROUGH_ERROR \
2025-08-14T21:21:31.0812315Z   -e VERBOSE_TEST_LOGS \
2025-08-14T21:21:31.0812811Z   -e TEST_SHOWLOCALS \
2025-08-14T21:21:31.0813288Z   -e NO_TEST_TIMEOUT \
2025-08-14T21:21:31.0813744Z   -e NO_TD \
2025-08-14T21:21:31.0814230Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2025-08-14T21:21:31.0814856Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2025-08-14T21:21:31.0815499Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2025-08-14T21:21:31.0816074Z   -e TESTS_TO_INCLUDE \
2025-08-14T21:21:31.0816569Z   -e DASHBOARD_TAG \
2025-08-14T21:21:31.0817207Z   --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \
2025-08-14T21:21:31.0817909Z   --ulimit stack=10485760:83886080 \
2025-08-14T21:21:31.0818470Z   --ulimit core=0 \
2025-08-14T21:21:31.0819003Z   --security-opt seccomp=unconfined \
2025-08-14T21:21:31.0819582Z   --cap-add=SYS_PTRACE \
2025-08-14T21:21:31.0820078Z   --shm-size="8g" \
2025-08-14T21:21:31.0820528Z   --tty \
2025-08-14T21:21:31.0820925Z   --detach \
2025-08-14T21:21:31.0821384Z   --name="${container_name}" \
2025-08-14T21:21:31.0821916Z   --user jenkins \
2025-08-14T21:21:31.0822511Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2025-08-14T21:21:31.0823204Z   -w /var/lib/jenkins/workspace \
2025-08-14T21:21:31.0823759Z   "${DOCKER_IMAGE}"
2025-08-14T21:21:31.0824204Z )
2025-08-14T21:21:31.0824630Z # save container name for later step
2025-08-14T21:21:31.0825764Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV"
2025-08-14T21:21:31.0827059Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home
2025-08-14T21:21:31.0828682Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}"
2025-08-14T21:21:31.0898630Z shell: /usr/bin/bash -e {0}
2025-08-14T21:21:31.0899124Z env:
2025-08-14T21:21:31.0899507Z   GIT_DEFAULT_BRANCH: main
2025-08-14T21:21:31.0900269Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T21:21:31.0901368Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T21:21:31.0902535Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T21:21:31.0904561Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T21:21:31.0906152Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T21:21:31.0906662Z   AWS_REGION: us-east-1
2025-08-14T21:21:31.0907219Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T21:21:31.0907877Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T21:21:31.0917993Z   AWS_SESSION_TOKEN: ***
2025-08-14T21:21:31.0918520Z   BUILD_ENVIRONMENT: linux-jammy-rocm-py3.10
2025-08-14T21:21:31.0919097Z   PR_NUMBER: 
2025-08-14T21:21:31.0919546Z   GITHUB_REPOSITORY: pytorch/pytorch
2025-08-14T21:21:31.0920095Z   GITHUB_WORKFLOW: periodic
2025-08-14T21:21:31.0920554Z   GITHUB_JOB: test
2025-08-14T21:21:31.0920970Z   GITHUB_RUN_ID: 16976370303
2025-08-14T21:21:31.0921435Z   GITHUB_RUN_NUMBER: 17921
2025-08-14T21:21:31.0921888Z   GITHUB_RUN_ATTEMPT: 1
2025-08-14T21:21:31.0922306Z   JOB_ID: 48128182504
2025-08-14T21:21:31.0923188Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:21:31.0924133Z   BRANCH: main
2025-08-14T21:21:31.0924588Z   SHA1: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:21:31.0925758Z   CONTINUE_THROUGH_ERROR: True
2025-08-14T21:21:31.0926270Z   VERBOSE_TEST_LOGS: False
2025-08-14T21:21:31.0926722Z   TEST_SHOWLOCALS: False
2025-08-14T21:21:31.0927162Z   NO_TEST_TIMEOUT: False
2025-08-14T21:21:31.0927572Z   NO_TD: False
2025-08-14T21:21:31.0927970Z   TEST_CONFIG: distributed
2025-08-14T21:21:31.0928399Z   SHARD_NUMBER: 3
2025-08-14T21:21:31.0928781Z   NUM_TEST_SHARDS: 3
2025-08-14T21:21:31.0929190Z   REENABLED_ISSUES: 
2025-08-14T21:21:31.0930486Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:21:31.0932092Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2025-08-14T21:21:31.0932736Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2025-08-14T21:21:31.0933339Z   TESTS_TO_INCLUDE: 
2025-08-14T21:21:31.0933816Z   DASHBOARD_TAG: 
2025-08-14T21:21:31.0934276Z ##[endgroup]
2025-08-14T21:21:31.1034167Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2025-08-14T21:21:31.1034892Z + [[ linux-jammy-rocm-py3.10 == *onnx* ]]
2025-08-14T21:21:31.1035533Z + TEST_COMMAND=.ci/pytorch/test.sh
2025-08-14T21:21:31.1056878Z +++ nproc --ignore=2
2025-08-14T21:21:31.1097055Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_16976370303 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe
2025-08-14T21:21:31.3326727Z + container_name=e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T21:21:31.3332876Z + echo CONTAINER_NAME=e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T21:21:31.3334847Z + docker exec -t e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh'
2025-08-14T21:21:47.8190236Z Processing ./dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl
2025-08-14T21:21:48.5849045Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.18.0)
2025-08-14T21:21:48.5851502Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (4.14.1)
2025-08-14T21:21:48.5856552Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (1.13.3)
2025-08-14T21:21:48.5859663Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2.8.8)
2025-08-14T21:21:48.5862206Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.1.6)
2025-08-14T21:21:48.5867592Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2025.5.1)
2025-08-14T21:21:48.6168095Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git1fc683c) (1.3.0)
2025-08-14T21:21:48.6212553Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.9.0a0+git1fc683c) (3.0.2)
2025-08-14T21:21:48.9737420Z Installing collected packages: torch
2025-08-14T21:21:59.8268101Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
2025-08-14T21:21:59.8269892Z helion 0.1.0 requires filecheck, which is not installed.
2025-08-14T21:21:59.8270817Z Successfully installed torch-2.9.0a0+git1fc683c
2025-08-14T21:21:59.8771672Z + export TERM=vt100
2025-08-14T21:21:59.8772188Z + TERM=vt100
2025-08-14T21:21:59.8780874Z ++ dirname .ci/pytorch/test.sh
2025-08-14T21:21:59.8809810Z + source .ci/pytorch/common.sh
2025-08-14T21:21:59.8822574Z +++ dirname .ci/pytorch/common.sh
2025-08-14T21:21:59.8851110Z ++ source .ci/pytorch/common_utils.sh
2025-08-14T21:21:59.8854319Z +++ declare -f -t trap_add
2025-08-14T21:21:59.8864291Z ++ set -ex -o pipefail
2025-08-14T21:21:59.8864900Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:21:59.8865500Z ++ unset HIP_PLATFORM
2025-08-14T21:21:59.8865978Z ++ export PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:21:59.8866511Z ++ PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:21:59.8867325Z ++ BUILD_TEST_LIBTORCH=0
2025-08-14T21:21:59.8877455Z ++ dirname .ci/pytorch/test.sh
2025-08-14T21:21:59.8907994Z + source .ci/pytorch/common-build.sh
2025-08-14T21:21:59.8912336Z ++ [[ linux-jammy-rocm-py3.10 != *win-* ]]
2025-08-14T21:21:59.8934079Z ++++ dirname .ci/pytorch/common-build.sh
2025-08-14T21:21:59.8964265Z +++ cd .ci/pytorch
2025-08-14T21:21:59.8965194Z +++ pwd -P
2025-08-14T21:21:59.8975728Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch
2025-08-14T21:21:59.8977373Z ++ [[ linux-jammy-rocm-py3.10 == *-pch* ]]
2025-08-14T21:21:59.8978086Z ++ which sccache
2025-08-14T21:21:59.9009094Z ++ [[ -z '' ]]
2025-08-14T21:21:59.9009622Z ++ unset SCCACHE_BUCKET
2025-08-14T21:21:59.9010106Z ++ unset SCCACHE_REGION
2025-08-14T21:21:59.9010563Z ++ sccache --stop-server
2025-08-14T21:21:59.9083641Z ++ true
2025-08-14T21:21:59.9084156Z ++ rm -f /var/lib/jenkins/sccache_error.log
2025-08-14T21:21:59.9123169Z ++ trap_add sccache_epilogue EXIT
2025-08-14T21:21:59.9123824Z ++ trap_add_cmd=sccache_epilogue
2025-08-14T21:21:59.9124327Z ++ shift
2025-08-14T21:21:59.9124720Z ++ for trap_add_name in "$@"
2025-08-14T21:21:59.9143430Z ++++ trap -p EXIT
2025-08-14T21:21:59.9151403Z +++ eval 'extract_trap_cmd '
2025-08-14T21:21:59.9152107Z ++++ extract_trap_cmd
2025-08-14T21:21:59.9152580Z ++++ printf '%s\n' ''
2025-08-14T21:21:59.9153051Z +++ printf '%s\n' sccache_epilogue
2025-08-14T21:21:59.9160491Z ++ trap -- '
2025-08-14T21:21:59.9161007Z sccache_epilogue' EXIT
2025-08-14T21:21:59.9161460Z ++ [[ -n '' ]]
2025-08-14T21:21:59.9161945Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:21:59.9162639Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log
2025-08-14T21:21:59.9163304Z ++ SCCACHE_IDLE_TIMEOUT=0
2025-08-14T21:21:59.9163792Z ++ sccache --start-server
2025-08-14T21:21:59.9216844Z sccache: Starting the server...
2025-08-14T21:21:59.9928851Z sccache: Listening on address 127.0.0.1:4226
2025-08-14T21:21:59.9960958Z ++ sccache --zero-stats
2025-08-14T21:22:00.0015740Z Statistics zeroed.
2025-08-14T21:22:00.0022775Z ++ which ccache
2025-08-14T21:22:00.0053775Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-08-14T21:22:00.0054391Z + echo 'Environment variables:'
2025-08-14T21:22:00.0054922Z Environment variables:
2025-08-14T21:22:00.0055341Z + env
2025-08-14T21:22:00.0083612Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:22:00.0084591Z CONTINUE_THROUGH_ERROR=True
2025-08-14T21:22:00.0085156Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-08-14T21:22:00.0086004Z HOSTNAME=gpud535.jax.cs.cpe.ice.amd.com
2025-08-14T21:22:00.0087218Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0088995Z GITHUB_ACTION=__self
2025-08-14T21:22:00.0089463Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-08-14T21:22:00.0089995Z GITHUB_RUN_NUMBER=17921
2025-08-14T21:22:00.0090441Z TEST_CONFIG=distributed
2025-08-14T21:22:00.0090893Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-08-14T21:22:00.0091431Z AWS_DEFAULT_REGION=us-east-1
2025-08-14T21:22:00.0091940Z GITHUB_TRIGGERING_ACTOR=huydhn
2025-08-14T21:22:00.0092417Z GITHUB_REF_TYPE=branch
2025-08-14T21:22:00.0093151Z ***
2025-08-14T21:22:00.0093545Z GITHUB_REPOSITORY_ID=65600975
2025-08-14T21:22:00.0094044Z GITHUB_ACTIONS=true
2025-08-14T21:22:00.0094512Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0095158Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0096073Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main
2025-08-14T21:22:00.0096896Z UCC_HOME=/usr
2025-08-14T21:22:00.0097301Z VERBOSE_TEST_LOGS=False
2025-08-14T21:22:00.0097757Z GITHUB_REF=refs/heads/main
2025-08-14T21:22:00.0098205Z SHARD_NUMBER=3
2025-08-14T21:22:00.0098612Z GITHUB_REF_PROTECTED=true
2025-08-14T21:22:00.0099064Z HOME=/var/lib/jenkins
2025-08-14T21:22:00.0099551Z GITHUB_API_URL=https://api.github.com
2025-08-14T21:22:00.0100142Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-08-14T21:22:00.0100645Z LANG=C.UTF-8
2025-08-14T21:22:00.0101123Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-08-14T21:22:00.0101745Z PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:22:00.0102199Z NUM_TEST_SHARDS=3
2025-08-14T21:22:00.0102586Z UCX_HOME=/usr
2025-08-14T21:22:00.0103635Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0105242Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:22:00.0106584Z MAGMA_HOME=/opt/rocm/magma
2025-08-14T21:22:00.0107696Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0109201Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-08-14T21:22:00.0110090Z GITHUB_EVENT_NAME=schedule
2025-08-14T21:22:00.0110535Z DASHBOARD_TAG=
2025-08-14T21:22:00.0110953Z GITHUB_RUN_ID=16976370303
2025-08-14T21:22:00.0112091Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0113283Z GITHUB_ACTOR=huydhn
2025-08-14T21:22:00.0113681Z PR_NUMBER=
2025-08-14T21:22:00.0114060Z GITHUB_RUN_ATTEMPT=1
2025-08-14T21:22:00.0114511Z ANACONDA_PYTHON_VERSION=3.10
2025-08-14T21:22:00.0115096Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-08-14T21:22:00.0115743Z TERM=vt100
2025-08-14T21:22:00.0116128Z INSTALLED_VISION=yes
2025-08-14T21:22:00.0139867Z BRANCH=main
2025-08-14T21:22:00.0140338Z OPENSSL_ROOT_DIR=/opt/openssl
2025-08-14T21:22:00.0140882Z TESTS_TO_INCLUDE=
2025-08-14T21:22:00.0141842Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-08-14T21:22:00.0142936Z GITHUB_SERVER_URL=https://github.com
2025-08-14T21:22:00.0143532Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-08-14T21:22:00.0144101Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-08-14T21:22:00.0144690Z REENABLED_ISSUES=
2025-08-14T21:22:00.0145089Z SHLVL=1
2025-08-14T21:22:00.0145436Z MAX_JOBS=126
2025-08-14T21:22:00.0145828Z GITHUB_ACTOR_ID=475357
2025-08-14T21:22:00.0146413Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0147076Z GITHUB_REF_NAME=main
2025-08-14T21:22:00.0147509Z ROCM_PATH=/opt/rocm
2025-08-14T21:22:00.0147913Z GITHUB_JOB=test
2025-08-14T21:22:00.0148314Z NO_TEST_TIMEOUT=False
2025-08-14T21:22:00.0148770Z GITHUB_REPOSITORY=pytorch/pytorch
2025-08-14T21:22:00.0149282Z LC_ALL=C.UTF-8
2025-08-14T21:22:00.0149685Z GITHUB_RETENTION_DAYS=90
2025-08-14T21:22:00.0150606Z OPENSSL_DIR=/opt/openssl
2025-08-14T21:22:00.0151067Z GITHUB_ACTION_REPOSITORY=
2025-08-14T21:22:00.0152770Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:22:00.0154468Z GITHUB_BASE_REF=
2025-08-14T21:22:00.0154861Z CI=true
2025-08-14T21:22:00.0155250Z GITHUB_REPOSITORY_OWNER=pytorch
2025-08-14T21:22:00.0155742Z JOB_ID=48128182504
2025-08-14T21:22:00.0156142Z GITHUB_HEAD_REF=
2025-08-14T21:22:00.0156526Z GITHUB_ACTION_REF=
2025-08-14T21:22:00.0156937Z TEST_SHOWLOCALS=False
2025-08-14T21:22:00.0157472Z GITHUB_WORKFLOW=periodic
2025-08-14T21:22:00.0158054Z DEBIAN_FRONTEND=noninteractive
2025-08-14T21:22:00.0159454Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0160628Z NO_TD=False
2025-08-14T21:22:00.0161022Z OLDPWD=/var/lib/jenkins
2025-08-14T21:22:00.0161446Z _=/usr/bin/env
2025-08-14T21:22:00.0162015Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2025-08-14T21:22:00.0363379Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch
2025-08-14T21:22:00.0364589Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin
2025-08-14T21:22:00.0365752Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib
2025-08-14T21:22:00.0366797Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test
2025-08-14T21:22:00.0367574Z + BUILD_DIR=build
2025-08-14T21:22:00.0368036Z + BUILD_RENAMED_DIR=build_renamed
2025-08-14T21:22:00.0368558Z + BUILD_BIN_DIR=build/bin
2025-08-14T21:22:00.0368998Z + SHARD_NUMBER=3
2025-08-14T21:22:00.0369391Z + NUM_TEST_SHARDS=3
2025-08-14T21:22:00.0369843Z + export TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:22:00.0371072Z + TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:22:00.0371598Z + export VALGRIND=ON
2025-08-14T21:22:00.0372027Z + VALGRIND=ON
2025-08-14T21:22:00.0372455Z + [[ linux-jammy-rocm-py3.10 == *clang9* ]]
2025-08-14T21:22:00.0373078Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-08-14T21:22:00.0373659Z + [[ linux-jammy-rocm-py3.10 == *s390x* ]]
2025-08-14T21:22:00.0374183Z + [[ 0 == \1 ]]
2025-08-14T21:22:00.0374563Z + [[ True == \1 ]]
2025-08-14T21:22:00.0375000Z + [[ linux-jammy-rocm-py3.10 != *bazel* ]]
2025-08-14T21:22:00.0375586Z ++ realpath build/custom_test_artifacts
2025-08-14T21:22:00.0404062Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts
2025-08-14T21:22:00.0404962Z + [[ -n '' ]]
2025-08-14T21:22:00.0405393Z + echo 'Environment variables'
2025-08-14T21:22:00.0406191Z Environment variables
2025-08-14T21:22:00.0406706Z + env
2025-08-14T21:22:00.0427750Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T21:22:00.0428771Z CONTINUE_THROUGH_ERROR=True
2025-08-14T21:22:00.0429354Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10
2025-08-14T21:22:00.0429998Z HOSTNAME=gpud535.jax.cs.cpe.ice.amd.com
2025-08-14T21:22:00.0431194Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0432327Z GITHUB_ACTION=__self
2025-08-14T21:22:00.0432801Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2025-08-14T21:22:00.0433321Z GITHUB_RUN_NUMBER=17921
2025-08-14T21:22:00.0433769Z TEST_CONFIG=distributed
2025-08-14T21:22:00.0434238Z GITHUB_REPOSITORY_OWNER_ID=21003710
2025-08-14T21:22:00.0434790Z AWS_DEFAULT_REGION=us-east-1
2025-08-14T21:22:00.0435295Z GITHUB_TRIGGERING_ACTOR=huydhn
2025-08-14T21:22:00.0435775Z GITHUB_REF_TYPE=branch
2025-08-14T21:22:00.0436311Z ***
2025-08-14T21:22:00.0436704Z GITHUB_REPOSITORY_ID=65600975
2025-08-14T21:22:00.0437221Z GITHUB_ACTIONS=true
2025-08-14T21:22:00.0437685Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0438331Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0439237Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main
2025-08-14T21:22:00.0440579Z UCC_HOME=/usr
2025-08-14T21:22:00.0440989Z TORCH_SERIALIZATION_DEBUG=1
2025-08-14T21:22:00.0441459Z VERBOSE_TEST_LOGS=False
2025-08-14T21:22:00.0441904Z GITHUB_REF=refs/heads/main
2025-08-14T21:22:00.0442340Z SHARD_NUMBER=3
2025-08-14T21:22:00.0442738Z GITHUB_REF_PROTECTED=true
2025-08-14T21:22:00.0443181Z HOME=/var/lib/jenkins
2025-08-14T21:22:00.0443671Z GITHUB_API_URL=https://api.github.com
2025-08-14T21:22:00.0444253Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2025-08-14T21:22:00.0444758Z LANG=C.UTF-8
2025-08-14T21:22:00.0445231Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6
2025-08-14T21:22:00.0446268Z PYTORCH_TEST_WITH_ROCM=1
2025-08-14T21:22:00.0446737Z NUM_TEST_SHARDS=3
2025-08-14T21:22:00.0447127Z UCX_HOME=/usr
2025-08-14T21:22:00.0448196Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0449860Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T21:22:00.0450853Z MAGMA_HOME=/opt/rocm/magma
2025-08-14T21:22:00.0451923Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0453413Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json
2025-08-14T21:22:00.0454315Z GITHUB_EVENT_NAME=schedule
2025-08-14T21:22:00.0454753Z DASHBOARD_TAG=
2025-08-14T21:22:00.0455154Z GITHUB_RUN_ID=16976370303
2025-08-14T21:22:00.0456303Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0457508Z GITHUB_ACTOR=huydhn
2025-08-14T21:22:00.0457909Z PR_NUMBER=
2025-08-14T21:22:00.0458279Z GITHUB_RUN_ATTEMPT=1
2025-08-14T21:22:00.0459051Z VALGRIND=ON
2025-08-14T21:22:00.0459464Z ANACONDA_PYTHON_VERSION=3.10
2025-08-14T21:22:00.0460059Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2025-08-14T21:22:00.0460659Z TERM=vt100
2025-08-14T21:22:00.0461030Z INSTALLED_VISION=yes
2025-08-14T21:22:00.0461428Z BRANCH=main
2025-08-14T21:22:00.0461814Z OPENSSL_ROOT_DIR=/opt/openssl
2025-08-14T21:22:00.0462293Z TESTS_TO_INCLUDE=
2025-08-14T21:22:00.0463206Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm
2025-08-14T21:22:00.0464247Z GITHUB_SERVER_URL=https://github.com
2025-08-14T21:22:00.0464809Z PYTORCH_ROCM_ARCH=gfx90a;gfx942
2025-08-14T21:22:00.0465373Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d
2025-08-14T21:22:00.0465948Z REENABLED_ISSUES=
2025-08-14T21:22:00.0466329Z SHLVL=1
2025-08-14T21:22:00.0466670Z MAX_JOBS=126
2025-08-14T21:22:00.0467048Z GITHUB_ACTOR_ID=475357
2025-08-14T21:22:00.0467620Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T21:22:00.0468277Z GITHUB_REF_NAME=main
2025-08-14T21:22:00.0468696Z ROCM_PATH=/opt/rocm
2025-08-14T21:22:00.0469100Z GITHUB_JOB=test
2025-08-14T21:22:00.0469489Z NO_TEST_TIMEOUT=False
2025-08-14T21:22:00.0469948Z GITHUB_REPOSITORY=pytorch/pytorch
2025-08-14T21:22:00.0470448Z LC_ALL=C.UTF-8
2025-08-14T21:22:00.0470843Z GITHUB_RETENTION_DAYS=90
2025-08-14T21:22:00.0471297Z OPENSSL_DIR=/opt/openssl
2025-08-14T21:22:00.0471740Z GITHUB_ACTION_REPOSITORY=
2025-08-14T21:22:00.0473395Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:22:00.0475058Z GITHUB_BASE_REF=
2025-08-14T21:22:00.0475439Z CI=true
2025-08-14T21:22:00.0475819Z GITHUB_REPOSITORY_OWNER=pytorch
2025-08-14T21:22:00.0476298Z JOB_ID=48128182504
2025-08-14T21:22:00.0476684Z GITHUB_HEAD_REF=
2025-08-14T21:22:00.0477067Z GITHUB_ACTION_REF=
2025-08-14T21:22:00.0477468Z TEST_SHOWLOCALS=False
2025-08-14T21:22:00.0477911Z GITHUB_WORKFLOW=periodic
2025-08-14T21:22:00.0478369Z DEBIAN_FRONTEND=noninteractive
2025-08-14T21:22:00.0479847Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_5ea68eb3-9af6-4d72-96f0-10eebc01434c
2025-08-14T21:22:00.0480975Z NO_TD=False
2025-08-14T21:22:00.0481345Z OLDPWD=/var/lib/jenkins
2025-08-14T21:22:00.0481757Z _=/usr/bin/env
2025-08-14T21:22:00.0482182Z + echo 'Testing pytorch'
2025-08-14T21:22:00.0482605Z Testing pytorch
2025-08-14T21:22:00.0482999Z + export LANG=C.UTF-8
2025-08-14T21:22:00.0483403Z + LANG=C.UTF-8
2025-08-14T21:22:00.0483767Z + PR_NUMBER=
2025-08-14T21:22:00.0484168Z + [[ distributed == \d\e\f\a\u\l\t ]]
2025-08-14T21:22:00.0484727Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2025-08-14T21:22:00.0485319Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:22:00.0486003Z + export HIP_VISIBLE_DEVICES=0,1,2,3
2025-08-14T21:22:00.0486542Z + HIP_VISIBLE_DEVICES=0,1,2,3
2025-08-14T21:22:00.0487056Z + [[ distributed == \s\l\o\w ]]
2025-08-14T21:22:00.0487622Z + [[ linux-jammy-rocm-py3.10 == *slow-gradcheck* ]]
2025-08-14T21:22:00.0488272Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-08-14T21:22:00.0488838Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:22:00.0489422Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-08-14T21:22:00.0490018Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2025-08-14T21:22:00.0490559Z + [[ distributed == *crossref* ]]
2025-08-14T21:22:00.0491081Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T21:22:00.0491599Z + export VALGRIND=OFF
2025-08-14T21:22:00.0491996Z + VALGRIND=OFF
2025-08-14T21:22:00.0492361Z + rocminfo
2025-08-14T21:22:00.0613766Z ROCk module version 6.10.5 is loaded
2025-08-14T21:22:00.2798978Z =====================    
2025-08-14T21:22:00.2799703Z HSA System Attributes    
2025-08-14T21:22:00.2800204Z =====================    
2025-08-14T21:22:00.2800741Z Runtime Version:         1.15
2025-08-14T21:22:00.2801239Z Runtime Ext Version:     1.7
2025-08-14T21:22:00.2802398Z System Timestamp Freq.:  1000.000000MHz
2025-08-14T21:22:00.2803260Z Sig. Max Wait Duration:  18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count)
2025-08-14T21:22:00.2804183Z Machine Model:           LARGE                              
2025-08-14T21:22:00.2804904Z System Endianness:       LITTLE                             
2025-08-14T21:22:00.2805764Z Mwaitx:                  DISABLED
2025-08-14T21:22:00.2806301Z XNACK enabled:           NO
2025-08-14T21:22:00.2806900Z DMAbuf Support:          YES
2025-08-14T21:22:00.2807394Z VMM Support:             YES
2025-08-14T21:22:00.2807698Z 
2025-08-14T21:22:00.2807863Z ==========               
2025-08-14T21:22:00.2808306Z HSA Agents               
2025-08-14T21:22:00.2808719Z ==========               
2025-08-14T21:22:00.2809117Z *******                  
2025-08-14T21:22:00.2809572Z Agent 1                  
2025-08-14T21:22:00.2809980Z *******                  
2025-08-14T21:22:00.2810513Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.2811186Z   Uuid:                    CPU-XX                             
2025-08-14T21:22:00.2811889Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.2812600Z   Vendor Name:             CPU                                
2025-08-14T21:22:00.2813257Z   Feature:                 None specified                     
2025-08-14T21:22:00.2813922Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:22:00.2814604Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.2815328Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:22:00.2816010Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:22:00.2816669Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:22:00.2817319Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.2817936Z   Node:                    0                                  
2025-08-14T21:22:00.2818575Z   Device Type:             CPU                                
2025-08-14T21:22:00.2819699Z   Cache Info:              
2025-08-14T21:22:00.2820200Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:22:00.2820810Z   Chip ID:                 0(0x0)                             
2025-08-14T21:22:00.2821489Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:22:00.2822163Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:22:00.2822851Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:22:00.2823476Z   BDFID:                   0                                  
2025-08-14T21:22:00.2824116Z   Internal Node ID:        0                                  
2025-08-14T21:22:00.2824853Z   Compute Unit:            64                                 
2025-08-14T21:22:00.2825528Z   SIMDs per CU:            0                                  
2025-08-14T21:22:00.2826216Z   Shader Engines:          0                                  
2025-08-14T21:22:00.2826918Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:22:00.2827675Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:22:00.2828357Z   Memory Properties:       
2025-08-14T21:22:00.2828848Z   Features:                None
2025-08-14T21:22:00.2829320Z   Pool Info:               
2025-08-14T21:22:00.2829766Z     Pool 1                   
2025-08-14T21:22:00.2830342Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.2831037Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:22:00.2831713Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2832407Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2833151Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2834218Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2834929Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2835534Z     Pool 2                   
2025-08-14T21:22:00.2836090Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.2836747Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:22:00.2837386Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2838060Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2838762Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2839485Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2840189Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2840799Z     Pool 3                   
2025-08-14T21:22:00.2841389Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:22:00.2842086Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:22:00.2842767Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2843459Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2844182Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2844915Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2845740Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2846360Z     Pool 4                   
2025-08-14T21:22:00.2846918Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.2847561Z       Size:                    528196732(0x1f7ba47c) KB           
2025-08-14T21:22:00.2848203Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2848881Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2849601Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2850671Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2851362Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2851963Z   ISA Info:                
2025-08-14T21:22:00.2852400Z *******                  
2025-08-14T21:22:00.2852812Z Agent 2                  
2025-08-14T21:22:00.2853213Z *******                  
2025-08-14T21:22:00.2853717Z   Name:                    AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.2854358Z   Uuid:                    CPU-XX                             
2025-08-14T21:22:00.2855044Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.2855740Z   Vendor Name:             CPU                                
2025-08-14T21:22:00.2856403Z   Feature:                 None specified                     
2025-08-14T21:22:00.2857074Z   Profile:                 FULL_PROFILE                       
2025-08-14T21:22:00.2857748Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.2858446Z   Max Queue Number:        0(0x0)                             
2025-08-14T21:22:00.2859130Z   Queue Min Size:          0(0x0)                             
2025-08-14T21:22:00.2859812Z   Queue Max Size:          0(0x0)                             
2025-08-14T21:22:00.2860467Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.2861083Z   Node:                    1                                  
2025-08-14T21:22:00.2861702Z   Device Type:             CPU                                
2025-08-14T21:22:00.2862284Z   Cache Info:              
2025-08-14T21:22:00.2862768Z     L1:                      32768(0x8000) KB                   
2025-08-14T21:22:00.2863376Z   Chip ID:                 0(0x0)                             
2025-08-14T21:22:00.2864248Z   ASIC Revision:           0(0x0)                             
2025-08-14T21:22:00.2865250Z   Cacheline Size:          64(0x40)                           
2025-08-14T21:22:00.2865942Z   Max Clock Freq. (MHz):   2000                               
2025-08-14T21:22:00.2866584Z   BDFID:                   0                                  
2025-08-14T21:22:00.2867222Z   Internal Node ID:        1                                  
2025-08-14T21:22:00.2867887Z   Compute Unit:            64                                 
2025-08-14T21:22:00.2868530Z   SIMDs per CU:            0                                  
2025-08-14T21:22:00.2869182Z   Shader Engines:          0                                  
2025-08-14T21:22:00.2869859Z   Shader Arrs. per Eng.:   0                                  
2025-08-14T21:22:00.2870562Z   WatchPts on Addr. Ranges:1                                  
2025-08-14T21:22:00.2871184Z   Memory Properties:       
2025-08-14T21:22:00.2871658Z   Features:                None
2025-08-14T21:22:00.2872111Z   Pool Info:               
2025-08-14T21:22:00.2872548Z     Pool 1                   
2025-08-14T21:22:00.2873124Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.2873800Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:22:00.2874463Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2875150Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2875873Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2876606Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2877308Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2877911Z     Pool 2                   
2025-08-14T21:22:00.2878453Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.2879120Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:22:00.2879766Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2880449Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2881441Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2882158Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2882848Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2883447Z     Pool 3                   
2025-08-14T21:22:00.2883988Z       Segment:                 GLOBAL; FLAGS: KERNARG, FINE GRAINED
2025-08-14T21:22:00.2884630Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:22:00.2885272Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2886288Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2887016Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2887740Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2888441Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2889046Z     Pool 4                   
2025-08-14T21:22:00.2889584Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.2890224Z       Size:                    528455480(0x1f7f9738) KB           
2025-08-14T21:22:00.2890861Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2891555Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2892263Z       Alloc Recommended Granule:4KB                                
2025-08-14T21:22:00.2892974Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2893674Z       Accessible by all:       TRUE                               
2025-08-14T21:22:00.2894274Z   ISA Info:                
2025-08-14T21:22:00.2894701Z *******                  
2025-08-14T21:22:00.2895117Z Agent 3                  
2025-08-14T21:22:00.2895858Z *******                  
2025-08-14T21:22:00.2896344Z   Name:                    gfx90a                             
2025-08-14T21:22:00.2896976Z   Uuid:                    GPU-d8bf612a1b3d9d46               
2025-08-14T21:22:00.2897663Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.2898358Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.2899012Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.2899667Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.2900347Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.2901039Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.2901706Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.2902361Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.2903035Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.2903664Z   Node:                    2                                  
2025-08-14T21:22:00.2904301Z   Device Type:             GPU                                
2025-08-14T21:22:00.2904893Z   Cache Info:              
2025-08-14T21:22:00.2905380Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.2905954Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.2906547Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.2907197Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.2907880Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.2908556Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.2909198Z   BDFID:                   12800                              
2025-08-14T21:22:00.2909853Z   Internal Node ID:        2                                  
2025-08-14T21:22:00.2910546Z   Compute Unit:            104                                
2025-08-14T21:22:00.2911211Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.2912270Z   Shader Engines:          8                                  
2025-08-14T21:22:00.2912968Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.2913680Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.2914405Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.2915037Z   Memory Properties:       
2025-08-14T21:22:00.2915530Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.2916166Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.2916871Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.2917567Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.2918209Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.2918750Z     x                        1024(0x400)                        
2025-08-14T21:22:00.2919312Z     y                        1024(0x400)                        
2025-08-14T21:22:00.2919870Z     z                        1024(0x400)                        
2025-08-14T21:22:00.2920483Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.2921176Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.2921860Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.2922470Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.2922967Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2923539Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2924124Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2924803Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.2941133Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.2941954Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.2942685Z   IOMMU Support::          None                               
2025-08-14T21:22:00.2943303Z   Pool Info:               
2025-08-14T21:22:00.2943780Z     Pool 1                   
2025-08-14T21:22:00.2944360Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.2945052Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.2945729Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2946438Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2947176Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.2947923Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2948639Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.2949272Z     Pool 2                   
2025-08-14T21:22:00.2949848Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.2950516Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.2951172Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2951871Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2952597Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.2953333Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2954048Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.2954662Z     Pool 3                   
2025-08-14T21:22:00.2955210Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.2955865Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.2956532Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.2957218Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.2958285Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.2959015Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.2959716Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.2960314Z     Pool 4                   
2025-08-14T21:22:00.2960831Z       Segment:                 GROUP                              
2025-08-14T21:22:00.2961458Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.2962101Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.2962789Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.2963506Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.2964237Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.2964945Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.2965689Z   ISA Info:                
2025-08-14T21:22:00.2966146Z     ISA 1                    
2025-08-14T21:22:00.2966735Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.2967494Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.2968215Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.2968926Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.2969664Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.2970352Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.2971038Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.2971695Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.2972595Z         x                        1024(0x400)                        
2025-08-14T21:22:00.2973187Z         y                        1024(0x400)                        
2025-08-14T21:22:00.2973769Z         z                        1024(0x400)                        
2025-08-14T21:22:00.2974387Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.2975005Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.2975525Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2976108Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2976679Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.2977323Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.2977918Z *******                  
2025-08-14T21:22:00.2978341Z Agent 4                  
2025-08-14T21:22:00.2978758Z *******                  
2025-08-14T21:22:00.2979251Z   Name:                    gfx90a                             
2025-08-14T21:22:00.2979896Z   Uuid:                    GPU-ecc83b03e3da2c15               
2025-08-14T21:22:00.2980600Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.2981304Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.2981981Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.2982721Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.2983517Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.2984348Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.2985109Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.2985775Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.2986429Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.2987071Z   Node:                    3                                  
2025-08-14T21:22:00.2987703Z   Device Type:             GPU                                
2025-08-14T21:22:00.2988631Z   Cache Info:              
2025-08-14T21:22:00.2989121Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.2989702Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.2990293Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.2990935Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.2991617Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.2992295Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.2992932Z   BDFID:                   13568                              
2025-08-14T21:22:00.2993581Z   Internal Node ID:        3                                  
2025-08-14T21:22:00.2994255Z   Compute Unit:            104                                
2025-08-14T21:22:00.2994930Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.2995739Z   Shader Engines:          8                                  
2025-08-14T21:22:00.2996560Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.2997306Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.2998028Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.2998662Z   Memory Properties:       
2025-08-14T21:22:00.2999163Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.2999815Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3000541Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3001264Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3001946Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3002497Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3003344Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3003911Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3004538Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3005240Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3006238Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3006861Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3007374Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3007954Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3008522Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3009181Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3009938Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3010667Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3011359Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3011973Z   Pool Info:               
2025-08-14T21:22:00.3012434Z     Pool 1                   
2025-08-14T21:22:00.3012995Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3013680Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3014350Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3015044Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3015775Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3016510Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3017222Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3017825Z     Pool 2                   
2025-08-14T21:22:00.3018394Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3019066Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3020078Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3020767Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3021481Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3022204Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3022902Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3023506Z     Pool 3                   
2025-08-14T21:22:00.3024054Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3024701Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3025357Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3026049Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3026773Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3027512Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3028216Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3028818Z     Pool 4                   
2025-08-14T21:22:00.3029336Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3029964Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3030613Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3031298Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3032009Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3032801Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3034087Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3034824Z   ISA Info:                
2025-08-14T21:22:00.3035323Z     ISA 1                    
2025-08-14T21:22:00.3035892Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3036646Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3037360Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3038077Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3038808Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3039485Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3040157Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3040806Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3041358Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3041939Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3042514Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3043141Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3043761Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3044278Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3044852Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3045423Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3046192Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3046793Z *******                  
2025-08-14T21:22:00.3047263Z Agent 5                  
2025-08-14T21:22:00.3047761Z *******                  
2025-08-14T21:22:00.3048323Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3049067Z   Uuid:                    GPU-17d29e8be4a2b908               
2025-08-14T21:22:00.3049758Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3050794Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3051479Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3052155Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3052827Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3053509Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3054174Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3054838Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3055513Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3056136Z   Node:                    4                                  
2025-08-14T21:22:00.3056769Z   Device Type:             GPU                                
2025-08-14T21:22:00.3057368Z   Cache Info:              
2025-08-14T21:22:00.3057855Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3058427Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3059018Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3059657Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3060340Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3061032Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3061670Z   BDFID:                   4352                               
2025-08-14T21:22:00.3062317Z   Internal Node ID:        4                                  
2025-08-14T21:22:00.3063086Z   Compute Unit:            104                                
2025-08-14T21:22:00.3064250Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3065046Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3065754Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3066465Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3067179Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3067796Z   Memory Properties:       
2025-08-14T21:22:00.3068294Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3068923Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3069620Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3070318Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3070957Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3071494Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3072065Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3072617Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3073246Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3073932Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3074614Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3075215Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3075699Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3076269Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3076831Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3077485Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3078229Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3078944Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3079643Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3080242Z   Pool Info:               
2025-08-14T21:22:00.3080995Z     Pool 1                   
2025-08-14T21:22:00.3081558Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3082233Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3082900Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3083593Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3084312Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3085046Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3086074Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3086679Z     Pool 2                   
2025-08-14T21:22:00.3087233Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3087904Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3088564Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3089252Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3089972Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3090699Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3091405Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3092007Z     Pool 3                   
2025-08-14T21:22:00.3092553Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3093210Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3093784Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3094393Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3095331Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3095995Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3096618Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3097153Z     Pool 4                   
2025-08-14T21:22:00.3097618Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3098171Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3098738Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3099349Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3099989Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3100635Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3101253Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3101806Z   ISA Info:                
2025-08-14T21:22:00.3102189Z     ISA 1                    
2025-08-14T21:22:00.3102702Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3103364Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3104000Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3104631Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3105279Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3105878Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3106472Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3107045Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3107540Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3108052Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3108552Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3109395Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3109938Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3110387Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3110892Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3111397Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3111960Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3112486Z *******                  
2025-08-14T21:22:00.3112857Z Agent 6                  
2025-08-14T21:22:00.3113224Z *******                  
2025-08-14T21:22:00.3113645Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3114206Z   Uuid:                    GPU-4fe0daed09e32bd4               
2025-08-14T21:22:00.3114823Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3115448Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3116039Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3116625Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3117225Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3117839Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3118431Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3119014Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3119592Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3120140Z   Node:                    5                                  
2025-08-14T21:22:00.3120693Z   Device Type:             GPU                                
2025-08-14T21:22:00.3121460Z   Cache Info:              
2025-08-14T21:22:00.3121897Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3122416Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3122935Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3123506Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3124114Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3124714Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3125277Z   BDFID:                   5120                               
2025-08-14T21:22:00.3125964Z   Internal Node ID:        5                                  
2025-08-14T21:22:00.3126563Z   Compute Unit:            104                                
2025-08-14T21:22:00.3127143Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3127730Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3128340Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3128979Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3129617Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3130170Z   Memory Properties:       
2025-08-14T21:22:00.3130608Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3131163Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3131789Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3132405Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3132965Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3133437Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3133926Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3134420Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3134967Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3135878Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3136480Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3137013Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3137454Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3137954Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3138451Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3139030Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3139699Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3140327Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3140940Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3141479Z   Pool Info:               
2025-08-14T21:22:00.3141877Z     Pool 1                   
2025-08-14T21:22:00.3142390Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3164938Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3165912Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3166612Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3167287Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3167945Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3168576Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3169131Z     Pool 2                   
2025-08-14T21:22:00.3169655Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3170718Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3171358Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3172000Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3172654Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3173314Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3173945Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3174496Z     Pool 3                   
2025-08-14T21:22:00.3174999Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3175573Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3176134Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3176739Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3177381Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3178034Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3178653Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3179188Z     Pool 4                   
2025-08-14T21:22:00.3179658Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3180210Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3180777Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3181391Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3182027Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3182666Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3183288Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3183837Z   ISA Info:                
2025-08-14T21:22:00.3184241Z     ISA 1                    
2025-08-14T21:22:00.3184755Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3185721Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3186360Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3186992Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3187638Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3188237Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3188827Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3189398Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3189913Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3190430Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3190926Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3191480Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3192032Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3192498Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3192994Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3193487Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3194048Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3194580Z *******                  
2025-08-14T21:22:00.3194978Z Agent 7                  
2025-08-14T21:22:00.3195344Z *******                  
2025-08-14T21:22:00.3195771Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3196341Z   Uuid:                    GPU-acdc18a5a79ca32e               
2025-08-14T21:22:00.3197200Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3197825Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3198425Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3199015Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3199605Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3200205Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3200787Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3201365Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3201940Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3202482Z   Node:                    6                                  
2025-08-14T21:22:00.3203023Z   Device Type:             GPU                                
2025-08-14T21:22:00.3203540Z   Cache Info:              
2025-08-14T21:22:00.3203979Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3204492Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3205012Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3205698Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3206308Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3206907Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3207474Z   BDFID:                   44544                              
2025-08-14T21:22:00.3208039Z   Internal Node ID:        6                                  
2025-08-14T21:22:00.3208624Z   Compute Unit:            104                                
2025-08-14T21:22:00.3209195Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3209776Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3210397Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3211029Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3211968Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3212535Z   Memory Properties:       
2025-08-14T21:22:00.3212983Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3213548Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3214166Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3214778Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3215336Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3215814Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3216306Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3216784Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3217332Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3217948Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3218565Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3219109Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3219544Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3220049Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3220544Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3221117Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3221782Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3222410Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3223015Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3223545Z   Pool Info:               
2025-08-14T21:22:00.3224237Z     Pool 1                   
2025-08-14T21:22:00.3224747Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3225364Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3225942Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3226544Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3227175Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3227820Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3228437Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3228963Z     Pool 2                   
2025-08-14T21:22:00.3229448Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3230025Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3230607Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3231206Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3231842Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3232473Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3233086Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3233616Z     Pool 3                   
2025-08-14T21:22:00.3234089Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3234657Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3235217Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3235816Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3236446Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3237075Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3237691Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3238475Z     Pool 4                   
2025-08-14T21:22:00.3238932Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3239475Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3240033Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3240630Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3241253Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3241882Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3242524Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3243067Z   ISA Info:                
2025-08-14T21:22:00.3243460Z     ISA 1                    
2025-08-14T21:22:00.3243980Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3244636Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3245279Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3245996Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3246628Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3247219Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3247837Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3248422Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3248933Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3249436Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3249919Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3250758Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3251327Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3251796Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3252299Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3252794Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3253349Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3253877Z *******                  
2025-08-14T21:22:00.3254256Z Agent 8                  
2025-08-14T21:22:00.3254621Z *******                  
2025-08-14T21:22:00.3255044Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3255609Z   Uuid:                    GPU-a46db4ad480ce7bc               
2025-08-14T21:22:00.3256219Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3256837Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3257429Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3258029Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3258630Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3259233Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3259821Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3260399Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3261003Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3261559Z   Node:                    7                                  
2025-08-14T21:22:00.3262111Z   Device Type:             GPU                                
2025-08-14T21:22:00.3262632Z   Cache Info:              
2025-08-14T21:22:00.3263071Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3263589Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3264450Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3265016Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3265606Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3266214Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3266777Z   BDFID:                   45824                              
2025-08-14T21:22:00.3267338Z   Internal Node ID:        7                                  
2025-08-14T21:22:00.3267931Z   Compute Unit:            104                                
2025-08-14T21:22:00.3268502Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3269085Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3269685Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3270312Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3270944Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3271506Z   Memory Properties:       
2025-08-14T21:22:00.3271946Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3272511Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3273115Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3273723Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3274280Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3274758Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3275247Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3275725Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3276254Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3277102Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3277712Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3278258Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3278696Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3279197Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3279690Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3280263Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3280913Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3281545Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3282157Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3282684Z   Pool Info:               
2025-08-14T21:22:00.3283083Z     Pool 1                   
2025-08-14T21:22:00.3283592Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3284195Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3284776Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3285385Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3286270Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3286921Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3287561Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3288122Z     Pool 2                   
2025-08-14T21:22:00.3288622Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3289214Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3289790Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3290405Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3291050Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3292143Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3292760Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3293303Z     Pool 3                   
2025-08-14T21:22:00.3293791Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3294367Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3294936Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3295549Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3296192Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3296835Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3297464Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3298004Z     Pool 4                   
2025-08-14T21:22:00.3298472Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3299025Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3299595Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3300204Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3300830Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3301466Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3302089Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3302634Z   ISA Info:                
2025-08-14T21:22:00.3303022Z     ISA 1                    
2025-08-14T21:22:00.3303529Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3304485Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3305144Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3305772Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3306421Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3307021Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3307623Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3308195Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3308690Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3309200Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3309708Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3310262Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3310817Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3311283Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3311802Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3312311Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3312877Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3313407Z *******                  
2025-08-14T21:22:00.3313784Z Agent 9                  
2025-08-14T21:22:00.3314153Z *******                  
2025-08-14T21:22:00.3314587Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3315135Z   Uuid:                    GPU-6250e8145e0cb8ff               
2025-08-14T21:22:00.3315743Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3316360Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3316957Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3317551Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3318405Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3319017Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3319610Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3320199Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3320781Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3321339Z   Node:                    8                                  
2025-08-14T21:22:00.3321895Z   Device Type:             GPU                                
2025-08-14T21:22:00.3322417Z   Cache Info:              
2025-08-14T21:22:00.3322852Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3323353Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3323886Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3324463Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3325062Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3325757Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3326333Z   BDFID:                   36352                              
2025-08-14T21:22:00.3326900Z   Internal Node ID:        8                                  
2025-08-14T21:22:00.3327490Z   Compute Unit:            104                                
2025-08-14T21:22:00.3328070Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3328653Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3329261Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3329885Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3330797Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3331367Z   Memory Properties:       
2025-08-14T21:22:00.3331811Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3332374Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3332983Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3333599Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3334164Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3334636Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3335129Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3335616Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3336159Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3336769Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3337386Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3337936Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3338371Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3338879Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3339380Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3339956Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3340608Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3341241Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3341855Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3342396Z   Pool Info:               
2025-08-14T21:22:00.3342798Z     Pool 1                   
2025-08-14T21:22:00.3343299Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3343900Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3344780Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3345398Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3346038Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3346682Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3347314Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3347859Z     Pool 2                   
2025-08-14T21:22:00.3348346Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3348934Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3349509Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3350118Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3350760Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3351408Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3352028Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3352558Z     Pool 3                   
2025-08-14T21:22:00.3353037Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3353606Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3354177Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3354777Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3355402Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3356048Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3356671Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3357446Z     Pool 4                   
2025-08-14T21:22:00.3357919Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3358481Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3359048Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3359656Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3360287Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3360928Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3361556Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3362098Z   ISA Info:                
2025-08-14T21:22:00.3362478Z     ISA 1                    
2025-08-14T21:22:00.3362982Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3363644Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3364289Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3364927Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3365670Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3366290Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3366880Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3367450Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3367943Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3368446Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3368939Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3369479Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3370020Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3370480Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3371327Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3371823Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3372381Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3372900Z *******                  
2025-08-14T21:22:00.3373273Z Agent 10                 
2025-08-14T21:22:00.3373637Z *******                  
2025-08-14T21:22:00.3374060Z   Name:                    gfx90a                             
2025-08-14T21:22:00.3374615Z   Uuid:                    GPU-bef9ac6669b03ca0               
2025-08-14T21:22:00.3375220Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.3375824Z   Vendor Name:             AMD                                
2025-08-14T21:22:00.3376416Z   Feature:                 KERNEL_DISPATCH                    
2025-08-14T21:22:00.3377008Z   Profile:                 BASE_PROFILE                       
2025-08-14T21:22:00.3377606Z   Float Round Mode:        NEAR                               
2025-08-14T21:22:00.3378218Z   Max Queue Number:        128(0x80)                          
2025-08-14T21:22:00.3378803Z   Queue Min Size:          64(0x40)                           
2025-08-14T21:22:00.3379382Z   Queue Max Size:          131072(0x20000)                    
2025-08-14T21:22:00.3379960Z   Queue Type:              MULTI                              
2025-08-14T21:22:00.3380502Z   Node:                    9                                  
2025-08-14T21:22:00.3381048Z   Device Type:             GPU                                
2025-08-14T21:22:00.3381561Z   Cache Info:              
2025-08-14T21:22:00.3381987Z     L1:                      16(0x10) KB                        
2025-08-14T21:22:00.3382494Z     L2:                      8192(0x2000) KB                    
2025-08-14T21:22:00.3383293Z   Chip ID:                 29708(0x740c)                      
2025-08-14T21:22:00.3383885Z   ASIC Revision:           1(0x1)                             
2025-08-14T21:22:00.3384489Z   Cacheline Size:          128(0x80)                          
2025-08-14T21:22:00.3385083Z   Max Clock Freq. (MHz):   1700                               
2025-08-14T21:22:00.3385646Z   BDFID:                   37632                              
2025-08-14T21:22:00.3386210Z   Internal Node ID:        9                                  
2025-08-14T21:22:00.3386797Z   Compute Unit:            104                                
2025-08-14T21:22:00.3387378Z   SIMDs per CU:            4                                  
2025-08-14T21:22:00.3387955Z   Shader Engines:          8                                  
2025-08-14T21:22:00.3388560Z   Shader Arrs. per Eng.:   1                                  
2025-08-14T21:22:00.3389182Z   WatchPts on Addr. Ranges:4                                  
2025-08-14T21:22:00.3389813Z   Coherent Host Access:    FALSE                              
2025-08-14T21:22:00.3390382Z   Memory Properties:       
2025-08-14T21:22:00.3390817Z   Features:                KERNEL_DISPATCH 
2025-08-14T21:22:00.3391380Z   Fast F16 Operation:      TRUE                               
2025-08-14T21:22:00.3391992Z   Wavefront Size:          64(0x40)                           
2025-08-14T21:22:00.3392606Z   Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3393159Z   Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3393629Z     x                        1024(0x400)                        
2025-08-14T21:22:00.3394112Z     y                        1024(0x400)                        
2025-08-14T21:22:00.3394593Z     z                        1024(0x400)                        
2025-08-14T21:22:00.3395126Z   Max Waves Per CU:        32(0x20)                           
2025-08-14T21:22:00.3395723Z   Max Work-item Per CU:    2048(0x800)                        
2025-08-14T21:22:00.3396334Z   Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3396871Z   Grid Max Size per Dimension:
2025-08-14T21:22:00.3397554Z     x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3398058Z     y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3398550Z     z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3399124Z   Max fbarriers/Workgrp:   32                                 
2025-08-14T21:22:00.3399781Z   Packet Processor uCode:: 92                                 
2025-08-14T21:22:00.3400415Z   SDMA engine uCode::      9                                  
2025-08-14T21:22:00.3401031Z   IOMMU Support::          None                               
2025-08-14T21:22:00.3401568Z   Pool Info:               
2025-08-14T21:22:00.3401969Z     Pool 1                   
2025-08-14T21:22:00.3402471Z       Segment:                 GLOBAL; FLAGS: COARSE GRAINED      
2025-08-14T21:22:00.3403069Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3403657Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3404270Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3404912Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3405825Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3406456Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3406998Z     Pool 2                   
2025-08-14T21:22:00.3407493Z       Segment:                 GLOBAL; FLAGS: EXTENDED FINE GRAINED
2025-08-14T21:22:00.3408076Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3408635Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3409236Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3409868Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3410801Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3411445Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3411978Z     Pool 3                   
2025-08-14T21:22:00.3412459Z       Segment:                 GLOBAL; FLAGS: FINE GRAINED        
2025-08-14T21:22:00.3413027Z       Size:                    67092480(0x3ffc000) KB             
2025-08-14T21:22:00.3413594Z       Allocatable:             TRUE                               
2025-08-14T21:22:00.3414196Z       Alloc Granule:           4KB                                
2025-08-14T21:22:00.3414502Z       Alloc Recommended Granule:2048KB                             
2025-08-14T21:22:00.3414793Z       Alloc Alignment:         4KB                                
2025-08-14T21:22:00.3415077Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3415322Z     Pool 4                   
2025-08-14T21:22:00.3415539Z       Segment:                 GROUP                              
2025-08-14T21:22:00.3415796Z       Size:                    64(0x40) KB                        
2025-08-14T21:22:00.3416062Z       Allocatable:             FALSE                              
2025-08-14T21:22:00.3416342Z       Alloc Granule:           0KB                                
2025-08-14T21:22:00.3416638Z       Alloc Recommended Granule:0KB                                
2025-08-14T21:22:00.3416937Z       Alloc Alignment:         0KB                                
2025-08-14T21:22:00.3417225Z       Accessible by all:       FALSE                              
2025-08-14T21:22:00.3417475Z   ISA Info:                
2025-08-14T21:22:00.3417653Z     ISA 1                    
2025-08-14T21:22:00.3417888Z       Name:                    amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack-
2025-08-14T21:22:00.3418191Z       Machine Models:          HSA_MACHINE_MODEL_LARGE            
2025-08-14T21:22:00.3418486Z       Profiles:                HSA_PROFILE_BASE                   
2025-08-14T21:22:00.3418781Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3419217Z       Default Rounding Mode:   NEAR                               
2025-08-14T21:22:00.3419493Z       Fast f16:                TRUE                               
2025-08-14T21:22:00.3419767Z       Workgroup Max Size:      1024(0x400)                        
2025-08-14T21:22:00.3420029Z       Workgroup Max Size per Dimension:
2025-08-14T21:22:00.3420261Z         x                        1024(0x400)                        
2025-08-14T21:22:00.3420497Z         y                        1024(0x400)                        
2025-08-14T21:22:00.3420725Z         z                        1024(0x400)                        
2025-08-14T21:22:00.3420978Z       Grid Max Size:           4294967295(0xffffffff)             
2025-08-14T21:22:00.3421230Z       Grid Max Size per Dimension:
2025-08-14T21:22:00.3421445Z         x                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3421685Z         y                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3421924Z         z                        4294967295(0xffffffff)             
2025-08-14T21:22:00.3422184Z       FBarrier Max Size:       32                                 
2025-08-14T21:22:00.3422431Z *** Done ***             
2025-08-14T21:22:00.3422608Z + rocminfo
2025-08-14T21:22:00.3422774Z + grep -E 'Name:.*\sgfx|Marketing'
2025-08-14T21:22:00.5209028Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.5209972Z   Marketing Name:          AMD EPYC 7713 64-Core Processor    
2025-08-14T21:22:00.5210710Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5211418Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5212136Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5212817Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5214032Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5214754Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5215423Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5216092Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5216760Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5217430Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5218150Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5218814Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5219515Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5220205Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5220872Z   Name:                    gfx90a                             
2025-08-14T21:22:00.5221578Z   Marketing Name:          AMD Instinct MI250X/MI250          
2025-08-14T21:22:00.5392934Z + MAYBE_ROCM=rocm/
2025-08-14T21:22:00.5393566Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]]
2025-08-14T21:22:00.5394271Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-08-14T21:22:00.5394913Z + pip_install ninja==1.10.2
2025-08-14T21:22:00.5395646Z + pip_install_pkg='python3 -m pip install --progress-bar off'
2025-08-14T21:22:00.5396502Z + python3 -m pip install --progress-bar off ninja==1.10.2
2025-08-14T21:22:01.0372007Z Collecting ninja==1.10.2
2025-08-14T21:22:01.0662771Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB)
2025-08-14T21:22:01.0751108Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2025-08-14T21:22:01.3809190Z Installing collected packages: ninja
2025-08-14T21:22:01.3809890Z   Attempting uninstall: ninja
2025-08-14T21:22:01.3813013Z     Found existing installation: ninja 1.11.1.3
2025-08-14T21:22:01.3832328Z     Uninstalling ninja-1.11.1.3:
2025-08-14T21:22:01.3912106Z       Successfully uninstalled ninja-1.11.1.3
2025-08-14T21:22:01.4197135Z Successfully installed ninja-1.10.2
2025-08-14T21:22:01.4693231Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:22:01.4696948Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-14T21:22:01.4699055Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-08-14T21:22:01.4699716Z + [[ linux-jammy-rocm-py3.10 == *asan* ]]
2025-08-14T21:22:01.4700325Z + [[ linux-jammy-rocm-py3.10 == *-debug* ]]
2025-08-14T21:22:01.4700951Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]]
2025-08-14T21:22:01.4701875Z + echo 'We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass'
2025-08-14T21:22:01.4703035Z We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass
2025-08-14T21:22:01.4703872Z + cd test
2025-08-14T21:22:01.4704510Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)'
2025-08-14T21:22:02.9696116Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2025-08-14T21:22:02.9696859Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2025-08-14T21:22:02.9697612Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]]
2025-08-14T21:22:02.9701493Z + DYNAMO_BENCHMARK_FLAGS=()
2025-08-14T21:22:02.9702127Z + [[ distributed == *pr_time_benchmarks* ]]
2025-08-14T21:22:02.9702729Z + [[ distributed == *dynamo_eager* ]]
2025-08-14T21:22:02.9703269Z + [[ distributed == *aot_eager* ]]
2025-08-14T21:22:02.9703783Z + [[ distributed == *aot_inductor* ]]
2025-08-14T21:22:02.9704359Z + [[ distributed == *max_autotune_inductor* ]]
2025-08-14T21:22:02.9705636Z + [[ distributed == *inductor* ]]
2025-08-14T21:22:02.9706169Z + [[ distributed == *dynamic* ]]
2025-08-14T21:22:02.9706673Z + [[ distributed == *cpu* ]]
2025-08-14T21:22:02.9707199Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda)
2025-08-14T21:22:02.9745664Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-08-14T21:22:02.9746535Z + [[ linux-jammy-rocm-py3.10 == *-bazel-* ]]
2025-08-14T21:22:02.9755392Z + cd test
2025-08-14T21:22:02.9756041Z + python -c 'import torch; print(torch.__config__.show())'
2025-08-14T21:22:04.4030438Z PyTorch built with:
2025-08-14T21:22:04.4030980Z   - GCC 11.4
2025-08-14T21:22:04.4031402Z   - C++ Version: 201703
2025-08-14T21:22:04.4032384Z   - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-08-14T21:22:04.4033630Z   - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-08-14T21:22:04.4034397Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-08-14T21:22:04.4035038Z   - LAPACK is enabled (usually provided by MKL)
2025-08-14T21:22:04.4035631Z   - NNPACK is enabled
2025-08-14T21:22:04.4036121Z   - CPU capability usage: AVX2
2025-08-14T21:22:04.4036632Z   - HIP Runtime 6.4.43484
2025-08-14T21:22:04.4037072Z   - MIOpen 3.4.0
2025-08-14T21:22:04.4037465Z   - Magma 2.7.2
2025-08-14T21:22:04.4045072Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=1fc683cf17c8c673044538d10266c00f92987be2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 
2025-08-14T21:22:04.4053637Z 
2025-08-14T21:22:04.7043405Z + cd test
2025-08-14T21:22:04.7044114Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2025-08-14T21:22:05.9138490Z ATen/Parallel:
2025-08-14T21:22:05.9139136Z 	at::get_num_threads() : 128
2025-08-14T21:22:05.9139812Z 	at::get_num_interop_threads() : 128
2025-08-14T21:22:05.9140389Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2025-08-14T21:22:05.9140989Z 	omp_get_max_threads() : 128
2025-08-14T21:22:05.9141994Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications
2025-08-14T21:22:05.9143057Z 	mkl_get_max_threads() : 128
2025-08-14T21:22:05.9143754Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d)
2025-08-14T21:22:05.9144543Z std::thread::hardware_concurrency() : 128
2025-08-14T21:22:05.9145105Z Environment variables:
2025-08-14T21:22:05.9145557Z 	OMP_NUM_THREADS : [not set]
2025-08-14T21:22:05.9146028Z 	MKL_NUM_THREADS : [not set]
2025-08-14T21:22:05.9146504Z ATen parallel backend: OpenMP
2025-08-14T21:22:05.9146823Z 
2025-08-14T21:22:06.1871758Z + [[ distributed == *numpy_2* ]]
2025-08-14T21:22:06.1872545Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]]
2025-08-14T21:22:06.1873175Z + [[ distributed == *backward* ]]
2025-08-14T21:22:06.1873688Z + [[ distributed == *xla* ]]
2025-08-14T21:22:06.1874198Z + [[ distributed == *executorch* ]]
2025-08-14T21:22:06.1874751Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2025-08-14T21:22:06.1875340Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]]
2025-08-14T21:22:06.1875920Z + [[ distributed == distributed ]]
2025-08-14T21:22:06.1877014Z + test_distributed
2025-08-14T21:22:06.1877523Z + echo 'Testing distributed python tests'
2025-08-14T21:22:06.1878133Z Testing distributed python tests
2025-08-14T21:22:06.1878845Z + python test/run_test.py --distributed-tests --shard 3 3 --verbose
2025-08-14T21:22:08.9235040Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T21:22:08.9237567Z   import pkg_resources
2025-08-14T21:22:09.6397329Z Excluding distributed/rpc/test_faulty_agent on ROCm
2025-08-14T21:22:09.6398200Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm
2025-08-14T21:22:09.6398934Z Excluding distributed/rpc/test_share_memory on ROCm
2025-08-14T21:22:09.6399707Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm
2025-08-14T21:22:10.9856623Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json
2025-08-14T21:22:11.2521890Z Ignoring disabled issues:  ['']
2025-08-14T21:22:11.2763543Z Found test times from artifacts
2025-08-14T21:22:11.3293609Z Found test times from artifacts
2025-08-14T21:22:11.3305853Z Running all tests
2025-08-14T21:22:11.3402579Z Running parallel tests on 8 processes
2025-08-14T21:22:11.3404154Z Name: tests to run (est. time: 123.95min)
2025-08-14T21:22:11.3404827Z   Serial tests (77):
2025-08-14T21:22:11.3405313Z     distributed/test_store 1/1
2025-08-14T21:22:11.3406059Z     distributed/test_inductor_collectives 1/1
2025-08-14T21:22:11.3406711Z     distributed/tensor/test_dtensor_compile 1/1
2025-08-14T21:22:11.3407339Z     distributed/_composable/test_contract 1/1
2025-08-14T21:22:11.3407987Z     distributed/_shard/sharded_tensor/test_logger 1/1
2025-08-14T21:22:11.3408703Z     distributed/pipelining/test_transformer 1/1
2025-08-14T21:22:11.3409295Z     distributed/tensor/test_dtensor 1/1
2025-08-14T21:22:11.3410595Z     distributed/tensor/debug/test_comm_mode_features 1/1
2025-08-14T21:22:11.3411354Z     distributed/checkpoint/_experimental/test_builder 1/1
2025-08-14T21:22:11.3412052Z     distributed/tensor/test_init 1/1
2025-08-14T21:22:11.3412694Z     distributed/tensor/parallel/test_tp_style 1/1
2025-08-14T21:22:11.3413428Z     distributed/tensor/experimental/test_local_map 1/1
2025-08-14T21:22:11.3414102Z     distributed/tensor/test_random_ops 1/1
2025-08-14T21:22:11.3414778Z     distributed/checkpoint/test_async_process_executor 1/1
2025-08-14T21:22:11.3415442Z     distributed/test_backends 1/1
2025-08-14T21:22:11.3416059Z     distributed/tensor/experimental/test_tp_transform 1/1
2025-08-14T21:22:11.3416717Z     distributed/test_device_mesh 1/1
2025-08-14T21:22:11.3417278Z     distributed/checkpoint/test_planner 1/1
2025-08-14T21:22:11.3417850Z     distributed/launcher/test_run 1/1
2025-08-14T21:22:11.3418478Z     distributed/checkpoint/test_dtensor_checkpoint 1/1
2025-08-14T21:22:11.3419121Z     distributed/tensor/test_view_ops 1/1
2025-08-14T21:22:11.3419698Z     distributed/test_p2p_ipc 1/1
2025-08-14T21:22:11.3420375Z     distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1
2025-08-14T21:22:11.3421167Z     distributed/checkpoint/test_compatibility 1/1
2025-08-14T21:22:11.3421846Z     distributed/tensor/parallel/test_tp_examples 1/1
2025-08-14T21:22:11.3422585Z     distributed/_composable/fsdp/test_fully_shard_logging 1/1
2025-08-14T21:22:11.3423462Z     distributed/_composable/test_composability/test_2d_composability 2/2
2025-08-14T21:22:11.3424244Z     distributed/tensor/test_op_strategy 1/1
2025-08-14T21:22:11.3424969Z     distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1
2025-08-14T21:22:11.3425733Z     distributed/checkpoint/test_pg_transport 1/1
2025-08-14T21:22:11.3426381Z     distributed/tensor/debug/test_comm_mode 1/1
2025-08-14T21:22:11.3427410Z     distributed/fsdp/test_fsdp_pure_fp16 1/1
2025-08-14T21:22:11.3428033Z     distributed/fsdp/test_fsdp_state_dict 1/2
2025-08-14T21:22:11.3428624Z     distributed/test_fake_pg 1/1
2025-08-14T21:22:11.3429209Z     distributed/fsdp/test_fsdp_backward_prefetch 1/1
2025-08-14T21:22:11.3429966Z     distributed/_composable/fsdp/test_fully_shard_autograd 1/1
2025-08-14T21:22:11.3430757Z     distributed/_composable/fsdp/test_fully_shard_frozen 1/1
2025-08-14T21:22:11.3431595Z     distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1
2025-08-14T21:22:11.3432351Z     distributed/test_serialization 1/1
2025-08-14T21:22:11.3432935Z     distributed/checkpoint/test_state_dict 1/1
2025-08-14T21:22:11.3433649Z     distributed/checkpoint/test_file_system_checkpoint_cpu 1/1
2025-08-14T21:22:11.3434486Z     distributed/_composable/fsdp/test_fully_shard_extensions 1/1
2025-08-14T21:22:11.3435244Z     distributed/fsdp/test_fsdp_freezing_weights 1/1
2025-08-14T21:22:11.3435873Z     distributed/test_distributed_spawn 3/8
2025-08-14T21:22:11.3436521Z     distributed/test_distributed_spawn 6/8
2025-08-14T21:22:11.3437129Z     distributed/fsdp/test_fsdp_unshard_params 1/1
2025-08-14T21:22:11.3437749Z     distributed/test_multi_threaded_pg 1/1
2025-08-14T21:22:11.3438346Z     distributed/fsdp/test_fsdp_tp_integration 1/1
2025-08-14T21:22:11.3439058Z     distributed/_composable/fsdp/test_fully_shard_training 1/1
2025-08-14T21:22:11.3439744Z     distributed/fsdp/test_fsdp_misc 1/1
2025-08-14T21:22:11.3440395Z     distributed/fsdp/test_fsdp_clip_grad_norm 1/1
2025-08-14T21:22:11.3441221Z     distributed/_shard/sharding_spec/test_sharding_spec 1/1
2025-08-14T21:22:11.3442069Z     distributed/checkpoint/test_state_dict_utils 1/1
2025-08-14T21:22:11.3442943Z     distributed/algorithms/quantization/test_quantization 1/1
2025-08-14T21:22:11.3443723Z     distributed/fsdp/test_fsdp_traversal 1/1
2025-08-14T21:22:11.3444358Z     distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1
2025-08-14T21:22:11.3444988Z     distributed/test_composability 1/1
2025-08-14T21:22:11.3445660Z     distributed/fsdp/test_fsdp_core 2/3
2025-08-14T21:22:11.3446328Z     distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1
2025-08-14T21:22:11.3447384Z     distributed/fsdp/test_fsdp_mixed_precision 1/1
2025-08-14T21:22:11.3447983Z     distributed/test_c10d_ucc 1/1
2025-08-14T21:22:11.3448497Z     distributed/test_c10d_common 1/1
2025-08-14T21:22:11.3449150Z     distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1
2025-08-14T21:22:11.3449852Z     distributed/test_c10d_spawn_gloo 1/1
2025-08-14T21:22:11.3450482Z     distributed/test_c10d_spawn_nccl 1/1
2025-08-14T21:22:11.3451109Z     distributed/test_launcher 1/1
2025-08-14T21:22:11.3451722Z     distributed/test_c10d_spawn_ucc 1/1
2025-08-14T21:22:11.3452354Z     distributed/test_c10d_nccl 1/3
2025-08-14T21:22:11.3452961Z     distributed/test_c10d_ops_nccl 1/1
2025-08-14T21:22:11.3453596Z     distributed/elastic/events/lib_test 1/1
2025-08-14T21:22:11.3454290Z     distributed/elastic/metrics/api_test 1/1
2025-08-14T21:22:11.3455039Z     distributed/elastic/multiprocessing/api_test 1/1
2025-08-14T21:22:11.3455873Z     distributed/elastic/timer/local_timer_example 1/1
2025-08-14T21:22:11.3456658Z     distributed/elastic/timer/local_timer_test 1/1
2025-08-14T21:22:11.3457305Z     distributed/elastic/utils/distributed_test 1/1
2025-08-14T21:22:11.3457924Z     distributed/elastic/utils/logging_test 1/1
2025-08-14T21:22:11.3458508Z     distributed/elastic/utils/util_test 1/1
2025-08-14T21:22:11.3459044Z   Parallel tests (0):
2025-08-14T21:22:11.3459489Z Name: excluded (est. time: 0.0min)
2025-08-14T21:22:11.3459980Z   Serial tests (0):
2025-08-14T21:22:11.3460374Z   Parallel tests (0):
2025-08-14T21:22:11.3460974Z Running distributed/test_store 1/1 ... [2025-08-14 21:22:11.340733]
2025-08-14T21:22:11.3461689Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:22:11.3464317Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_store.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:22:11.341027]
2025-08-14T21:31:05.6050043Z 
2025-08-14T21:31:05.6051415Z distributed/test_store 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_store_1.1_1bf494cf9df82b3c_.log
2025-08-14T21:31:05.6118702Z Running 120 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append, test/distributed/test_store.py::FileStoreTest::test_clone, test/distributed/test_store.py::FileStoreTest::test_compare_set, test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file, test/distributed/test_store.py::FileStoreTest::test_multi_get, test/distributed/test_store.py::FileStoreTest::test_multi_set, test/distributed/test_store.py::FileStoreTest::test_queues, test/distributed/test_store.py::FileStoreTest::test_queues_bidirectional, test/distributed/test_store.py::FileStoreTest::test_queues_nonblocking, test/distributed/test_store.py::FileStoreTest::test_queues_timeout, test/distributed/test_store.py::FileStoreTest::test_refcount, test/distributed/test_store.py::FileStoreTest::test_set_get_check, test/distributed/test_store.py::FileStoreTest::test_simple_wait, test/distributed/test_store.py::HashStoreTest::test_append, test/distributed/test_store.py::HashStoreTest::test_clone, test/distributed/test_store.py::HashStoreTest::test_compare_set, test/distributed/test_store.py::HashStoreTest::test_multi_get, test/distributed/test_store.py::HashStoreTest::test_multi_set, test/distributed/test_store.py::HashStoreTest::test_queues, test/distributed/test_store.py::HashStoreTest::test_queues_bidirectional, test/distributed/test_store.py::HashStoreTest::test_queues_nonblocking, test/distributed/test_store.py::HashStoreTest::test_queues_timeout, test/distributed/test_store.py::HashStoreTest::test_set_get_check, test/distributed/test_store.py::HashStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store, test/distributed/test_store.py::PrefixFileStoreTest::test_append, test/distributed/test_store.py::PrefixFileStoreTest::test_clone, test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get, test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set, test/distributed/test_store.py::PrefixFileStoreTest::test_queues, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_bidirectional, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_nonblocking, test/distributed/test_store.py::PrefixFileStoreTest::test_queues_timeout, test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::TCPStoreTest::test_agent_store, test/distributed/test_store.py::TCPStoreTest::test_append, test/distributed/test_store.py::TCPStoreTest::test_clone, test/distributed/test_store.py::TCPStoreTest::test_compare_set, test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::TCPStoreTest::test_multi_get, test/distributed/test_store.py::TCPStoreTest::test_multi_set, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::TCPStoreTest::test_multitenancy, test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::TCPStoreTest::test_queues, test/distributed/test_store.py::TCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::TCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::TCPStoreTest::test_queues_timeout, test/distributed/test_store.py::TCPStoreTest::test_repr, test/distributed/test_store.py::TCPStoreTest::test_set_get_check, test/distributed/test_store.py::TCPStoreTest::test_simple_wait, test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use, test/distributed/test_store.py::LibUvTCPStoreTest::test_agent_store, test/distributed/test_store.py::LibUvTCPStoreTest::test_append, test/distributed/test_store.py::LibUvTCPStoreTest::test_clone, test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size, test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy, test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_timeout, test/distributed/test_store.py::LibUvTCPStoreTest::test_repr, test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check, test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait, test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients, test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket, test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises, test/distributed/test_store.py::PrefixTCPStoreTest::test_append, test/distributed/test_store.py::PrefixTCPStoreTest::test_clone, test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get, test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_bidirectional, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_nonblocking, test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_timeout, test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check, test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait, test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store, test/distributed/test_store.py::PythonStoreTest::test_set_get, test/distributed/test_store.py::RendezvousTest::test_unknown_handler, test/distributed/test_store.py::RendezvousTest::test_url_with_node_params, test/distributed/test_store.py::RendezvousEnvTest::test_nominal, test/distributed/test_store.py::RendezvousFileTest::test_common_errors, test/distributed/test_store.py::RendezvousFileTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_common_errors, test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout, test/distributed/test_store.py::RendezvousTCPTest::test_nominal, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set, test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv, test/distributed/test_store.py::TestPythonStore::test_append_roundtrip, test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough, test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip, test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip, test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store, test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv, test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var, test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param, test/distributed/test_store.py::TestClientProtocol::test_client_connect
2025-08-14T21:31:05.6182486Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_append
2025-08-14T21:31:05.6183725Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_clone
2025-08-14T21:31:05.6184963Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_compare_set
2025-08-14T21:31:05.6186349Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_init_pg_and_rpc_with_same_file
2025-08-14T21:31:05.6187670Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_get
2025-08-14T21:31:05.6188833Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_multi_set
2025-08-14T21:31:05.6189973Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues
2025-08-14T21:31:05.6191186Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6192517Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6193790Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_queues_timeout
2025-08-14T21:31:05.6195005Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_refcount
2025-08-14T21:31:05.6196637Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_set_get_check
2025-08-14T21:31:05.6197845Z Running 1 items in this shard: test/distributed/test_store.py::FileStoreTest::test_simple_wait
2025-08-14T21:31:05.6199005Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_append
2025-08-14T21:31:05.6200108Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_clone
2025-08-14T21:31:05.6201270Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_compare_set
2025-08-14T21:31:05.6202437Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_get
2025-08-14T21:31:05.6203589Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_multi_set
2025-08-14T21:31:05.6204719Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues
2025-08-14T21:31:05.6206111Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6207465Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6208933Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_queues_timeout
2025-08-14T21:31:05.6210170Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_set_get_check
2025-08-14T21:31:05.6211499Z Running 1 items in this shard: test/distributed/test_store.py::HashStoreTest::test_simple_wait
2025-08-14T21:31:05.6213044Z Running 1 items in this shard: test/distributed/test_store.py::PrefixStoreTest::test_get_underlying_store
2025-08-14T21:31:05.6214357Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_append
2025-08-14T21:31:05.6215914Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_clone
2025-08-14T21:31:05.6217213Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_compare_set
2025-08-14T21:31:05.6218504Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_get
2025-08-14T21:31:05.6219765Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_multi_set
2025-08-14T21:31:05.6221002Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues
2025-08-14T21:31:05.6222334Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6223743Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6225087Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_queues_timeout
2025-08-14T21:31:05.6226635Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_set_get_check
2025-08-14T21:31:05.6228021Z Running 1 items in this shard: test/distributed/test_store.py::PrefixFileStoreTest::test_simple_wait
2025-08-14T21:31:05.6229333Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_address_already_in_use
2025-08-14T21:31:05.6230685Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_agent_store
2025-08-14T21:31:05.6231842Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_append
2025-08-14T21:31:05.6232955Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_clone
2025-08-14T21:31:05.6234098Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_compare_set
2025-08-14T21:31:05.6235412Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_init_pg_and_rpc_with_same_socket
2025-08-14T21:31:05.6236723Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_get
2025-08-14T21:31:05.6238223Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_set
2025-08-14T21:31:05.6239544Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_fixed_world_size
2025-08-14T21:31:05.6241031Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multi_worker_with_nonfixed_world_size
2025-08-14T21:31:05.6242382Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_multitenancy
2025-08-14T21:31:05.6243617Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_numkeys_delkeys
2025-08-14T21:31:05.6244787Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues
2025-08-14T21:31:05.6246328Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6247638Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6248898Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_queues_timeout
2025-08-14T21:31:05.6250101Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_repr
2025-08-14T21:31:05.6251257Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_set_get_check
2025-08-14T21:31:05.6252459Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_simple_wait
2025-08-14T21:31:05.6253776Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_store_timeout_on_missing_clients
2025-08-14T21:31:05.6255163Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_take_over_listen_socket
2025-08-14T21:31:05.6256474Z Running 1 items in this shard: test/distributed/test_store.py::TCPStoreTest::test_world_size_0_raises
2025-08-14T21:31:05.6258024Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_address_already_in_use
2025-08-14T21:31:05.6259250Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_agent_store
2025-08-14T21:31:05.6260334Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_append
2025-08-14T21:31:05.6261388Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_clone
2025-08-14T21:31:05.6262467Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_compare_set
2025-08-14T21:31:05.6263705Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_init_pg_and_rpc_with_same_socket
2025-08-14T21:31:05.6264917Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_get
2025-08-14T21:31:05.6266003Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_set
2025-08-14T21:31:05.6267236Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_fixed_world_size
2025-08-14T21:31:05.6268637Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multi_worker_with_nonfixed_world_size
2025-08-14T21:31:05.6269911Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_multitenancy
2025-08-14T21:31:05.6271054Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_numkeys_delkeys
2025-08-14T21:31:05.6272152Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues
2025-08-14T21:31:05.6273288Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6274480Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6275654Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_queues_timeout
2025-08-14T21:31:05.6277287Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_repr
2025-08-14T21:31:05.6278374Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_set_get_check
2025-08-14T21:31:05.6279494Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_simple_wait
2025-08-14T21:31:05.6280723Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_store_timeout_on_missing_clients
2025-08-14T21:31:05.6282013Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_take_over_listen_socket
2025-08-14T21:31:05.6283230Z Running 1 items in this shard: test/distributed/test_store.py::LibUvTCPStoreTest::test_world_size_0_raises
2025-08-14T21:31:05.6284363Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_append
2025-08-14T21:31:05.6285437Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_clone
2025-08-14T21:31:05.6286660Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_compare_set
2025-08-14T21:31:05.6287832Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_get
2025-08-14T21:31:05.6288954Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_multi_set
2025-08-14T21:31:05.6290036Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues
2025-08-14T21:31:05.6291194Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_bidirectional
2025-08-14T21:31:05.6292410Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_nonblocking
2025-08-14T21:31:05.6293605Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_queues_timeout
2025-08-14T21:31:05.6295052Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_set_get_check
2025-08-14T21:31:05.6296211Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_simple_wait
2025-08-14T21:31:05.6297438Z Running 1 items in this shard: test/distributed/test_store.py::PrefixTCPStoreTest::test_underlying_non_prefix_store
2025-08-14T21:31:05.6298624Z Running 1 items in this shard: test/distributed/test_store.py::PythonStoreTest::test_set_get
2025-08-14T21:31:05.6299703Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_unknown_handler
2025-08-14T21:31:05.6300850Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTest::test_url_with_node_params
2025-08-14T21:31:05.6301965Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousEnvTest::test_nominal
2025-08-14T21:31:05.6303066Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_common_errors
2025-08-14T21:31:05.6304186Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousFileTest::test_nominal
2025-08-14T21:31:05.6305303Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_common_errors
2025-08-14T21:31:05.6306425Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_dns_timeout
2025-08-14T21:31:05.6307492Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_nominal
2025-08-14T21:31:05.6308717Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_doest_break_client
2025-08-14T21:31:05.6310045Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_timeout_set
2025-08-14T21:31:05.6311285Z Running 1 items in this shard: test/distributed/test_store.py::RendezvousTCPTest::test_tcp_store_url_with_libuv
2025-08-14T21:31:05.6312495Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_append_roundtrip
2025-08-14T21:31:05.6313697Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_extended_methods_fallbacks
2025-08-14T21:31:05.6315262Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_passthrough
2025-08-14T21:31:05.6316525Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_has_extended_api_roundtrip
2025-08-14T21:31:05.6317736Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_get_roundtrip
2025-08-14T21:31:05.6318907Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_multi_set_roundtrip
2025-08-14T21:31:05.6320084Z Running 1 items in this shard: test/distributed/test_store.py::TestPythonStore::test_optional_methods_fail
2025-08-14T21:31:05.6321281Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_file_store
2025-08-14T21:31:05.6322486Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_hash_store
2025-08-14T21:31:05.6323746Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_prefix_file_store
2025-08-14T21:31:05.6324985Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store
2025-08-14T21:31:05.6326297Z Running 1 items in this shard: test/distributed/test_store.py::TestMultiThreadedWait::test_wait_tcp_store_uv
2025-08-14T21:31:05.6327543Z Running 1 items in this shard: test/distributed/test_store.py::TimeoutTest::test_interrupt_doesnt_break_wait
2025-08-14T21:31:05.6328728Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_env_var
2025-08-14T21:31:05.6329911Z Running 1 items in this shard: test/distributed/test_store.py::InitPgWithNonUvStore::test_with_url_param
2025-08-14T21:31:05.6331380Z Running 1 items in this shard: test/distributed/test_store.py::TestClientProtocol::test_client_connect
2025-08-14T21:31:05.6332058Z 
2025-08-14T21:31:05.6332465Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T21:31:05.6333205Z Uploading artifacts took 0.00 seconds
2025-08-14T21:31:05.6333918Z Running distributed/test_inductor_collectives 1/1 ... [2025-08-14 21:31:05.609632]
2025-08-14T21:31:05.6334668Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:31:05.6336347Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_inductor_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:31:05.610253]
2025-08-14T21:35:32.4990604Z 
2025-08-14T21:35:32.4993407Z distributed/test_inductor_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_inductor_collectives_1.1_1102cc51790cacfd_.log
2025-08-14T21:35:32.5046658Z Running 58 items in this shard: test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_False, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_recompute_is_always_banned_override_with_ac_True, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_all_to_all_single_inductor_split_sizes_none, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_contiguous_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_into_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_output_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allgather_scalar_tensor_input, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_inductor_cudagraph_trees, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_allreduce_input_buffer_reuse, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_broadcast_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_c10d_functional_tagged_pt2_compliant, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_eager_async_allreduce_inductor_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_inductor_allreduce_eager_wait, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_permute_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesMultiProc::test_reduce_scatter_tensor_inductor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_all_gather_bucket, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_backwards, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_GroupMember_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source__get_default_group, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_get_world_group_source_group_WORLD, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_graphbreaks_unsupported_async_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_pg_var, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_args_match, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_gather_list, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_all_to_all_single, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_kwargs_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_positional_none, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_pg_mode_unspecified, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op0, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op1, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op2, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op3, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_allreduce_reduce_op_reduce_op4, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_rewrite_dist_reduce_scatter, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_support_collective_op_with_async_op_False, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_all_gather_tensor_pg, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allgather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_allreduce, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_dynamo_trace_reduce_scatter_tensor, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_all_gather_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_doesnt_mutate_shared_graph_partition, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_reduce_scatter_coalesced, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_single_op, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_inductor_steal_buffer, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_meta, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reduce_scatter_bucket, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_peak_memory_bucketed, test/distributed/test_inductor_collectives.py::TestCollectivesInductor::test_reorder_respects_wait_dep, test/distributed/test_inductor_collectives.py::TestSyncDecisionCrossRanks::test_sync_decision_cross_ranks
2025-08-14T21:35:32.5094380Z 
2025-08-14T21:35:32.5094880Z Running distributed/tensor/test_dtensor_compile 1/1 ... [2025-08-14 21:35:32.499565]
2025-08-14T21:35:32.5095765Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:35:32.5097707Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_compile.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:35:32.500101]
2025-08-14T21:39:04.6643574Z 
2025-08-14T21:39:04.6646273Z distributed/tensor/test_dtensor_compile 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_compile_1.1_7e922960d884c6b1_.log
2025-08-14T21:39:04.6680305Z Running 39 items in this shard: test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_device_mesh_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_attribute_access_on_intermediate, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_basic_export, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_dynamo_disable, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_constructor_w_graph_break, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_contiguous_dtensor_noncontiguous_local_as_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_different_gradient_placement, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dont_recompile_on_same_placement_devicemesh, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamic_slice, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_dynamo_device_mesh_attrs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_noncontiguous_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_graph_output, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dtensor_partial_placement_redistribute_unbalanced_correct_strides, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_dynamic_shapes, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_from_local_redistribute_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_dtensor_recompile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_dynamo_to_local_kwargs_forward_hook, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_fakify_dtensor, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_graph_input_is_async, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_placement_compile, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_tp_compile_comm_reordering_graph_partition, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompile::test_unwrap_async_collective_tensor_tangent, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_ac_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_2d_fsdp_tp_compile_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_compile_dtensor_redistribute_backward_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_False_use_ca_True, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_False, test/distributed/tensor/test_dtensor_compile.py::TestDTensorCompileE2E::test_tp_compile_fullgraph_is_seq_parallel_True_use_ca_True
2025-08-14T21:39:04.6712077Z 
2025-08-14T21:39:04.6712559Z Running distributed/_composable/test_contract 1/1 ... [2025-08-14 21:39:04.664932]
2025-08-14T21:39:04.6713416Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:39:04.6715341Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_contract.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:39:04.665526]
2025-08-14T21:39:08.4907144Z 
2025-08-14T21:39:08.4908912Z distributed/_composable/test_contract 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_contract_1.1_9fcedf1d071070b5_.log
2025-08-14T21:39:08.4913329Z Running 5 items in this shard: test/distributed/_composable/test_contract.py::TestContract::test_add_hooks, test/distributed/_composable/test_contract.py::TestContract::test_modify_fqn, test/distributed/_composable/test_contract.py::TestContract::test_multi_module_api, test/distributed/_composable/test_contract.py::TestContract::test_registry, test/distributed/_composable/test_contract.py::TestContract::test_state
2025-08-14T21:39:08.4916735Z 
2025-08-14T21:39:08.4917389Z Running distributed/_shard/sharded_tensor/test_logger 1/1 ... [2025-08-14 21:39:08.490828]
2025-08-14T21:39:08.4918545Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:39:08.4921022Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:39:08.491342]
2025-08-14T21:39:12.3165154Z 
2025-08-14T21:39:12.3166983Z distributed/_shard/sharded_tensor/test_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_logger_1.1_eb469f26dcfbe13a_.log
2025-08-14T21:39:12.3169343Z Running 1 items in this shard: test/distributed/_shard/sharded_tensor/test_logger.py::ShardingSpecLoggerTest::test_get_or_create_logger
2025-08-14T21:39:12.3170383Z 
2025-08-14T21:39:12.3172473Z Running distributed/pipelining/test_transformer 1/1 ... [2025-08-14 21:39:12.316778]
2025-08-14T21:39:12.3173393Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:39:12.3179007Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_transformer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:39:12.317311]
2025-08-14T21:39:20.1023661Z 
2025-08-14T21:39:20.1025437Z distributed/pipelining/test_transformer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_transformer_1.1_82268277db82907d_.log
2025-08-14T21:39:20.1027765Z Running 1 items in this shard: test/distributed/pipelining/test_transformer.py::TransformerTestsCUDA::test_ir_cuda
2025-08-14T21:39:20.1028670Z 
2025-08-14T21:39:20.1029176Z Running distributed/tensor/test_dtensor 1/1 ... [2025-08-14 21:39:20.102482]
2025-08-14T21:39:20.1029992Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:39:20.1037861Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:39:20.103032]
2025-08-14T21:42:49.2037531Z 
2025-08-14T21:42:49.2043609Z distributed/tensor/test_dtensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_1.1_900c5ada1611c881_.log
2025-08-14T21:42:49.2066643Z Running 35 items in this shard: test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_async_output, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_constructor, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_new_empty_strided, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_properties, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_save_load_import, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_hash, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_spec_read_only_after_set, test/distributed/tensor/test_dtensor.py::DTensorTest::test_dtensor_stride, test/distributed/tensor/test_dtensor.py::DTensorTest::test_from_local, test/distributed/tensor/test_dtensor.py::DTensorTest::test_from_local_negative_dim, test/distributed/tensor/test_dtensor.py::DTensorTest::test_from_local_then_to_local, test/distributed/tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding, test/distributed/tensor/test_dtensor.py::DTensorTest::test_from_local_uneven_sharding_raise_error, test/distributed/tensor/test_dtensor.py::DTensorTest::test_full_tensor_grad_hint, test/distributed/tensor/test_dtensor.py::DTensorTest::test_full_tensor_sync, test/distributed/tensor/test_dtensor.py::DTensorTest::test_meta_dtensor, test/distributed/tensor/test_dtensor.py::DTensorTest::test_modules_w_meta_dtensor, test/distributed/tensor/test_dtensor.py::DTensorTest::test_shard_tensor, test/distributed/tensor/test_dtensor.py::DTensorTest::test_shard_tensor_2d, test/distributed/tensor/test_dtensor.py::DTensorTest::test_to_local, test/distributed/tensor/test_dtensor.py::DTensorTest::test_to_local_grad_hint, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_auto_implicit_replication, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_default_value_sub_mesh, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_device_mesh_nd, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_2d_mesh, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_api_device_mesh_context_manager, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_device_mesh_device_conversion, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_dtensor_spec_local_shard_offset, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_from_local_sub_mesh, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_implicit_replication, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_metadata_consistency_check, test/distributed/tensor/test_dtensor.py::DTensorMeshTest::test_redistribute_sub_mesh, test/distributed/tensor/test_dtensor.py::TestDTensorPlacementTypes::test_split_tensor_1D, test/distributed/tensor/test_dtensor.py::DTensorLogTest::test_dtensor_log
2025-08-14T21:42:49.2089371Z 
2025-08-14T21:42:49.2089967Z Running distributed/tensor/debug/test_comm_mode_features 1/1 ... [2025-08-14 21:42:49.204606]
2025-08-14T21:42:49.2090947Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:42:49.2093042Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/debug/test_comm_mode_features.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:42:49.205176]
2025-08-14T21:43:22.4993151Z 
2025-08-14T21:43:22.4995354Z distributed/tensor/debug/test_comm_mode_features 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.debug.test_comm_mode_features_1.1_c219b7eec564c421_.log
2025-08-14T21:43:22.4998436Z Running 4 items in this shard: test/distributed/tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLPStacked_distributed_sharding_display, test/distributed/tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_distributed_sharding_display, test/distributed/tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_MLP_module_tracing, test/distributed/tensor/debug/test_comm_mode_features.py::TestCommModeFeatures::test_transformer_module_tracing
2025-08-14T21:43:22.5000649Z 
2025-08-14T21:43:22.5001430Z Running distributed/checkpoint/_experimental/test_builder 1/1 ... [2025-08-14 21:43:22.499622]
2025-08-14T21:43:22.5002109Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:43:22.5007378Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_builder.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:22.500193]
2025-08-14T21:43:29.3825333Z 
2025-08-14T21:43:29.3827350Z distributed/checkpoint/_experimental/test_builder 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_builder_1.1_ab66b744a5b837df_.log
2025-08-14T21:43:29.3832883Z Running 4 items in this shard: test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_async_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_config_first, test/distributed/checkpoint/_experimental/test_builder.py::TestMakeCheckpointer::test_make_sync_checkpointer_with_custom_config
2025-08-14T21:43:29.3836651Z 
2025-08-14T21:43:29.3837092Z Running distributed/tensor/test_init 1/1 ... [2025-08-14 21:43:29.382411]
2025-08-14T21:43:29.3837935Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:43:29.3839762Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:29.382766]
2025-08-14T21:44:17.1100758Z 
2025-08-14T21:44:17.1102677Z distributed/tensor/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_init_1.1_f31a90cca6958170_.log
2025-08-14T21:44:17.1107933Z Running 7 items in this shard: test/distributed/tensor/test_init.py::DTensorInitOpsTest::test_init_ops, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_empty, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_full, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_ones, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros_full_mesh, test/distributed/tensor/test_init.py::DTensorConstructorTest::test_zeros_submesh
2025-08-14T21:44:17.1111967Z 
2025-08-14T21:44:17.1112500Z Running distributed/tensor/parallel/test_tp_style 1/1 ... [2025-08-14 21:44:17.110462]
2025-08-14T21:44:17.1113404Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:44:17.1115800Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_style.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:44:17.111024]
2025-08-14T21:45:22.7791435Z 
2025-08-14T21:45:22.7793572Z distributed/tensor/parallel/test_tp_style 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_style_1.1_cd83ede11e93fd21_.log
2025-08-14T21:45:22.7803227Z Running 9 items in this shard: test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_colwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_input_multiple_inputs, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_kwargs_input, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_prepare_module_output, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_embedding, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_rowwise_parallel_style, test/distributed/tensor/parallel/test_tp_style.py::TensorParallelStyleTest::test_sequence_parallel_style
2025-08-14T21:45:22.7810915Z 
2025-08-14T21:45:22.7811497Z Running distributed/tensor/experimental/test_local_map 1/1 ... [2025-08-14 21:45:22.779570]
2025-08-14T21:45:22.7812450Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:45:22.7814459Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_local_map.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:45:22.780144]
2025-08-14T21:46:06.2497650Z 
2025-08-14T21:46:06.2504005Z distributed/tensor/experimental/test_local_map 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_local_map_1.1_c2577f88072fa976_.log
2025-08-14T21:46:06.2510184Z Running 6 items in this shard: test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_correctness, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_in_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_out_placements, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_redistribute, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_local_map_with_grad_placement, test/distributed/tensor/experimental/test_local_map.py::TestLocalMap::test_multi_mesh_inputs
2025-08-14T21:46:06.2515472Z 
2025-08-14T21:46:06.2515924Z Running distributed/tensor/test_random_ops 1/1 ... [2025-08-14 21:46:06.250038]
2025-08-14T21:46:06.2516768Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:46:06.2518926Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_random_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:46:06.250585]
2025-08-14T21:47:34.3691816Z 
2025-08-14T21:47:34.3693648Z distributed/tensor/test_random_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_random_ops_1.1_37bcfc7f0694e634_.log
2025-08-14T21:47:34.3703478Z Running 13 items in this shard: test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_fsdp_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_ops, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_init_with_user_generator, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_meta_tensor_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomInitTest::test_tp_model_meta_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_dropout_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_rand_1d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_deterministic_uniform_2d, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_manual_seed_submesh, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_pipeline_parallel_manual_seed, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpTest::test_rng_tracker_init, test/distributed/tensor/test_random_ops.py::DistTensorRandomOpsTest3D::test_hsdp_tp_model_meta_init
2025-08-14T21:47:34.3709157Z 
2025-08-14T21:47:34.3709525Z Running distributed/checkpoint/test_async_process_executor 1/1 ... [2025-08-14 21:47:34.369453]
2025-08-14T21:47:34.3710110Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:47:34.3711344Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_async_process_executor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:47:34.370025]
2025-08-14T21:47:51.9269205Z 
2025-08-14T21:47:51.9271502Z distributed/checkpoint/test_async_process_executor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_async_process_executor_1.1_37513525dce6cd49_.log
2025-08-14T21:47:51.9274280Z Running 1 items in this shard: test/distributed/checkpoint/test_async_process_executor.py::TestAsyncProcessExecutor::test_checkpoint_save_failure_continues_serving
2025-08-14T21:47:51.9275515Z 
2025-08-14T21:47:51.9280645Z Running distributed/test_backends 1/1 ... [2025-08-14 21:47:51.927443]
2025-08-14T21:47:51.9281516Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:47:51.9287129Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_backends.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:47:51.928032]
2025-08-14T21:47:56.3041446Z 
2025-08-14T21:47:56.3042981Z distributed/test_backends 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_backends_1.1_5185de2cc7611432_.log
2025-08-14T21:47:56.3047250Z Running 2 items in this shard: test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_create_pg_cuda, test/distributed/test_backends.py::TestMiscCollectiveUtilsCUDA::test_device_to_backend_mapping_cuda
2025-08-14T21:47:56.3049202Z 
2025-08-14T21:47:56.3050669Z Running distributed/tensor/experimental/test_tp_transform 1/1 ... [2025-08-14 21:47:56.304403]
2025-08-14T21:47:56.3051646Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:47:56.3055554Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_tp_transform.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:47:56.305012]
2025-08-14T21:48:41.5726568Z 
2025-08-14T21:48:41.5729051Z distributed/tensor/experimental/test_tp_transform 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_tp_transform_1.1_e7c0bc0a6fc44d3c_.log
2025-08-14T21:48:41.5733393Z Running 3 items in this shard: test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_e2e, test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_no_bias, test/distributed/tensor/experimental/test_tp_transform.py::TensorParallelTest::test_tp_transform_with_uncovered_op
2025-08-14T21:48:41.5736201Z 
2025-08-14T21:48:41.5752180Z Running distributed/test_device_mesh 1/1 ... [2025-08-14 21:48:41.572935]
2025-08-14T21:48:41.5753065Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:48:41.5755770Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_device_mesh.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:48:41.573515]
2025-08-14T21:52:57.6866968Z 
2025-08-14T21:52:57.6868193Z distributed/test_device_mesh 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_device_mesh_1.1_574804bdd51d454c_.log
2025-08-14T21:52:57.6906407Z Running 55 items in this shard: test/distributed/test_device_mesh.py::DeviceMeshTestGlooBackend::test_device_mesh_reuse_default_group, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_auto_set_device_from_heuristic, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_auto_set_device_from_local_rank, test/distributed/test_device_mesh.py::DeviceMeshSetDeviceTest::test_manual_set_device, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_2d_mesh_non_eager_init_subgroup, test/distributed/test_device_mesh.py::DeviceMeshTest::test_assert_invalid_mesh_tensor, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_2d, test/distributed/test_device_mesh.py::DeviceMeshTest::test_device_mesh_init_backend, test/distributed/test_device_mesh.py::DeviceMeshTest::test_fake_pg_device_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_global_pg, test/distributed/test_device_mesh.py::DeviceMeshTest::test_from_group_with_invalid_mesh, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_group_and_get_all_groups, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank, test/distributed/test_device_mesh.py::DeviceMeshTest::test_get_local_rank_raises_exception, test/distributed/test_device_mesh.py::DeviceMeshTest::test_init_process_group, test/distributed/test_device_mesh.py::DeviceMeshTest::test_raises_invalid_device_type, test/distributed/test_device_mesh.py::DeviceMeshTest::test_set_mesh_dim_group_options, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_nd, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_device_mesh_parent_child_hash, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_2d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_from_group_with_mesh_shape_3d, test/distributed/test_device_mesh.py::DeviceMeshTestNDim::test_get_local_rank_3d, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_idx_and_backend_eager, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_idx_and_backend_lazy, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_dict_with_name_and_options, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_backend_override_argument_errors, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_init_device_mesh, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_duplicate_mesh_dim_names, test/distributed/test_device_mesh.py::InitDeviceMeshTest::test_raises_mesh_shape_mesh_dim_names_mismatch, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_cache_and_reuse_submesh_slice_result, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_flatten_mesh_4d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_1d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_2d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_get_item_3d_noncontiguous_slicing, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_invalid_mesh_dim_name, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_raises_no_mesh_dim_found, test/distributed/test_device_mesh.py::TestDeviceMeshGetItem::test_reconstruct_mesh_with_flatten_dim, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_all_submeshes, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_mesh_dim_by_name, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_get_root_mesh_dim_not_exist, test/distributed/test_device_mesh.py::TestMeshEnv::test_mesh_slice_fake_tensor_mode, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_all_gather_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_broadcast_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_contiguous, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_reduce_scatter_uneven, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_1d, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_nd, test/distributed/test_device_mesh.py::DeviceMeshCollectiveTest::test_scatter_uneven
2025-08-14T21:52:57.6940377Z 
2025-08-14T21:52:57.6940878Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T21:52:57.6941761Z Uploading artifacts took 0.00 seconds
2025-08-14T21:52:57.6942616Z Running distributed/checkpoint/test_planner 1/1 ... [2025-08-14 21:52:57.688605]
2025-08-14T21:52:57.6943528Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:52:57.6945550Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:52:57.689165]
2025-08-14T21:53:01.7150872Z 
2025-08-14T21:53:01.7152930Z distributed/checkpoint/test_planner 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_planner_1.1_2454922e4c519931_.log
2025-08-14T21:53:01.7163844Z Running 15 items in this shard: test/distributed/checkpoint/test_planner.py::TestSavePlan::test_dedup_plans, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_finish_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_global_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_resharding, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_load_with_world_size_diff_by_one, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_load_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan, test/distributed/checkpoint/test_planner.py::TestSavePlan::test_local_plan_with_caching, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_compare_save_plans, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_create_read_item_from_chunks, test/distributed/checkpoint/test_planner.py::TestPlannerHelpers::test_merge_delta_local_plans, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_load_different_sizes_throws, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_strict, test/distributed/checkpoint/test_planner.py::TestLoadPlanner::test_version_key_in_planner_data
2025-08-14T21:53:01.7174726Z 
2025-08-14T21:53:01.7175187Z Running distributed/launcher/test_run 1/1 ... [2025-08-14 21:53:01.715503]
2025-08-14T21:53:01.7175985Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:53:01.7177843Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/launcher/test_run.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:53:01.716055]
2025-08-14T21:53:44.0832518Z 
2025-08-14T21:53:44.0835044Z distributed/launcher/test_run 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.launcher.test_run_1.1_1f0e9a20eeddda4a_.log
2025-08-14T21:53:44.0853634Z Running 24 items in this shard: test/distributed/launcher/test_run.py::ElasticLaunchTest::test_capture_logs_using_default_logs_specs, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_env_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_init_method_tcp_with_torchelastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_not_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_is_torchelastic_launched_with_logs_spec_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_agent_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_multiple_agents, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_elastic_worker_raise_exception, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_run_path, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_shutdown, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_standalone, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_bash, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_default_nproc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_user_script_python_caffe2_bc, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_launch_with_env_vars, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_logs_logs_spec_entrypoint_must_be_defined, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_min_max_nodes_parse, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_gpu_launch_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_auto_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_number_configurations, test/distributed/launcher/test_run.py::ElasticLaunchTest::test_nproc_launch_unknown_configurations
2025-08-14T21:53:44.0871759Z 
2025-08-14T21:53:44.0872326Z Running distributed/checkpoint/test_dtensor_checkpoint 1/1 ... [2025-08-14 21:53:44.084404]
2025-08-14T21:53:44.0873278Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:53:44.0875291Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_dtensor_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:53:44.084993]
2025-08-14T21:53:54.8254686Z 
2025-08-14T21:53:54.8256594Z distributed/checkpoint/test_dtensor_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_dtensor_checkpoint_1.1_e7cc91d1f7310309_.log
2025-08-14T21:53:54.8259083Z Running 1 items in this shard: test/distributed/checkpoint/test_dtensor_checkpoint.py::DTensorPlanner::test_distributed_tensor_planner
2025-08-14T21:53:54.8263094Z 
2025-08-14T21:53:54.8263684Z Running distributed/tensor/test_view_ops 1/1 ... [2025-08-14 21:53:54.825839]
2025-08-14T21:53:54.8264606Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:53:54.8269985Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_view_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:53:54.826414]
2025-08-14T21:54:26.8128228Z 
2025-08-14T21:54:26.8135678Z distributed/tensor/test_view_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_view_ops_1.1_3a7b5825da0481a8_.log
2025-08-14T21:54:26.8141195Z Running 7 items in this shard: test/distributed/tensor/test_view_ops.py::TestViewOps::test_complex_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOps::test_dtensor_view_op_uneven, test/distributed/tensor/test_view_ops.py::TestViewOps::test_illegal_views, test/distributed/tensor/test_view_ops.py::TestViewOps::test_squeeze_, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_groups, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_ops, test/distributed/tensor/test_view_ops.py::TestViewOps::test_view_redistribution
2025-08-14T21:54:26.8143559Z 
2025-08-14T21:54:26.8143809Z Running distributed/test_p2p_ipc 1/1 ... [2025-08-14 21:54:26.813144]
2025-08-14T21:54:26.8144270Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:54:26.8145392Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_p2p_ipc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:54:26.813698]
2025-08-14T21:54:34.3476411Z 
2025-08-14T21:54:34.3477979Z distributed/test_p2p_ipc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_p2p_ipc_1.1_e3931338d856a529_.log
2025-08-14T21:54:34.3479870Z Running 1 items in this shard: test/distributed/test_p2p_ipc.py::P2PIpcTest::test_p2p_ipc
2025-08-14T21:54:34.3480628Z 
2025-08-14T21:54:34.3483892Z Running distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 ... [2025-08-14 21:54:34.347843]
2025-08-14T21:54:34.3484977Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:54:34.3489949Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_ignore_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:54:34.348378]
2025-08-14T21:54:45.5404786Z 
2025-08-14T21:54:45.5407548Z distributed/_composable/fsdp/test_fully_shard_ignore_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_ignore_params_1.1_2947eaf153e41628_.log
2025-08-14T21:54:45.5411171Z Running 1 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_ignore_params.py::TestFullyShardIgnoreParams::test_ddp_A_fsdp_B_ddp_C
2025-08-14T21:54:45.5412307Z 
2025-08-14T21:54:45.5412846Z Running distributed/checkpoint/test_compatibility 1/1 ... [2025-08-14 21:54:45.540804]
2025-08-14T21:54:45.5413752Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:54:45.5420241Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_compatibility.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:54:45.541380]
2025-08-14T21:54:49.4168325Z 
2025-08-14T21:54:49.4170378Z distributed/checkpoint/test_compatibility 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_compatibility_1.1_eb331a48349301e6_.log
2025-08-14T21:54:49.4174764Z Running 4 items in this shard: test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_metadata, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_sharded_tensor_dependency, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_storage_meta, test/distributed/checkpoint/test_compatibility.py::TestDCPCompatbility::test_with_v_2_3
2025-08-14T21:54:49.4177798Z 
2025-08-14T21:54:49.4178471Z Running distributed/tensor/parallel/test_tp_examples 1/1 ... [2025-08-14 21:54:49.417039]
2025-08-14T21:54:49.4179577Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:54:49.4183103Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_examples.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:54:49.417581]
2025-08-14T21:56:25.9094724Z 
2025-08-14T21:56:25.9101430Z distributed/tensor/parallel/test_tp_examples 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_examples_1.1_500ec6facf9b75c7_.log
2025-08-14T21:56:25.9120399Z Running 16 items in this shard: test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_loss_parallel, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_inference, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_False_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_mlp_training_is_seq_parallel_True_recompute_activation_False, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_float64_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_all, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_0_attention_wv__layers_0_feed_forward_w1__layers_1_feed_forward_w2__layers_1_ffn_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_layers_1_ffn_norm__norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_norm__output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_req_grad_seq_parallel_float32_thaw_output__tok_embeddings, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_False_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float32, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_transformer_training_is_seq_parallel_True_float64, test/distributed/tensor/parallel/test_tp_examples.py::DistTensorParallelExampleTest::test_weight_tying
2025-08-14T21:56:25.9138715Z 
2025-08-14T21:56:25.9139315Z Running distributed/_composable/fsdp/test_fully_shard_logging 1/1 ... [2025-08-14 21:56:25.909789]
2025-08-14T21:56:25.9140282Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:56:25.9142286Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_logging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:56:25.910332]
2025-08-14T21:56:29.4834916Z 
2025-08-14T21:56:29.4837026Z distributed/_composable/fsdp/test_fully_shard_logging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_logging_1.1_dc60f84dea13040f_.log
2025-08-14T21:56:29.4838926Z Running 0 items in this shard:
2025-08-14T21:56:29.4839285Z 
2025-08-14T21:56:29.4843623Z Running distributed/_composable/test_composability/test_2d_composability 2/2 ... [2025-08-14 21:56:29.483841]
2025-08-14T21:56:29.4844845Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:56:29.4855539Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=2', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:56:29.484408]
2025-08-14T21:58:00.3692983Z 
2025-08-14T21:58:00.3695233Z distributed/_composable/test_composability/test_2d_composability 2/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_2.2_d59ae37c10c4eb08_.log
2025-08-14T21:58:00.3703983Z Running 9 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_mlp, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer_checkpoint_resume, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DStateDict::test_fully_shard_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::Test2dFSDP1ParallelIntegration::test_2d_ddp_integration_functionality, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_default, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_not_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_fsdp_state_enable_extension, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp1_tp_2d_set_full_state_dict, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_fsdp_2d_extension
2025-08-14T21:58:00.3713300Z 
2025-08-14T21:58:00.3713769Z Running distributed/tensor/test_op_strategy 1/1 ... [2025-08-14 21:58:00.369383]
2025-08-14T21:58:00.3714609Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:58:00.3716494Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_op_strategy.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:58:00.369996]
2025-08-14T21:58:53.8036528Z 
2025-08-14T21:58:53.8038270Z distributed/tensor/test_op_strategy 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_op_strategy_1.1_77da81ae31b03315_.log
2025-08-14T21:58:53.8056029Z Running 20 items in this shard: test/distributed/tensor/test_op_strategy.py::TestEinsumDims::test_batch_dims, test/distributed/tensor/test_op_strategy.py::TestEinsumDims::test_bmm_dims, test/distributed/tensor/test_op_strategy.py::TestEinsumDims::test_free_dims, test/distributed/tensor/test_op_strategy.py::TestEinsumDims::test_mm_dims, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_1d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_2d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_diffinndim_2d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_bmm_diffoutndim_2d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_linearity_1d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_1d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_mm_2d_mesh, test/distributed/tensor/test_op_strategy.py::TestEinsumStrategies::test_pointwise_1d_mesh, test/distributed/tensor/test_op_strategy.py::TestCostModel::test_bmm_strategies, test/distributed/tensor/test_op_strategy.py::TestCostModel::test_mm_strategies, test/distributed/tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_latency, test/distributed/tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_1d, test/distributed/tensor/test_op_strategy.py::TestCostModel::test_redistribute_cost_mesh_2d, test/distributed/tensor/test_op_strategy.py::DistTensorReplicateStrategyRegistrationTest::test_replicate_strategy_placement, test/distributed/tensor/test_op_strategy.py::DistTensorReplicateStrategyRegistrationTest::test_tuple_replicate_strategy_placement, test/distributed/tensor/test_op_strategy.py::TestStrategyHashing::test_call_with_different_nontensor_args
2025-08-14T21:58:53.8069582Z 
2025-08-14T21:58:53.8070227Z Running distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 ... [2025-08-14 21:58:53.804003]
2025-08-14T21:58:53.8071240Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:58:53.8073318Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:58:53.804609]
2025-08-14T21:59:07.2011230Z 
2025-08-14T21:59:07.2013616Z distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_ 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_clip_grad_norm__1.1_ce08b99b4d5a94ef_.log
2025-08-14T21:59:07.2017464Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize2::test_clip_grad_norm_1d, test/distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_.py::TestClipGradNormWorldSize4::test_clip_grad_norm_2d
2025-08-14T21:59:07.2019388Z 
2025-08-14T21:59:07.2019907Z Running distributed/checkpoint/test_pg_transport 1/1 ... [2025-08-14 21:59:07.201343]
2025-08-14T21:59:07.2020806Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:07.2026256Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_pg_transport.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:07.201894]
2025-08-14T21:59:23.5548567Z 
2025-08-14T21:59:23.5551219Z distributed/checkpoint/test_pg_transport 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_pg_transport_1.1_997e45675c32af91_.log
2025-08-14T21:59:23.5569741Z Running 21 items in this shard: test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport, test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport_with_mixed_content, test/distributed/checkpoint/test_pg_transport.py::PgTransportCPU::test_pg_transport_with_sharded_tensor, test/distributed/checkpoint/test_pg_transport.py::PgTransportCUDA::test_pg_transport, test/distributed/checkpoint/test_pg_transport.py::PgTransportCUDA::test_pg_transport_with_mixed_content, test/distributed/checkpoint/test_pg_transport.py::PgTransportCUDA::test_pg_transport_with_sharded_tensor, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_different_dtypes, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_with_offset, test/distributed/checkpoint/test_pg_transport.py::TestCastTensor::test_cast_tensor_with_stride, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_basic, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_different_shapes, test/distributed/checkpoint/test_pg_transport.py::TestPrepareTensor::test_prepare_tensor_with_stride, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_basic, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_nested, test/distributed/checkpoint/test_pg_transport.py::TestPrepareStateDict::test_prepare_state_dict_with_non_tensor_values, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_recv_checkpoint_basic, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_recv_checkpoint_with_state_dict_callback, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_basic, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_empty_state_dict, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportMocked::test_send_checkpoint_with_non_tensor_values, test/distributed/checkpoint/test_pg_transport.py::TestPGTransportEdgeCases::test_send_checkpoint_with_cpu_tensors
2025-08-14T21:59:23.5585894Z 
2025-08-14T21:59:23.5586419Z Running distributed/tensor/debug/test_comm_mode 1/1 ... [2025-08-14 21:59:23.555219]
2025-08-14T21:59:23.5587294Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:23.5589241Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/debug/test_comm_mode.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:23.555781]
2025-08-14T21:59:28.3332312Z 
2025-08-14T21:59:28.3334292Z distributed/tensor/debug/test_comm_mode 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.debug.test_comm_mode_1.1_706de68d0b427ee0_.log
2025-08-14T21:59:28.3338265Z Running 4 items in this shard: test/distributed/tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode, test/distributed/tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_coalesced, test/distributed/tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_c10d, test/distributed/tensor/debug/test_comm_mode.py::TestCommMode::test_comm_mode_with_dtensor
2025-08-14T21:59:28.3340935Z 
2025-08-14T21:59:28.3341424Z Running distributed/fsdp/test_fsdp_pure_fp16 1/1 ... [2025-08-14 21:59:28.333457]
2025-08-14T21:59:28.3342272Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:28.3345675Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_pure_fp16.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:28.334009]
2025-08-14T21:59:49.5501111Z 
2025-08-14T21:59:49.5502801Z distributed/fsdp/test_fsdp_pure_fp16 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_pure_fp16_1.1_9c38bd02eec94efe_.log
2025-08-14T21:59:49.5505574Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_fp16_dtypes_cuda, test/distributed/fsdp/test_fsdp_pure_fp16.py::TestPureFP16CUDA::test_pure_fp16_training_cuda
2025-08-14T21:59:49.5507087Z 
2025-08-14T21:59:49.5509073Z Running distributed/fsdp/test_fsdp_state_dict 1/2 ... [2025-08-14 21:59:49.550456]
2025-08-14T21:59:49.5509940Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T21:59:49.5517359Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_state_dict.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:59:49.551078]
2025-08-14T22:08:08.5478021Z 
2025-08-14T22:08:08.5480085Z distributed/fsdp/test_fsdp_state_dict 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_state_dict_1.2_8ea5c55937a04ce4_.log
2025-08-14T22:08:08.5585742Z Running 78 items in this shard: test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_False_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_basic_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_fp16_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_local_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_sharded_state_dict_cpu_offload1_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_False_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_False_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload0_mixed_precision_True_state_dict_rank0_and_offload_True_use_orig_params_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_buffers_save_and_load_state_dict_state_dict_type_state_dict_cpu_offload1_mixed_precision_False_state_dict_rank0_and_offload_True_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_local_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_keys_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_sharded_state_dict_checkpoint_wrap_source_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_after_wrap_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_both_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_dest_rank0_only_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_after_wrap_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_fsdp_state_dict_with_activation_checkpoint_state_dict_type_state_dict_checkpoint_wrap_source_rank0_only_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_full_state_dict_missing_unexpected_keys_cleaned, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_local_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_False_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_sharded_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_save_and_load_after_forward_state_dict_state_dict_type_state_dict_mixed_precision_True_state_dict_rank0_and_offload_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_sharded_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_False_fsdp_root_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_load_into_local_module_state_dict_type_state_dict_state_dict_rank0_and_offload_True_fsdp_root_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_rank0_offload_save_load_flow_use_orig_params_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_save_load_flow_state_dict_type_sharded_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_type, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_False_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_False_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_sharded_state_dict_prefix_True_ignore_inner_True_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_False_mixed_precision_False, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_False_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_ignored_modules_state_dict_type_state_dict_prefix_True_ignore_inner_True_mixed_precision_True, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_state_dict_with_shared_parameters_state_dict_type_state_dict, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict::test_torch_save_load, test/distributed/fsdp/test_fsdp_state_dict.py::TestFSDPStateDict4GPUs::test_local_state_dict_reshard
2025-08-14T22:08:08.5686110Z 
2025-08-14T22:08:08.5686506Z Running distributed/test_fake_pg 1/1 ... [2025-08-14 22:08:08.550407]
2025-08-14T22:08:08.5687316Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:08:08.5689426Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_fake_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:08:08.550967]
2025-08-14T22:08:14.1304558Z 
2025-08-14T22:08:14.1305992Z distributed/test_fake_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_fake_pg_1.1_45c675f0bef0e7d4_.log
2025-08-14T22:08:14.1315954Z Running 13 items in this shard: test/distributed/test_fake_pg.py::TestFakePG::test_all_reduce, test/distributed/test_fake_pg.py::TestFakePG::test_allgather, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall, test/distributed/test_fake_pg.py::TestFakePG::test_alltoall_base, test/distributed/test_fake_pg.py::TestFakePG::test_broadcast, test/distributed/test_fake_pg.py::TestFakePG::test_construct_fsdp, test/distributed/test_fake_pg.py::TestFakePG::test_fake_pg_tracing, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_fsdp_tp_fake_e2e, test/distributed/test_fake_pg.py::TestFakePG::test_recv, test/distributed/test_fake_pg.py::TestFakePG::test_reduce_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_scatter, test/distributed/test_fake_pg.py::TestFakePG::test_send
2025-08-14T22:08:14.1321717Z 
2025-08-14T22:08:14.1322238Z Running distributed/fsdp/test_fsdp_backward_prefetch 1/1 ... [2025-08-14 22:08:14.130696]
2025-08-14T22:08:14.1323141Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:08:14.1325080Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_backward_prefetch.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:08:14.131270]
2025-08-14T22:08:25.1744914Z 
2025-08-14T22:08:25.1746656Z distributed/fsdp/test_fsdp_backward_prefetch 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_backward_prefetch_1.1_7f02237d5041db2f_.log
2025-08-14T22:08:25.1749063Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_backward_prefetch.py::TestBackwardPrefetch::test_backward_prefetch
2025-08-14T22:08:25.1750051Z 
2025-08-14T22:08:25.1753636Z Running distributed/_composable/fsdp/test_fully_shard_autograd 1/1 ... [2025-08-14 22:08:25.174785]
2025-08-14T22:08:25.1754696Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:08:25.1760192Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_autograd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:08:25.175357]
2025-08-14T22:09:06.3370777Z 
2025-08-14T22:09:06.3373589Z distributed/_composable/fsdp/test_fully_shard_autograd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_autograd_1.1_e5d0d9b815a71e08_.log
2025-08-14T22:09:06.3380015Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_nontensor_activations, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_module, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardAutograd::test_unused_forward_output, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiThread::test_post_acc_grad_hook_runs, test/distributed/_composable/fsdp/test_fully_shard_autograd.py::TestFullyShardPostAccGradHookMultiProcess::test_post_acc_grad_hook_optim_parity
2025-08-14T22:09:06.3385688Z 
2025-08-14T22:09:06.3386323Z Running distributed/_composable/fsdp/test_fully_shard_frozen 1/1 ... [2025-08-14 22:09:06.337465]
2025-08-14T22:09:06.3387273Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:09:06.3389313Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_frozen.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:09:06.338013]
2025-08-14T22:10:00.9491724Z 
2025-08-14T22:10:00.9493930Z distributed/_composable/fsdp/test_fully_shard_frozen 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_frozen_1.1_2aeb577731720d68_.log
2025-08-14T22:10:00.9498318Z Running 3 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_multi_forward_mixed_requires_grad, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_across_groups, test/distributed/_composable/fsdp/test_fully_shard_frozen.py::TestFullyShardFrozen::test_train_mixed_requires_grad_per_group
2025-08-14T22:10:00.9501388Z 
2025-08-14T22:10:00.9502031Z Running distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 ... [2025-08-14 22:10:00.949313]
2025-08-14T22:10:00.9503893Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:10:00.9506007Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:00.949874]
2025-08-14T22:10:53.1894287Z 
2025-08-14T22:10:53.1896212Z distributed/_composable/fsdp/test_fully_shard_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_mixed_precision_1.1_a44e6c9a46ec78cd_.log
2025-08-14T22:10:53.1909700Z Running 9 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_compute_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_grad_acc_with_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionTraining::test_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_clamp_reduce_dtype, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_dataclass_input, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_float16_on_one_submodule, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_bf16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_norm_modules_fp16, test/distributed/_composable/fsdp/test_fully_shard_mixed_precision.py::TestFullyShardMixedPrecisionCasts::test_submodules_with_external_inputs
2025-08-14T22:10:53.1919214Z 
2025-08-14T22:10:53.1919754Z Running distributed/test_serialization 1/1 ... [2025-08-14 22:10:53.189602]
2025-08-14T22:10:53.1920664Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:10:53.1922543Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_serialization.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:53.190248]
2025-08-14T22:10:57.2163882Z 
2025-08-14T22:10:57.2165702Z distributed/test_serialization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_serialization_1.1_c525ed3e4f948c7c_.log
2025-08-14T22:10:57.2172896Z Running 10 items in this shard: test/distributed/test_serialization.py::TestSerialization::test_cuda, test/distributed/test_serialization.py::TestSerialization::test_dtensor, test/distributed/test_serialization.py::TestSerialization::test_nested_tensors, test/distributed/test_serialization.py::TestSerialization::test_python_object, test/distributed/test_serialization.py::TestSerialization::test_scalar_tensor, test/distributed/test_serialization.py::TestSerialization::test_str_utf8, test/distributed/test_serialization.py::TestSerialization::test_strided_tensor, test/distributed/test_serialization.py::TestSerialization::test_tensor_with_offset, test/distributed/test_serialization.py::TestSerialization::test_various_data_types, test/distributed/test_serialization.py::TestSerialization::test_weights_only
2025-08-14T22:10:57.2178820Z 
2025-08-14T22:10:57.2179315Z Running distributed/checkpoint/test_state_dict 1/1 ... [2025-08-14 22:10:57.216715]
2025-08-14T22:10:57.2180219Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:10:57.2182155Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:57.217297]
2025-08-14T22:14:13.8637733Z 
2025-08-14T22:14:13.8645377Z distributed/checkpoint/test_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_1.1_f86157b5f9964a1e_.log
2025-08-14T22:14:13.8663668Z Running 25 items in this shard: test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_activation_ckpt_fqns_fsdp1, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_broadcast_from_rank0, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_compiled_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_cpu_offload_full_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_deprecate_api, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_extra_state, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_flattened_osd, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp2, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_ddp, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_fsdp_root_not_initialized, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_multi_device_load_model_state_dict, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_multi_param_groups, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_non_persistent_buffers, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_optim_state_dict_param_matching, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_set_cpu_model_state_dict_broadcast_from_rank0, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_setting_meta_device_model_broadcasting_and_memory, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_shared_weight, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_single_gpu, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_state_dict_with_hook_on_keys, test/distributed/checkpoint/test_state_dict.py::TestStateDict::test_strict, test/distributed/checkpoint/test_state_dict.py::TestNoComm::test_no_dist
2025-08-14T22:14:13.8680603Z 
2025-08-14T22:14:13.8681072Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T22:14:13.8681907Z Uploading artifacts took 0.00 seconds
2025-08-14T22:14:13.8689420Z Running distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 ... [2025-08-14 22:14:13.868373]
2025-08-14T22:14:13.8690496Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:14:13.8696145Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:14:13.868945]
2025-08-14T22:14:57.2354896Z 
2025-08-14T22:14:57.2357107Z distributed/checkpoint/test_file_system_checkpoint_cpu 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_cpu_1.1_6b1c6028d17caa09_.log
2025-08-14T22:14:57.2379055Z Running 16 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadRot13::test_read_write_tensor_and_blob_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadZStandard::test_read_write_only_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_save_load_bytes_thread_count_2, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_1, test/distributed/checkpoint/test_file_system_checkpoint_cpu.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor_thread_count_2
2025-08-14T22:14:57.2396980Z 
2025-08-14T22:14:57.2397593Z Running distributed/_composable/fsdp/test_fully_shard_extensions 1/1 ... [2025-08-14 22:14:57.235976]
2025-08-14T22:14:57.2398590Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:14:57.2400666Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_extensions.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:14:57.236520]
2025-08-14T22:15:36.1951611Z 
2025-08-14T22:15:36.1954024Z distributed/_composable/fsdp/test_fully_shard_extensions 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_extensions_1.1_fa3dde28938cdd5a_.log
2025-08-14T22:15:36.1961473Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiProcess::test_all_gather_extensions_train_parity, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_hsdp_mesh, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extension_outer_size_stride, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_end_to_end, test/distributed/_composable/fsdp/test_fully_shard_extensions.py::TestFullyShardAllGatherExtensionsMultiThread::test_all_gather_extensions_monkey_patch
2025-08-14T22:15:36.1967548Z 
2025-08-14T22:15:36.1968081Z Running distributed/fsdp/test_fsdp_freezing_weights 1/1 ... [2025-08-14 22:15:36.195401]
2025-08-14T22:15:36.1968971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:15:36.1970918Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_freezing_weights.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:15:36.195968]
2025-08-14T22:19:51.2534357Z 
2025-08-14T22:19:51.2537696Z distributed/fsdp/test_fsdp_freezing_weights 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_freezing_weights_1.1_c35b22be41d1ff78_.log
2025-08-14T22:19:51.2595471Z Running 32 items in this shard: test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_False_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_GradToNone_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_False_disable_autograd_True_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_False_forward_prefetch_True, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_False, test/distributed/fsdp/test_fsdp_freezing_weights.py::TestFreezingWeights::test_freezing_weights_with_nested_trunk_True_freezing_method_FreezingMethod_RequiresGrad_freeze_after_wrap_fsdp_True_disable_autograd_True_forward_prefetch_True
2025-08-14T22:19:51.2650062Z 
2025-08-14T22:19:51.2650526Z Running distributed/test_distributed_spawn 3/8 ... [2025-08-14 22:19:51.254389]
2025-08-14T22:19:51.2651470Z MPI not available -- MPI backend tests will be skipped
2025-08-14T22:19:51.2652335Z Running distributed tests for the test backend with env init_method
2025-08-14T22:19:51.2653439Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:19:51.2655456Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:19:51.258211]
2025-08-14T22:19:54.9263790Z 
2025-08-14T22:19:54.9265384Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_5627778cdc937f28_.log
2025-08-14T22:19:54.9266946Z Running 0 items in this shard:
2025-08-14T22:19:54.9267286Z 
2025-08-14T22:19:54.9278279Z Running distributed tests for the test backend with file init_method
2025-08-14T22:19:54.9280825Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:19:54.9287466Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:19:54.928097]
2025-08-14T22:19:58.5404435Z 
2025-08-14T22:19:58.5406155Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_f55c69c8f7cae77d_.log
2025-08-14T22:19:58.5407840Z Running 0 items in this shard:
2025-08-14T22:19:58.5408243Z 
2025-08-14T22:19:58.5412984Z Running distributed tests for the nccl backend with env init_method
2025-08-14T22:19:58.5413856Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:19:58.5417609Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:19:58.541202]
2025-08-14T22:24:23.7861726Z 
2025-08-14T22:24:23.7864657Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_5e6696dc7ccb23b9_.log
2025-08-14T22:24:23.7893193Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:24:23.7920872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2025-08-14T22:24:23.7922761Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-08-14T22:24:23.7924867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-08-14T22:24:23.7926723Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min
2025-08-14T22:24:23.7928713Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2025-08-14T22:24:23.7930362Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-08-14T22:24:23.7931967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2025-08-14T22:24:23.7933683Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split
2025-08-14T22:24:23.7935584Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda
2025-08-14T22:24:23.7937550Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group
2025-08-14T22:24:23.7939467Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-08-14T22:24:23.7941433Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group
2025-08-14T22:24:23.7943261Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2025-08-14T22:24:23.7945193Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-08-14T22:24:23.7946808Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group
2025-08-14T22:24:23.7948656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2025-08-14T22:24:23.7950726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
2025-08-14T22:24:23.7952766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-08-14T22:24:23.7954565Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-08-14T22:24:23.7956352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-08-14T22:24:23.7958125Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu
2025-08-14T22:24:23.7959770Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd
2025-08-14T22:24:23.7961477Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2025-08-14T22:24:23.7963157Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2025-08-14T22:24:23.7964817Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2025-08-14T22:24:23.7966851Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2025-08-14T22:24:23.7968917Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks
2025-08-14T22:24:23.7971047Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather
2025-08-14T22:24:23.7972692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups
2025-08-14T22:24:23.7974330Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product
2025-08-14T22:24:23.7976039Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda
2025-08-14T22:24:23.7977854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-08-14T22:24:23.7979746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl
2025-08-14T22:24:23.7981590Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2025-08-14T22:24:23.7983379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:24:23.7984427Z 
2025-08-14T22:24:23.7984819Z Running distributed tests for the nccl backend with file init_method
2025-08-14T22:24:23.7985579Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:24:23.7987569Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:24:23.789095]
2025-08-14T22:28:49.5309834Z 
2025-08-14T22:28:49.5311765Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_eaefc04a9abec3aa_.log
2025-08-14T22:28:49.5338536Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:28:49.5363344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2025-08-14T22:28:49.5365634Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-08-14T22:28:49.5367489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-08-14T22:28:49.5369272Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min
2025-08-14T22:28:49.5370994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2025-08-14T22:28:49.5372657Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-08-14T22:28:49.5374559Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2025-08-14T22:28:49.5376601Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split
2025-08-14T22:28:49.5377967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda
2025-08-14T22:28:49.5379165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group
2025-08-14T22:28:49.5380353Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-08-14T22:28:49.5381551Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group
2025-08-14T22:28:49.5382664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2025-08-14T22:28:49.5383788Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-08-14T22:28:49.5384944Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group
2025-08-14T22:28:49.5386032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2025-08-14T22:28:49.5386981Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
2025-08-14T22:28:49.5388024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-08-14T22:28:49.5388951Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-08-14T22:28:49.5389861Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-08-14T22:28:49.5390781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu
2025-08-14T22:28:49.5391628Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd
2025-08-14T22:28:49.5392511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2025-08-14T22:28:49.5393370Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2025-08-14T22:28:49.5394228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2025-08-14T22:28:49.5395128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2025-08-14T22:28:49.5396188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks
2025-08-14T22:28:49.5397084Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather
2025-08-14T22:28:49.5397940Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups
2025-08-14T22:28:49.5398785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product
2025-08-14T22:28:49.5399672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda
2025-08-14T22:28:49.5400532Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-08-14T22:28:49.5401359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl
2025-08-14T22:28:49.5402171Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2025-08-14T22:28:49.5403077Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:28:49.5403627Z 
2025-08-14T22:28:49.5403833Z Running distributed tests for the gloo backend with env init_method
2025-08-14T22:28:49.5404227Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:28:49.5405231Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:28:49.534035]
2025-08-14T22:32:59.3824350Z 
2025-08-14T22:32:59.3825438Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_5d3ddfa23cf9e19a_.log
2025-08-14T22:32:59.3839241Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:32:59.3852571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2025-08-14T22:32:59.3853470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-08-14T22:32:59.3854363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-08-14T22:32:59.3855522Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min
2025-08-14T22:32:59.3856382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2025-08-14T22:32:59.3857215Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-08-14T22:32:59.3858011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2025-08-14T22:32:59.3858862Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split
2025-08-14T22:32:59.3859796Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda
2025-08-14T22:32:59.3860755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group
2025-08-14T22:32:59.3861712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-08-14T22:32:59.3862686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group
2025-08-14T22:32:59.3863585Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2025-08-14T22:32:59.3864373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-08-14T22:32:59.3865164Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group
2025-08-14T22:32:59.3866118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2025-08-14T22:32:59.3867032Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
2025-08-14T22:32:59.3868035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-08-14T22:32:59.3868919Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-08-14T22:32:59.3869792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-08-14T22:32:59.3870663Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu
2025-08-14T22:32:59.3871476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd
2025-08-14T22:32:59.3872325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2025-08-14T22:32:59.3873146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2025-08-14T22:32:59.3873963Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2025-08-14T22:32:59.3874824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2025-08-14T22:32:59.3875673Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks
2025-08-14T22:32:59.3876531Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather
2025-08-14T22:32:59.3877479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups
2025-08-14T22:32:59.3878299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product
2025-08-14T22:32:59.3879294Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda
2025-08-14T22:32:59.3880261Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-08-14T22:32:59.3881022Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl
2025-08-14T22:32:59.3881670Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2025-08-14T22:32:59.3882394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:32:59.3882827Z 
2025-08-14T22:32:59.3883002Z Running distributed tests for the gloo backend with file init_method
2025-08-14T22:32:59.3883325Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:32:59.3884141Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=3', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:32:59.385979]
2025-08-14T22:37:09.5859347Z 
2025-08-14T22:37:09.5861410Z distributed/test_distributed_spawn 3/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_3.8_f566825a104f1fb9_.log
2025-08-14T22:37:09.5889931Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:37:09.5917188Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_full_group
2025-08-14T22:37:09.5919065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_object_default_pg
2025-08-14T22:37:09.5920834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_max
2025-08-14T22:37:09.5922873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_min
2025-08-14T22:37:09.5924629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_full_group_sum
2025-08-14T22:37:09.5926401Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_max
2025-08-14T22:37:09.5928244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_complex
2025-08-14T22:37:09.5930063Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split
2025-08-14T22:37:09.5931978Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group_cuda
2025-08-14T22:37:09.5933958Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group
2025-08-14T22:37:09.5935915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda_complex
2025-08-14T22:37:09.5937904Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group
2025-08-14T22:37:09.5939795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_err
2025-08-14T22:37:09.5941733Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast
2025-08-14T22:37:09.5943421Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_full_group
2025-08-14T22:37:09.5945060Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_group
2025-08-14T22:37:09.5946915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_grad_as_bucket_view_false
2025-08-14T22:37:09.5949372Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_apply_optim_in_backward_ignored_params
2025-08-14T22:37:09.5951426Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_broadcast_buffer
2025-08-14T22:37:09.5953243Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce_process_group
2025-08-14T22:37:09.5955037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_logging_data_gpu
2025-08-14T22:37:09.5956710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_new_tensor_in_fwd
2025-08-14T22:37:09.5958446Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_torch_profiler
2025-08-14T22:37:09.5960146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sink_noclone
2025-08-14T22:37:09.5961818Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_bn_training_vs_eval
2025-08-14T22:37:09.5963575Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_input_exception
2025-08-14T22:37:09.5965320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_different_graph_across_ranks
2025-08-14T22:37:09.5967516Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_nccl_backend_bool_allgather
2025-08-14T22:37:09.5969806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups
2025-08-14T22:37:09.5971491Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_full_group_product
2025-08-14T22:37:09.5973231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_tensor_cuda
2025-08-14T22:37:09.5975023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_object_list
2025-08-14T22:37:09.5976920Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl
2025-08-14T22:37:09.5978774Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sync_bn_logged
2025-08-14T22:37:09.5980621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_without_logger
2025-08-14T22:37:09.5981696Z 
2025-08-14T22:37:09.5982143Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T22:37:09.5982959Z Uploading artifacts took 0.00 seconds
2025-08-14T22:37:09.5983740Z Running distributed/test_distributed_spawn 6/8 ... [2025-08-14 22:37:09.589187]
2025-08-14T22:37:09.5984658Z MPI not available -- MPI backend tests will be skipped
2025-08-14T22:37:09.5998532Z Running distributed tests for the test backend with env init_method
2025-08-14T22:37:09.5999493Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:37:09.6001502Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:37:09.592892]
2025-08-14T22:37:13.2671358Z 
2025-08-14T22:37:13.2672883Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_feb895a74ba048d9_.log
2025-08-14T22:37:13.2675244Z Running 0 items in this shard:
2025-08-14T22:37:13.2675607Z 
2025-08-14T22:37:13.2680373Z Running distributed tests for the test backend with file init_method
2025-08-14T22:37:13.2681249Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:37:13.2685331Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:37:13.267938]
2025-08-14T22:37:16.8965328Z 
2025-08-14T22:37:16.8967274Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_19534b2bb7aebce2_.log
2025-08-14T22:37:16.8968879Z Running 0 items in this shard:
2025-08-14T22:37:16.8969238Z 
2025-08-14T22:37:16.8972739Z Running distributed tests for the nccl backend with env init_method
2025-08-14T22:37:16.8973771Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:37:16.8979029Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:37:16.897320]
2025-08-14T22:41:51.9673490Z 
2025-08-14T22:41:51.9677789Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_c10c8f16c16dd40a_.log
2025-08-14T22:41:51.9707182Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:41:51.9735003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last
2025-08-14T22:41:51.9737261Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-08-14T22:41:51.9739546Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-08-14T22:41:51.9741913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-08-14T22:41:51.9743700Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-08-14T22:41:51.9745387Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2025-08-14T22:41:51.9747136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum
2025-08-14T22:41:51.9749124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2025-08-14T22:41:51.9750983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group
2025-08-14T22:41:51.9752923Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-08-14T22:41:51.9754868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda
2025-08-14T22:41:51.9756592Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-08-14T22:41:51.9758195Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda
2025-08-14T22:41:51.9759854Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda
2025-08-14T22:41:51.9761506Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo
2025-08-14T22:41:51.9763521Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2025-08-14T22:41:51.9765432Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2025-08-14T22:41:51.9767960Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-08-14T22:41:51.9769763Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook
2025-08-14T22:41:51.9771498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2025-08-14T22:41:51.9773174Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple
2025-08-14T22:41:51.9775079Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2025-08-14T22:41:51.9777122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-08-14T22:41:51.9778863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states
2025-08-14T22:41:51.9780534Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2025-08-14T22:41:51.9782210Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params
2025-08-14T22:41:51.9783810Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future
2025-08-14T22:41:51.9785394Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-08-14T22:41:51.9787631Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup
2025-08-14T22:41:51.9789452Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min
2025-08-14T22:41:51.9791038Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex
2025-08-14T22:41:51.9792598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-08-14T22:41:51.9794197Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-08-14T22:41:51.9795879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:41:51.9796841Z 
2025-08-14T22:41:51.9797235Z Running distributed tests for the nccl backend with file init_method
2025-08-14T22:41:51.9798018Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:41:51.9799984Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:41:51.970619]
2025-08-14T22:46:28.2153969Z 
2025-08-14T22:46:28.2155917Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_317b6274ef72bfb1_.log
2025-08-14T22:46:28.2186956Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:46:28.2216771Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last
2025-08-14T22:46:28.2219660Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-08-14T22:46:28.2222318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-08-14T22:46:28.2224494Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-08-14T22:46:28.2226435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-08-14T22:46:28.2228264Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2025-08-14T22:46:28.2230465Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum
2025-08-14T22:46:28.2232349Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2025-08-14T22:46:28.2234316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group
2025-08-14T22:46:28.2236395Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-08-14T22:46:28.2238472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda
2025-08-14T22:46:28.2240318Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-08-14T22:46:28.2242016Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda
2025-08-14T22:46:28.2243789Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda
2025-08-14T22:46:28.2245821Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo
2025-08-14T22:46:28.2247747Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2025-08-14T22:46:28.2249650Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2025-08-14T22:46:28.2251866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-08-14T22:46:28.2253802Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook
2025-08-14T22:46:28.2255643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2025-08-14T22:46:28.2257418Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple
2025-08-14T22:46:28.2259443Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2025-08-14T22:46:28.2261606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-08-14T22:46:28.2263474Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states
2025-08-14T22:46:28.2265386Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2025-08-14T22:46:28.2267502Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params
2025-08-14T22:46:28.2269331Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future
2025-08-14T22:46:28.2271023Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-08-14T22:46:28.2272879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup
2025-08-14T22:46:28.2274692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min
2025-08-14T22:46:28.2276403Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex
2025-08-14T22:46:28.2278431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-08-14T22:46:28.2280128Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-08-14T22:46:28.2281927Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:46:28.2282960Z 
2025-08-14T22:46:28.2283368Z Running distributed tests for the gloo backend with env init_method
2025-08-14T22:46:28.2284179Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:46:28.2286377Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:46:28.218851]
2025-08-14T22:51:10.5220951Z 
2025-08-14T22:51:10.5227295Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_40b955b7d63b970e_.log
2025-08-14T22:51:10.5259008Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:51:10.5288256Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last
2025-08-14T22:51:10.5291010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-08-14T22:51:10.5293453Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-08-14T22:51:10.5295625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-08-14T22:51:10.5297520Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-08-14T22:51:10.5299316Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2025-08-14T22:51:10.5301517Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum
2025-08-14T22:51:10.5303478Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2025-08-14T22:51:10.5305785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group
2025-08-14T22:51:10.5307952Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-08-14T22:51:10.5310005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda
2025-08-14T22:51:10.5311834Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-08-14T22:51:10.5313526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda
2025-08-14T22:51:10.5315299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda
2025-08-14T22:51:10.5317045Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo
2025-08-14T22:51:10.5318933Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2025-08-14T22:51:10.5320825Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2025-08-14T22:51:10.5322676Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-08-14T22:51:10.5324580Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook
2025-08-14T22:51:10.5326696Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2025-08-14T22:51:10.5329183Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple
2025-08-14T22:51:10.5331203Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2025-08-14T22:51:10.5333352Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-08-14T22:51:10.5335435Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states
2025-08-14T22:51:10.5337513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2025-08-14T22:51:10.5339515Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params
2025-08-14T22:51:10.5341244Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future
2025-08-14T22:51:10.5342922Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-08-14T22:51:10.5344902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup
2025-08-14T22:51:10.5347026Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min
2025-08-14T22:51:10.5348714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex
2025-08-14T22:51:10.5350698Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-08-14T22:51:10.5352415Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-08-14T22:51:10.5354207Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:51:10.5355228Z 
2025-08-14T22:51:10.5355667Z Running distributed tests for the gloo backend with file init_method
2025-08-14T22:51:10.5356469Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:51:10.5358553Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=6', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:51:10.525020]
2025-08-14T22:55:53.4504260Z 
2025-08-14T22:55:53.4506144Z distributed/test_distributed_spawn 6/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_6.8_aadf75bf1f7937af_.log
2025-08-14T22:55:53.4568109Z Running 34 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:55:53.4595647Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Channels_Last
2025-08-14T22:55:53.4597972Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient
2025-08-14T22:55:53.4600625Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process
2025-08-14T22:55:53.4602662Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedSampler_padding
2025-08-14T22:55:53.4604410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_SyncBatchNorm_process_group
2025-08-14T22:55:53.4606447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_v_cuda
2025-08-14T22:55:53.4608176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_sum
2025-08-14T22:55:53.4610283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_product
2025-08-14T22:55:53.4612135Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_full_group
2025-08-14T22:55:53.4614428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_complex
2025-08-14T22:55:53.4616571Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group_cuda
2025-08-14T22:55:53.4618612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier
2025-08-14T22:55:53.4620438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group_cuda
2025-08-14T22:55:53.4622136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_group_cuda
2025-08-14T22:55:53.4623795Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo
2025-08-14T22:55:53.4625630Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce_return_future
2025-08-14T22:55:53.4627757Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_comm_hook_logging
2025-08-14T22:55:53.4629593Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_same_across_ranks
2025-08-14T22:55:53.4631376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_forward_backward_hook
2025-08-14T22:55:53.4633083Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_parity_allreduce
2025-08-14T22:55:53.4635052Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_namedtuple
2025-08-14T22:55:53.4636976Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_no_set_grad_none
2025-08-14T22:55:53.4638996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_returns_tensor_with_no_grad
2025-08-14T22:55:53.4640726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_sync_module_states
2025-08-14T22:55:53.4642383Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather_object_subgroup
2025-08-14T22:55:53.4644067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_data_parallel_params
2025-08-14T22:55:53.4645823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_get_future
2025-08-14T22:55:53.4647487Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo
2025-08-14T22:55:53.4649479Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_gloo_subgroup
2025-08-14T22:55:53.4651173Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_min
2025-08-14T22:55:53.4652746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_complex
2025-08-14T22:55:53.4654308Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_group
2025-08-14T22:55:53.4656206Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source
2025-08-14T22:55:53.4658217Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_sparse_all_reduce_sum_cuda
2025-08-14T22:55:53.4659663Z 
2025-08-14T22:55:53.4660151Z Running distributed/fsdp/test_fsdp_unshard_params 1/1 ... [2025-08-14 22:55:53.453619]
2025-08-14T22:55:53.4661002Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:55:53.4662913Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_unshard_params.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:55:53.454188]
2025-08-14T22:57:22.2877106Z 
2025-08-14T22:57:22.2879212Z distributed/fsdp/test_fsdp_unshard_params 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_unshard_params_1.1_7d9d9d65500f654e_.log
2025-08-14T22:57:22.2894188Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_named_parameters_and_buffers, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_param_data, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_recurse, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_respects_reshard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_params_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_singleton_param_writeback, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_unshard_submodule, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_core, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParams::test_with_grads_none_grads, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_param_data_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsNoShard::test_unshard_params_writeback_no_shard, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_offload_to_cpu_no_shard_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_rank0_only_with_writeback_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_backward_raises, test/distributed/fsdp/test_fsdp_unshard_params.py::TestUnshardParamsErrors::test_unshard_params_from_forward_raises
2025-08-14T22:57:22.2906031Z 
2025-08-14T22:57:22.2906499Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T22:57:22.2907341Z Uploading artifacts took 0.00 seconds
2025-08-14T22:57:22.2908155Z Running distributed/test_multi_threaded_pg 1/1 ... [2025-08-14 22:57:22.288346]
2025-08-14T22:57:22.2909000Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:57:22.2910889Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_multi_threaded_pg.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:57:22.288911]
2025-08-14T22:57:26.7656592Z 
2025-08-14T22:57:26.7658124Z distributed/test_multi_threaded_pg 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_multi_threaded_pg_1.1_9911454cdd049562_.log
2025-08-14T22:57:26.7674561Z Running 22 items in this shard: test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_none, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_all_to_all_single_tensor, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_non_zero_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_collective_error_on_rank_zero, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithWrapper::test_skip, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_coalesced, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_reduce_ops, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_all_to_all, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_allgather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_assert_equal_on_rank, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_broadcast_object_list, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_bwd_sees_fwd_pg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_gather, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_reduce_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_scatter, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_subpg, test/distributed/test_multi_threaded_pg.py::TestCollectivesWithBaseClass::test_using_pg_from_another_thread
2025-08-14T22:57:26.7691014Z 
2025-08-14T22:57:26.7691525Z Running distributed/fsdp/test_fsdp_tp_integration 1/1 ... [2025-08-14 22:57:26.765937]
2025-08-14T22:57:26.7692386Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:57:26.7694686Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_tp_integration.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:57:26.766474]
2025-08-14T22:58:06.7311503Z 
2025-08-14T22:58:06.7317971Z distributed/fsdp/test_fsdp_tp_integration 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_tp_integration_1.1_beac7762b1088ad6_.log
2025-08-14T22:58:06.7321957Z Running 3 items in this shard: test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_extension_grad, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_integration, test/distributed/fsdp/test_fsdp_tp_integration.py::TestTPFSDPIntegration::test_fsdp_tp_sync_module_state
2025-08-14T22:58:06.7324604Z 
2025-08-14T22:58:06.7325180Z Running distributed/_composable/fsdp/test_fully_shard_training 1/1 ... [2025-08-14 22:58:06.731356]
2025-08-14T22:58:06.7326327Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T22:58:06.7328424Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_training.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:58:06.731930]
2025-08-14T23:05:25.1967969Z 
2025-08-14T23:05:25.1970084Z distributed/_composable/fsdp/test_fully_shard_training 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_training_1.1_83af809021ba5f92_.log
2025-08-14T23:05:25.2177002Z Running 24 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardForwardInputs::test_root_move_forward_input_to_device, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardRegisteredParams::test_param_registration_after_forward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCastAfterInit::test_to_float64_after_init, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_explicit_prefetching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_multi_forward_module, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_non_root_forward_backward, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_post_optim_event, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_cpu_offload_eager, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_multi_group_unshard_async_op, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group_shard_dim0, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCore::test_train_parity_single_group_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShard1DTrainingCompose::test_train_parity_with_activation_checkpointing, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardShardPlacementFnMultiProcess::test_train_parity_shard_placement_fn_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardShardPlacementFnMultiThread::test_shard_placement_fn_contiguous_params_grads, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardSharedParams::test_train_parity_with_shared_params, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_1f1b_microbatching, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardGradientAccumulation::test_gradient_accumulation, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardNDTraining::test_2d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDP3DTraining::test_3d_mlp_with_nd_mesh, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardHSDPTraining::test_train_parity_hsdp, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardCustomForwardMethod::test_register_fsdp_forward_method, test/distributed/_composable/fsdp/test_fully_shard_training.py::TestFullyShardWorldSize1::test_train_parity_single_worldsize1
2025-08-14T23:05:25.2583132Z 
2025-08-14T23:05:25.2595447Z Running distributed/fsdp/test_fsdp_misc 1/1 ... [2025-08-14 23:05:25.259026]
2025-08-14T23:05:25.2600317Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:05:25.2602579Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_misc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:05:25.259366]
2025-08-14T23:06:56.2926235Z 
2025-08-14T23:06:56.2927843Z distributed/fsdp/test_fsdp_misc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_misc_1.1_a0e24f0d0d15584a_.log
2025-08-14T23:06:56.2950955Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_cpu_init_with_sync_module_states, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_init_stays_on_cpu, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_cpu_training, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_False, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_device_id_use_index_True, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_module_no_compute_grad_use_second_layer_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_not_all_outputs_used_in_loss, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optim_overlap_no_use_orig_params_error, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_optimizer_overlap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiProcess::test_fsdp_zero2_eval_with_prefetch, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_cpu_gpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_device_id_auto_wrap, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_cpu_offload, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_device_id_no_move_ignored_params_and_bufs, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_ignored_module_meta, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_namedtuple, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_same_model_across_ranks, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_fsdp_unsupported_module_cls, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_homogeneous_attributes, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_module_device_mismatches_device_id, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_multigpu_module, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscMultiThread::test_no_params, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_training_device_mismatch_errors, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_unsafe_setattr, test/distributed/fsdp/test_fsdp_misc.py::TestFSDPMiscWorldSize1::test_world_size_1_sharding_strategy_warning
2025-08-14T23:06:56.2972318Z 
2025-08-14T23:06:56.2972813Z Running distributed/fsdp/test_fsdp_clip_grad_norm 1/1 ... [2025-08-14 23:06:56.293001]
2025-08-14T23:06:56.2973679Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:06:56.2975591Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:06:56.293650]
2025-08-14T23:07:51.7204351Z 
2025-08-14T23:07:51.7206250Z distributed/fsdp/test_fsdp_clip_grad_norm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_clip_grad_norm_1.1_6e0a58a5c0b813d2_.log
2025-08-14T23:07:51.7211652Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_ddp_parity_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_low_precision_grads_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_no_gradients_cuda, test/distributed/fsdp/test_fsdp_clip_grad_norm.py::TestClipGradNormCUDA::test_non_root_cuda
2025-08-14T23:07:51.7217453Z 
2025-08-14T23:07:51.7228924Z Running distributed/_shard/sharding_spec/test_sharding_spec 1/1 ... [2025-08-14 23:07:51.722258]
2025-08-14T23:07:51.7232520Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:07:51.7234830Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_spec/test_sharding_spec.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:07:51.722836]
2025-08-14T23:08:14.1401759Z 
2025-08-14T23:08:14.1403541Z distributed/_shard/sharding_spec/test_sharding_spec 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_spec.test_sharding_spec_1.1_0746d4b04335367c_.log
2025-08-14T23:08:14.1411710Z Running 11 items in this shard: test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_check_overlapping, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_chunked_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_device_placement, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_enumerable_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunk_sharding_params, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_chunked_dim_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_get_split_size, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestShardingSpec::test_infer_sharding_spec_from_shards_metadata, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_shard_tensor, test/distributed/_shard/sharding_spec/test_sharding_spec.py::TestCustomShardingSpec::test_custom_sharding_spec_tensor_ctor
2025-08-14T23:08:14.1416161Z 
2025-08-14T23:08:14.1416443Z Running distributed/checkpoint/test_state_dict_utils 1/1 ... [2025-08-14 23:08:14.140595]
2025-08-14T23:08:14.1416911Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:08:14.1418378Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_state_dict_utils.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:08:14.141130]
2025-08-14T23:09:03.0688551Z 
2025-08-14T23:09:03.0690227Z distributed/checkpoint/test_state_dict_utils 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_state_dict_utils_1.1_c6e8d9a0de2e32bf_.log
2025-08-14T23:09:03.0698593Z Running 7 items in this shard: test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_complicated_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_cpu_offload_for_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_create_cpu_state_dict, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_state_dict_dtensor, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_gather_with_cpu_and_ranks_only, test/distributed/checkpoint/test_state_dict_utils.py::TestStateDictUtils::test_state_dict_util_distribute_tensors
2025-08-14T23:09:03.0703983Z 
2025-08-14T23:09:03.0704633Z Running distributed/algorithms/quantization/test_quantization 1/1 ... [2025-08-14 23:09:03.069346]
2025-08-14T23:09:03.0723828Z MPI not available -- MPI backend tests will be skipped
2025-08-14T23:09:03.0729283Z Running distributed tests for the test backend with env init_method
2025-08-14T23:09:03.0730519Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:09:03.0737671Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:09:03.073135]
2025-08-14T23:09:06.5587612Z 
2025-08-14T23:09:06.5590018Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_2464d043598dd5a5_.log
2025-08-14T23:09:06.5599862Z Running 0 items in this shard:
2025-08-14T23:09:06.5600218Z 
2025-08-14T23:09:06.5601733Z Running distributed tests for the test backend with file init_method
2025-08-14T23:09:06.5606689Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:09:06.5613312Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:09:06.560663]
2025-08-14T23:09:10.0353869Z 
2025-08-14T23:09:10.0355994Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_69c6af474ac297ce_.log
2025-08-14T23:09:10.0357933Z Running 0 items in this shard:
2025-08-14T23:09:10.0358278Z 
2025-08-14T23:09:10.0371528Z Running distributed tests for the nccl backend with env init_method
2025-08-14T23:09:10.0373053Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:09:10.0379670Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:09:10.037250]
2025-08-14T23:09:56.6104856Z 
2025-08-14T23:09:56.6111369Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_9ac1808c81fe384a_.log
2025-08-14T23:09:56.6119718Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:09:56.6126009Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-08-14T23:09:56.6127937Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-08-14T23:09:56.6129938Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-08-14T23:09:56.6132161Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-08-14T23:09:56.6134394Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-08-14T23:09:56.6136370Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:09:56.6137448Z 
2025-08-14T23:09:56.6137866Z Running distributed tests for the nccl backend with file init_method
2025-08-14T23:09:56.6138635Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:09:56.6140982Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:09:56.612396]
2025-08-14T23:10:43.2020595Z 
2025-08-14T23:10:43.2022596Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_ffe43d175216f0b1_.log
2025-08-14T23:10:43.2029333Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:10:43.2035410Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-08-14T23:10:43.2037343Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-08-14T23:10:43.2039280Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-08-14T23:10:43.2041222Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-08-14T23:10:43.2043832Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-08-14T23:10:43.2046257Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:10:43.2047370Z 
2025-08-14T23:10:43.2047766Z Running distributed tests for the gloo backend with env init_method
2025-08-14T23:10:43.2048526Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:10:43.2050677Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:10:43.204103]
2025-08-14T23:11:16.7672288Z 
2025-08-14T23:11:16.7678633Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_4889fdfdbb9e975f_.log
2025-08-14T23:11:16.7686078Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:11:16.7692757Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-08-14T23:11:16.7695027Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-08-14T23:11:16.7696918Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-08-14T23:11:16.7699739Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-08-14T23:11:16.7702028Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-08-14T23:11:16.7704051Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:11:16.7705125Z 
2025-08-14T23:11:16.7705524Z Running distributed tests for the gloo backend with file init_method
2025-08-14T23:11:16.7706292Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:11:16.7708617Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:11:16.769272]
2025-08-14T23:11:49.5196692Z 
2025-08-14T23:11:49.5198840Z distributed/algorithms/quantization/test_quantization 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.algorithms.quantization.test_quantization_1.1_047168910421864c_.log
2025-08-14T23:11:49.5207201Z Running 6 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16, test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:11:49.5213270Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_bfp16
2025-08-14T23:11:49.5215196Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_gather_fp16
2025-08-14T23:11:49.5217253Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_bfp16
2025-08-14T23:11:49.5219477Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_fp16
2025-08-14T23:11:49.5221831Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_bfp16
2025-08-14T23:11:49.5222968Z Running 1 items in this shard: test/distributed/algorithms/quantization/test_quantization.py::DistQuantizationTests::test_all_to_all_single_fp16
2025-08-14T23:11:49.5223523Z 
2025-08-14T23:11:49.5223774Z Running distributed/fsdp/test_fsdp_traversal 1/1 ... [2025-08-14 23:11:49.521446]
2025-08-14T23:11:49.5224220Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:11:49.5225257Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_traversal.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:11:49.522073]
2025-08-14T23:11:59.5106654Z 
2025-08-14T23:11:59.5109273Z distributed/fsdp/test_fsdp_traversal 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_traversal_1.1_b2c04fe67b12076f_.log
2025-08-14T23:11:59.5111779Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_traversal.py::TestTraversalCUDA::test_fsdp_modules_cuda
2025-08-14T23:11:59.5113994Z 
2025-08-14T23:11:59.5120506Z Running distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 ... [2025-08-14 23:11:59.510914]
2025-08-14T23:11:59.5121557Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:11:59.5123681Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_sharded_grad_scaler.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:11:59.511544]
2025-08-14T23:14:59.6996396Z 
2025-08-14T23:14:59.6998301Z distributed/fsdp/test_fsdp_sharded_grad_scaler 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_sharded_grad_scaler_1.1_5c85ee2fa080e7cf_.log
2025-08-14T23:14:59.7038876Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_grad_scaling, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_inf_gradients_skip_optim_step, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardGradScaler::test_scaling_unscaling_sparse, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_false_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_none_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_mixed_precision_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_none, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_fsdp_ddp_parity_with_grad_scaler_offload_true_shard_grad_op_none_use_orig_params, test/distributed/fsdp/test_fsdp_sharded_grad_scaler.py::TestShardedGradScalerParityWithDDP::test_sharded_grad_scaler_found_inf
2025-08-14T23:14:59.7062396Z 
2025-08-14T23:14:59.7062841Z Running distributed/test_composability 1/1 ... [2025-08-14 23:14:59.700316]
2025-08-14T23:14:59.7063667Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:14:59.7065546Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_composability.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:14:59.700883]
2025-08-14T23:15:08.0861849Z 
2025-08-14T23:15:08.0863106Z distributed/test_composability 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_composability_1.1_c90cd0da5e53f71e_.log
2025-08-14T23:15:08.0867707Z Running 11 items in this shard: test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_ddp_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_MP_ScheduleClass3, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass0, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass1, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass2, test/distributed/test_composability.py::ComposabilityTest::test_pp_fsdp_dp_type_FSDP_ScheduleClass3
2025-08-14T23:15:08.0872591Z 
2025-08-14T23:15:08.0872829Z Running distributed/fsdp/test_fsdp_core 2/3 ... [2025-08-14 23:15:08.086544]
2025-08-14T23:15:08.0873257Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:15:08.0875196Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=2', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:15:08.087103]
2025-08-14T23:22:09.1415142Z 
2025-08-14T23:22:09.1416975Z distributed/fsdp/test_fsdp_core 2/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_2.3_2ea05ad8d8f3d4d1_.log
2025-08-14T23:22:09.1444614Z Running 20 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_register_functions_called_cuda_first_False_mixed_precision_False_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_True_cuda
2025-08-14T23:22:09.1469397Z 
2025-08-14T23:22:09.1470277Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T23:22:09.1471222Z Uploading artifacts took 0.00 seconds
2025-08-14T23:22:09.1480061Z Running distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 ... [2025-08-14 23:22:09.147429]
2025-08-14T23:22:09.1481120Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:22:09.1485318Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_tensor_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:22:09.147972]
2025-08-14T23:22:44.7469469Z 
2025-08-14T23:22:44.7472717Z distributed/_shard/sharded_tensor/ops/test_tensor_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_tensor_ops_1.1_5d07f068bcc5b13c_.log
2025-08-14T23:22:44.7477788Z Running 5 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_clone, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_deep_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_detach, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_inplace_copy, test/distributed/_shard/sharded_tensor/ops/test_tensor_ops.py::TestTensorOps::test_set_requires_grad
2025-08-14T23:22:44.7481263Z 
2025-08-14T23:22:44.7481788Z Running distributed/fsdp/test_fsdp_mixed_precision 1/1 ... [2025-08-14 23:22:44.747372]
2025-08-14T23:22:44.7482666Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:22:44.7484865Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:22:44.747918]
2025-08-14T23:30:31.2674340Z 
2025-08-14T23:30:31.2680621Z distributed/fsdp/test_fsdp_mixed_precision 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_mixed_precision_1.1_76ea397adeff33b0_.log
2025-08-14T23:30:31.2752288Z Running 66 items in this shard: test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_buffer_dtype_no_root_handle, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_eval_root_cast_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_buffers, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_full_precision_in_eval_comm, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_input_grads_with_param_mixed_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mixed_precision_resnet, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_False, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_batchnorm_convert_sync_bn_True, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_default, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_only_params_and_bufs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_params_and_reduce_diff, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionSharded::test_mp_embedding_reduce, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_grads_reduced_precision, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_e2e_full_shard, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionUnsharded::test_mixed_precision_no_reshard_after_forward, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPMixedPrecisionIgnoredModules::test_mixed_precision_with_ignored_module, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_float16_on_one_submodule_skip_inputs_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_different_precisions_error, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPDifferentSubmodulePrecision::test_submodules_with_external_inputs, test/distributed/fsdp/test_fsdp_mixed_precision.py::TestFSDPTrainEval::test_train_ema_eval_flow
2025-08-14T23:30:31.2823016Z 
2025-08-14T23:30:31.2823492Z Running distributed/test_c10d_ucc 1/1 ... [2025-08-14 23:30:31.269658]
2025-08-14T23:30:31.2824265Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:30:31.2826152Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:30:31.270242]
2025-08-14T23:30:32.7187524Z 
2025-08-14T23:30:32.7189255Z distributed/test_c10d_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ucc_1.1_1110dc47fb8427d7_.log
2025-08-14T23:30:32.7191267Z 
2025-08-14T23:30:32.7194670Z Running distributed/test_c10d_common 1/1 ... [2025-08-14 23:30:32.718989]
2025-08-14T23:30:32.7195556Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:30:32.7201121Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_common.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:30:32.719481]
2025-08-14T23:34:00.4627097Z 
2025-08-14T23:34:00.4632583Z distributed/test_c10d_common 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_common_1.1_cc1be417031775f9_.log
2025-08-14T23:34:00.4651660Z Running 27 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype, test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype, test/distributed/test_c10d_common.py::CommTest::test_debug_level, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv, test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal, test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank, test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv, test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-08-14T23:34:00.4671012Z Running 1 items in this shard: test/distributed/test_c10d_common.py::TimeoutTest::test_store_based_barrier
2025-08-14T23:34:00.4672556Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_multi_dtype
2025-08-14T23:34:00.4674218Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_multi_limit_single_dtype
2025-08-14T23:34:00.4675867Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_multi_dtype
2025-08-14T23:34:00.4677538Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ComputeBucketAssignmentTest::test_single_limit_single_dtype
2025-08-14T23:34:00.4679417Z Running 1 items in this shard: test/distributed/test_c10d_common.py::CommTest::test_debug_level
2025-08-14T23:34:00.4680812Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_abort
2025-08-14T23:34:00.4682407Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_class_attr
2025-08-14T23:34:00.4684033Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_backend_config
2025-08-14T23:34:00.4685772Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_canonicalize_helper
2025-08-14T23:34:00.4687384Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_collectives
2025-08-14T23:34:00.4688968Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_get_backend_name
2025-08-14T23:34:00.4690784Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_init_process_group_with_multiple_backends
2025-08-14T23:34:00.4692637Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_is_backend_available
2025-08-14T23:34:00.4694243Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_send_recv
2025-08-14T23:34:00.4695738Z Running 1 items in this shard: test/distributed/test_c10d_common.py::PythonProcessGroupExtensionTest::test_shutdown
2025-08-14T23:34:00.4697453Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_default_process_group
2025-08-14T23:34:00.4699501Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends
2025-08-14T23:34:00.4701691Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-08-14T23:34:00.4703551Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_op_isinstance_of_reduceop
2025-08-14T23:34:00.4705343Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_copyable
2025-08-14T23:34:00.4706906Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_equal
2025-08-14T23:34:00.4708470Z Running 1 items in this shard: test/distributed/test_c10d_common.py::ReduceOpTest::test_reduceop_pickle
2025-08-14T23:34:00.4709799Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRank
2025-08-14T23:34:00.4711231Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testNodeLocalRankOverridesFallback
2025-08-14T23:34:00.4712642Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnv
2025-08-14T23:34:00.4714014Z Running 1 items in this shard: test/distributed/test_c10d_common.py::LocalRankTest::testWithoutEnvWithFallback
2025-08-14T23:34:00.4714837Z 
2025-08-14T23:34:00.4715405Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 ... [2025-08-14 23:34:00.464042]
2025-08-14T23:34:00.4716480Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:34:00.4718694Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:34:00.464647]
2025-08-14T23:34:19.1734123Z 
2025-08-14T23:34:19.1736391Z distributed/_shard/sharded_tensor/ops/test_embedding_bag 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_bag_1.1_8db6219d0432b595_.log
2025-08-14T23:34:19.1740863Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding_bag.py::TestShardedEmbeddingBag::test_sharded_embedding_bag_rowwise
2025-08-14T23:34:19.1743279Z 
2025-08-14T23:34:19.1743711Z Running distributed/test_c10d_spawn_gloo 1/1 ... [2025-08-14 23:34:19.173707]
2025-08-14T23:34:19.1744514Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:34:19.1748475Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_gloo.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:34:19.174255]
2025-08-14T23:35:50.4215158Z 
2025-08-14T23:35:50.4221338Z distributed/test_c10d_spawn_gloo 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_gloo_1.1_9c4d186c687da089_.log
2025-08-14T23:35:50.4230424Z Running 11 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda, test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce, test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter
2025-08-14T23:35:50.4238992Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cpu
2025-08-14T23:35:50.4241397Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_cuda
2025-08-14T23:35:50.4243059Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::DistributedDataParallelSingleProcessTest::test_rnn
2025-08-14T23:35:50.4244659Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_gather
2025-08-14T23:35:50.4246681Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all
2025-08-14T23:35:50.4248271Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_all_to_all_single
2025-08-14T23:35:50.4249881Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_allreduce
2025-08-14T23:35:50.4251443Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_broadcast
2025-08-14T23:35:50.4252942Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_gather
2025-08-14T23:35:50.4254426Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_reduce
2025-08-14T23:35:50.4256126Z Running 1 items in this shard: test/distributed/test_c10d_spawn_gloo.py::TestDistributedNNFunctionsGloo::test_scatter
2025-08-14T23:35:50.4257124Z 
2025-08-14T23:35:50.4257635Z Running distributed/test_c10d_spawn_nccl 1/1 ... [2025-08-14 23:35:50.421890]
2025-08-14T23:35:50.4258576Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:35:50.4261124Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:35:50.422465]
2025-08-14T23:37:30.5990655Z 
2025-08-14T23:37:30.5996659Z distributed/test_c10d_spawn_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_nccl_1.1_92c224fa3154ea78_.log
2025-08-14T23:37:30.6004864Z Running 10 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_reduce_non_contiguous, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter, test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous
2025-08-14T23:37:30.6013371Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather
2025-08-14T23:37:30.6014682Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_gather_base
2025-08-14T23:37:30.6015720Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_reduce_non_contiguous
2025-08-14T23:37:30.6016734Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all
2025-08-14T23:37:30.6017697Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_all_to_all_single
2025-08-14T23:37:30.6019201Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_allreduce
2025-08-14T23:37:30.6020180Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_broadcast
2025-08-14T23:37:30.6021807Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce
2025-08-14T23:37:30.6023475Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter
2025-08-14T23:37:30.6025195Z Running 1 items in this shard: test/distributed/test_c10d_spawn_nccl.py::TestDistributedNNFunctionsNccl::test_reduce_scatter_non_contiguous
2025-08-14T23:37:30.6026208Z 
2025-08-14T23:37:30.6026623Z Running distributed/test_launcher 1/1 ... [2025-08-14 23:37:30.599581]
2025-08-14T23:37:30.6027383Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:37:30.6029190Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_launcher.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:37:30.600127]
2025-08-14T23:37:35.7284016Z 
2025-08-14T23:37:35.7285985Z distributed/test_launcher 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_launcher_1.1_de1cc04b5a1716f5_.log
2025-08-14T23:37:35.7288127Z Running 1 items in this shard: test/distributed/test_launcher.py::TestDistributedLaunch::test_launch_user_script
2025-08-14T23:37:35.7288989Z 
2025-08-14T23:37:35.7316792Z Running distributed/test_c10d_spawn_ucc 1/1 ... [2025-08-14 23:37:35.728753]
2025-08-14T23:37:35.7317674Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:37:35.7320369Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:37:35.729327]
2025-08-14T23:38:00.4579294Z 
2025-08-14T23:38:00.4584853Z distributed/test_c10d_spawn_ucc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_spawn_ucc_1.1_b7c4c7aac4d89c9d_.log
2025-08-14T23:38:00.4590781Z Running 6 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast, test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce
2025-08-14T23:38:00.4595656Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_gather
2025-08-14T23:38:00.4597289Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all
2025-08-14T23:38:00.4598906Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_all_to_all_single
2025-08-14T23:38:00.4600508Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_allreduce
2025-08-14T23:38:00.4602036Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_broadcast
2025-08-14T23:38:00.4603518Z Running 1 items in this shard: test/distributed/test_c10d_spawn_ucc.py::TestDistributedNNFunctionsUcc::test_reduce
2025-08-14T23:38:00.4604359Z 
2025-08-14T23:38:00.4604755Z Running distributed/test_c10d_nccl 1/3 ... [2025-08-14 23:38:00.458281]
2025-08-14T23:38:00.4619257Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:38:00.4621185Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=1', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:38:00.458876]
2025-08-14T23:52:49.1176657Z 
2025-08-14T23:52:49.1178098Z distributed/test_c10d_nccl 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_1.3_906b3ba6a8265b7b_.log
2025-08-14T23:52:49.1242078Z Running 83 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_block_current_stream, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_with_idx, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq, test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error, test/distributed/test_c10d_nccl.py::NcclRegistrationTest::test_nccl_window_registration, test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config, test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group, test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend, test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False, test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-08-14T23:52:49.1300590Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::RendezvousEnvTest::test_common_errors
2025-08-14T23:52:49.1302025Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLNoGPUTest::test_init_no_gpus
2025-08-14T23:52:49.1303584Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_multi_pgs
2025-08-14T23:52:49.1305156Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_pg
2025-08-14T23:52:49.1306874Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_block_current_stream
2025-08-14T23:52:49.1308537Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_False
2025-08-14T23:52:49.1310116Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_close_pg_eager_init_True
2025-08-14T23:52:49.1311631Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_split_group
2025-08-14T23:52:49.1313233Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extend_nccl_pg_timeout_backend_nccl
2025-08-14T23:52:49.1315181Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_file_store_check
2025-08-14T23:52:49.1316777Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_process_group_nccl_timeout
2025-08-14T23:52:49.1318328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_init_with_idx
2025-08-14T23:52:49.1319782Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_bfloat16
2025-08-14T23:52:49.1321289Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float8_e5m2
2025-08-14T23:52:49.1322843Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_new_group_eager_init_True
2025-08-14T23:52:49.1324374Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_init
2025-08-14T23:52:49.1326037Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_accumulate_gradients_module
2025-08-14T23:52:49.1327734Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_bf16_compress_wrapper_is_view
2025-08-14T23:52:49.1329319Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output
2025-08-14T23:52:49.1330993Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing
2025-08-14T23:52:49.1333020Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False
2025-08-14T23:52:49.1335293Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_False
2025-08-14T23:52:49.1337578Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_with_then_hook_nccl
2025-08-14T23:52:49.1339730Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_complex_params
2025-08-14T23:52:49.1341297Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_with_lazy_parameters
2025-08-14T23:52:49.1343005Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_default_ddp_comm_hooks_nccl_is_view
2025-08-14T23:52:49.1345009Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_info
2025-08-14T23:52:49.1347167Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_debug_off
2025-08-14T23:52:49.1349085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_detail
2025-08-14T23:52:49.1351111Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_off
2025-08-14T23:52:49.1352829Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16
2025-08-14T23:52:49.1354341Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_nccl
2025-08-14T23:52:49.1355902Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_grad_is_view
2025-08-14T23:52:49.1357592Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_grad_layout_1devicemodule_1replicaperprocess
2025-08-14T23:52:49.1359687Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_2gpu_module
2025-08-14T23:52:49.1361319Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_4gpu_module
2025-08-14T23:52:49.1363099Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_device_ids_None
2025-08-14T23:52:49.1365087Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_single_device_module_empty_device_ids
2025-08-14T23:52:49.1367214Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_propagate_error_reason
2025-08-14T23:52:49.1368764Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_no_grad
2025-08-14T23:52:49.1370323Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_param_layout_mismatch_error
2025-08-14T23:52:49.1371975Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_powerSGD_ddp_comm_hook_nccl
2025-08-14T23:52:49.1373870Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_all_gather_object
2025-08-14T23:52:49.1375611Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_broadcast
2025-08-14T23:52:49.1377258Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_seq
2025-08-14T23:52:49.1378733Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::WorkHookTest::test_on_completion_hook_with_ddp
2025-08-14T23:52:49.1380085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_error_detection_and_propagation
2025-08-14T23:52:49.1381503Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_invalid_nccl_blocking_wait_env
2025-08-14T23:52:49.1382939Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_non_blocking_wait_with_barrier
2025-08-14T23:52:49.1384859Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_restart_pg_after_error
2025-08-14T23:52:49.1386402Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclRegistrationTest::test_nccl_window_registration
2025-08-14T23:52:49.1387772Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_pass_nccl_options_config
2025-08-14T23:52:49.1388981Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_reduce_scatter_tensor_coalesced
2025-08-14T23:52:49.1390248Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_default
2025-08-14T23:52:49.1391568Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_incremented_nccl_subgroup
2025-08-14T23:52:49.1392839Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_nccl_new_group
2025-08-14T23:52:49.1393995Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_wait_tensor
2025-08-14T23:52:49.1395336Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_base
2025-08-14T23:52:49.1397065Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e5m2
2025-08-14T23:52:49.1398933Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_optional_backend
2025-08-14T23:52:49.1400603Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_batch_send_recv_subgroup_group_rank_True
2025-08-14T23:52:49.1402406Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device0_group_rank_False
2025-08-14T23:52:49.1404059Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_True
2025-08-14T23:52:49.1405615Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_subgroup_group_rank_True
2025-08-14T23:52:49.1406954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_False
2025-08-14T23:52:49.1408328Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_object_list_subgroup_group_rank_False
2025-08-14T23:52:49.1409701Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_True
2025-08-14T23:52:49.1411151Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device0_group_rank_False
2025-08-14T23:52:49.1412882Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_object_list_subgroup_set_device1_group_rank_False
2025-08-14T23:52:49.1414730Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_False
2025-08-14T23:52:49.1416519Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_True_async_op_False
2025-08-14T23:52:49.1418153Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::SparseCollective::test_ddp_set_sparse_metadata
2025-08-14T23:52:49.1419518Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_False
2025-08-14T23:52:49.1420756Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_dump_pipe
2025-08-14T23:52:49.1422079Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_True
2025-08-14T23:52:49.1423648Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_long
2025-08-14T23:52:49.1425191Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_False_include_collectives_False
2025-08-14T23:52:49.1427121Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_False
2025-08-14T23:52:49.1428750Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_pickle_timing_enabled_True_include_collectives_True
2025-08-14T23:52:49.1430341Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_False
2025-08-14T23:52:49.1431936Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_True_only_active_True
2025-08-14T23:52:49.1433375Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_all_works_retired
2025-08-14T23:52:49.1434794Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLLargerScaleTest::test_comm_split_group_larger_scale
2025-08-14T23:52:49.1435660Z 
2025-08-14T23:52:49.1436054Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading
2025-08-14T23:52:49.1436775Z Uploading artifacts took 0.00 seconds
2025-08-14T23:52:49.1437459Z Running distributed/test_c10d_ops_nccl 1/1 ... [2025-08-14 23:52:49.121347]
2025-08-14T23:52:49.1438151Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:52:49.1440040Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_ops_nccl.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:52:49.121893]
2025-08-14T23:53:03.8203974Z 
2025-08-14T23:53:03.8206022Z distributed/test_c10d_ops_nccl 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_ops_nccl_1.1_7e428afb14106395_.log
2025-08-14T23:53:03.8225360Z Running 30 items in this shard: test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_all_gather_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allgather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_in_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_alltoall_ops_with_cudafree_race, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_barrier, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_broadcast_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_empty_tensors, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_gather_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_nccl_watchdog_cudagraph, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_basics, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_base_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_bfloat16, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_float8, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_reduce_scatter_v, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_checks, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_scatter_stress, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_complex, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_send_recv_object_list, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_sparse_allreduce_ops, test/distributed/test_c10d_ops_nccl.py::ProcessGroupNCCLOpTest::test_tensor_register_hook
2025-08-14T23:53:03.8244529Z 
2025-08-14T23:53:03.8245016Z Running distributed/elastic/events/lib_test 1/1 ... [2025-08-14 23:53:03.820765]
2025-08-14T23:53:03.8246011Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:53:03.8247902Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/events/lib_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:53:03.821314]
2025-08-14T23:53:07.7967594Z 
2025-08-14T23:53:07.7969752Z distributed/elastic/events/lib_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.events.lib_test_1.1_4d0f9a8845c90652_.log
2025-08-14T23:53:07.7977498Z Running 8 items in this shard: test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_created, test/distributed/elastic/events/lib_test.py::EventLibTest::test_event_deser, test/distributed/elastic/events/lib_test.py::EventLibTest::test_get_or_create_logger, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_construct_and_record_rdzv_event_does_not_run_if_invalid_dest, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_created, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_deserialize, test/distributed/elastic/events/lib_test.py::RdzvEventLibTest::test_rdzv_event_str
2025-08-14T23:53:07.7982932Z 
2025-08-14T23:53:07.7983421Z Running distributed/elastic/metrics/api_test 1/1 ... [2025-08-14 23:53:07.796949]
2025-08-14T23:53:07.7984379Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:53:07.7986503Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:53:07.797516]
2025-08-14T23:53:11.6722629Z 
2025-08-14T23:53:11.6724579Z distributed/elastic/metrics/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.metrics.api_test_1.1_5cff25b06d25b285_.log
2025-08-14T23:53:11.6728094Z Running 3 items in this shard: test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_get_metric_name, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_inheritance, test/distributed/elastic/metrics/api_test.py::MetricsApiTest::test_profile
2025-08-14T23:53:11.6730241Z 
2025-08-14T23:53:11.6730803Z Running distributed/elastic/multiprocessing/api_test 1/1 ... [2025-08-14 23:53:11.672464]
2025-08-14T23:53:11.6731726Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:53:11.6735655Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/multiprocessing/api_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:53:11.673013]
2025-08-14T23:53:44.2148350Z 
2025-08-14T23:53:44.2150081Z distributed/elastic/multiprocessing/api_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.multiprocessing.api_test_1.1_9a4bb06cc3634c93_.log
2025-08-14T23:53:44.2171746Z Running 25 items in this shard: test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_get_failures, test/distributed/elastic/multiprocessing/api_test.py::RunProcResultsTest::test_is_failed, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_str_bad_input, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value, test/distributed/elastic/multiprocessing/api_test.py::StdTest::test_from_value_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_args_env_len_mismatch, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_large_ret_val, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_raise, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_function_with_tensor, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_invalid_log_dir, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_multiprocessing_context_poll_raises_exception, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_pcontext_wait_on_a_child_thread, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_to_map, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_void_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsFuncTest::test_wait_for_all_child_procs_to_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_exit, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_incorrect_entrypoint, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_binary_raises, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_subprocess_context_close, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesAsBinaryTest::test_validate_full_rank, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsFuncTest::test_function, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary, test/distributed/elastic/multiprocessing/api_test.py::StartProcessesListAsBinaryTest::test_binary_redirect_and_tee
2025-08-14T23:53:44.2191368Z 
2025-08-14T23:53:44.2191941Z Running distributed/elastic/timer/local_timer_example 1/1 ... [2025-08-14 23:53:44.215172]
2025-08-14T23:53:44.2192861Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:53:44.2194879Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:53:44.215722]
2025-08-14T23:54:01.4708031Z 
2025-08-14T23:54:01.4710080Z distributed/elastic/timer/local_timer_example 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_example_1.1_8a6427207ca18fdf_.log
2025-08-14T23:54:01.4713216Z Running 2 items in this shard: test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_example_start_method_spawn, test/distributed/elastic/timer/local_timer_example.py::LocalTimerExample::test_torch_mp_example
2025-08-14T23:54:01.4715120Z 
2025-08-14T23:54:01.4719996Z Running distributed/elastic/timer/local_timer_test 1/1 ... [2025-08-14 23:54:01.470922]
2025-08-14T23:54:01.4721794Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:54:01.4726146Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:54:01.471521]
2025-08-14T23:54:10.5584946Z 
2025-08-14T23:54:10.5587160Z distributed/elastic/timer/local_timer_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.timer.local_timer_test_1.1_d36fa79f3fa6caa7_.log
2025-08-14T23:54:10.5599649Z Running 14 items in this shard: test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_client_interaction, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_exception_propagation, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_get_timer_recursive, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_happy_path, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_no_client, test/distributed/elastic/timer/local_timer_test.py::LocalTimerTest::test_timer, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_less_than_size, test/distributed/elastic/timer/local_timer_test.py::MultiprocessingRequestQueueTest::test_get_size, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_acquire_release, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_expired_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_valid_timers, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_call_count, test/distributed/elastic/timer/local_timer_test.py::LocalTimerServerTest::test_watchdog_empty_queue
2025-08-14T23:54:10.5610482Z 
2025-08-14T23:54:10.5611044Z Running distributed/elastic/utils/distributed_test 1/1 ... [2025-08-14 23:54:10.558769]
2025-08-14T23:54:10.5611967Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:54:10.5613943Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:54:10.559367]
2025-08-14T23:54:16.6897572Z 
2025-08-14T23:54:16.6899443Z distributed/elastic/utils/distributed_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.distributed_test_1.1_1d9857e3198e94b0_.log
2025-08-14T23:54:16.6907293Z Running 8 items in this shard: test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_no_port_multi, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_single_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_timeout_on_worker, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_create_store_with_libuv_support, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_server, test/distributed/elastic/utils/distributed_test.py::DistributedUtilTest::test_port_already_in_use_on_worker
2025-08-14T23:54:16.6913743Z 
2025-08-14T23:54:16.6914253Z Running distributed/elastic/utils/logging_test 1/1 ... [2025-08-14 23:54:16.690085]
2025-08-14T23:54:16.6915144Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:54:16.6917095Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:54:16.690723]
2025-08-14T23:54:20.5150401Z 
2025-08-14T23:54:20.5152432Z distributed/elastic/utils/logging_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.logging_test_1.1_a1646ba81c850815_.log
2025-08-14T23:54:20.5155229Z Running 2 items in this shard: test/distributed/elastic/utils/logging_test.py::LoggingTest::test_derive_module_name, test/distributed/elastic/utils/logging_test.py::LoggingTest::test_logger_name
2025-08-14T23:54:20.5156653Z 
2025-08-14T23:54:20.5157824Z Running distributed/elastic/utils/util_test 1/1 ... [2025-08-14 23:54:20.515319]
2025-08-14T23:54:20.5158861Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set
2025-08-14T23:54:20.5165648Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/elastic/utils/util_test.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:54:20.515919]
2025-08-14T23:54:24.6412207Z 
2025-08-14T23:54:24.6413972Z distributed/elastic/utils/util_test 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.elastic.utils.util_test_1.1_2720cb90467126a3_.log
2025-08-14T23:54:24.6423106Z Running 12 items in this shard: test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_hash_store, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_operations, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_barrier_timeout_rank_tracing, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_0, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_get_all_rank_n, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize, test/distributed/elastic/utils/util_test.py::StoreUtilTest::test_synchronize_hash_store, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_custom_name, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_different, test/distributed/elastic/utils/util_test.py::UtilTest::test_get_logger_none
2025-08-14T23:54:24.6430341Z 
2025-08-14T23:54:27.7789357Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.7791058Z   import pkg_resources
2025-08-14T23:54:27.8621300Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.8622839Z   import pkg_resources
2025-08-14T23:54:27.8624286Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.8625738Z   import pkg_resources
2025-08-14T23:54:27.9053179Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.9055590Z   import pkg_resources
2025-08-14T23:54:27.9101174Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.9102754Z   import pkg_resources
2025-08-14T23:54:27.9261436Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.9262978Z   import pkg_resources
2025-08-14T23:54:27.9331227Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.9332749Z   import pkg_resources
2025-08-14T23:54:27.9335791Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
2025-08-14T23:54:27.9337255Z   import pkg_resources
2025-08-14T23:54:29.5713430Z Running test batch 'tests to run' cost 9138.23 seconds
2025-08-14T23:54:30.2351205Z 
2025-08-14T23:54:30.2351735Z real	152m24.048s
2025-08-14T23:54:30.2352217Z user	358m38.064s
2025-08-14T23:54:30.2352621Z sys	449m52.994s
2025-08-14T23:54:30.2353725Z + assert_git_not_dirty
2025-08-14T23:54:30.2354299Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]]
2025-08-14T23:54:30.2354954Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]]
2025-08-14T23:54:30.2355539Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]]
2025-08-14T23:54:30.2356044Z + [[ 3 == 1 ]]
2025-08-14T23:54:30.2356418Z + [[ 3 == 1 ]]
2025-08-14T23:54:30.2356801Z + sccache_epilogue
2025-08-14T23:54:30.2357296Z + echo '::group::Sccache Compilation Log'
2025-08-14T23:54:30.2358463Z ##[group]Sccache Compilation Log
2025-08-14T23:54:30.2359092Z + echo '=================== sccache compilation log ==================='
2025-08-14T23:54:30.2359801Z =================== sccache compilation log ===================
2025-08-14T23:54:30.2360839Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log
2025-08-14T23:54:30.2645330Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ==========='
2025-08-14T23:54:30.2646891Z =========== If your build fails, please take a look at the log above for possible reasons ===========
2025-08-14T23:54:30.2647932Z + sccache --show-stats
2025-08-14T23:54:30.2707039Z Compile requests                    358
2025-08-14T23:54:30.2707802Z Compile requests executed             0
2025-08-14T23:54:30.2708365Z Cache hits                            0
2025-08-14T23:54:30.2708869Z Cache misses                          0
2025-08-14T23:54:30.2709429Z Cache hits rate                       -
2025-08-14T23:54:30.2709955Z Cache timeouts                        0
2025-08-14T23:54:30.2710465Z Cache read errors                     0
2025-08-14T23:54:30.2710984Z Forced recaches                       0
2025-08-14T23:54:30.2711497Z Cache write errors                    0
2025-08-14T23:54:30.2712017Z Cache errors                          0
2025-08-14T23:54:30.2712524Z Compilations                          0
2025-08-14T23:54:30.2713042Z Compilation failures                  0
2025-08-14T23:54:30.2713582Z Non-cacheable compilations            0
2025-08-14T23:54:30.2714148Z Non-cacheable calls                   1
2025-08-14T23:54:30.2715291Z Non-compilation calls               357
2025-08-14T23:54:30.2715821Z Unsupported compiler calls            0
2025-08-14T23:54:30.2716366Z Average cache write               0.000 s
2025-08-14T23:54:30.2717004Z Average compiler                  0.000 s
2025-08-14T23:54:30.2717558Z Average cache read hit            0.000 s
2025-08-14T23:54:30.2718111Z Failed distributed compilations       0
2025-08-14T23:54:30.2718474Z 
2025-08-14T23:54:30.2718667Z Non-cacheable reasons:
2025-08-14T23:54:30.2719133Z -E                                    1
2025-08-14T23:54:30.2719503Z 
2025-08-14T23:54:30.2719841Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-08-14T23:54:30.2720585Z Use direct/preprocessor mode?   yes
2025-08-14T23:54:30.2721134Z Version (client)                0.10.0
2025-08-14T23:54:30.2721675Z Max cache size                       10 GiB
2025-08-14T23:54:30.2722407Z + sccache --stop-server
2025-08-14T23:54:30.2778057Z Stopping sccache server...
2025-08-14T23:54:30.2786108Z Compile requests                    358
2025-08-14T23:54:30.2786817Z Compile requests executed             0
2025-08-14T23:54:30.2787408Z Cache hits                            0
2025-08-14T23:54:30.2787914Z Cache misses                          0
2025-08-14T23:54:30.2788431Z Cache hits rate                       -
2025-08-14T23:54:30.2788957Z Cache timeouts                        0
2025-08-14T23:54:30.2789458Z Cache read errors                     0
2025-08-14T23:54:30.2789976Z Forced recaches                       0
2025-08-14T23:54:30.2790522Z Cache write errors                    0
2025-08-14T23:54:30.2791025Z Cache errors                          0
2025-08-14T23:54:30.2791583Z Compilations                          0
2025-08-14T23:54:30.2792100Z Compilation failures                  0
2025-08-14T23:54:30.2792627Z Non-cacheable compilations            0
2025-08-14T23:54:30.2793708Z Non-cacheable calls                   1
2025-08-14T23:54:30.2794254Z Non-compilation calls               357
2025-08-14T23:54:30.2794796Z Unsupported compiler calls            0
2025-08-14T23:54:30.2795338Z Average cache write               0.000 s
2025-08-14T23:54:30.2795909Z Average compiler                  0.000 s
2025-08-14T23:54:30.2796444Z Average cache read hit            0.000 s
2025-08-14T23:54:30.2797026Z Failed distributed compilations       0
2025-08-14T23:54:30.2797404Z 
2025-08-14T23:54:30.2797591Z Non-cacheable reasons:
2025-08-14T23:54:30.2798046Z -E                                    1
2025-08-14T23:54:30.2798402Z 
2025-08-14T23:54:30.2798760Z Cache location                  Local disk: "/var/lib/jenkins/.cache/sccache"
2025-08-14T23:54:30.2799535Z Use direct/preprocessor mode?   yes
2025-08-14T23:54:30.2800086Z Version (client)                0.10.0
2025-08-14T23:54:30.2800658Z Max cache size                       10 GiB
2025-08-14T23:54:30.2801241Z + echo ::endgroup::
2025-08-14T23:54:30.2802047Z ##[endgroup]
2025-08-14T23:54:30.2971229Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-08-14T23:54:30.2972812Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct
2025-08-14T23:54:30.2974586Z docker exec -t "e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test"
2025-08-14T23:54:30.3041192Z shell: /usr/bin/bash -e {0}
2025-08-14T23:54:30.3041686Z env:
2025-08-14T23:54:30.3042072Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:30.3042848Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:30.3043990Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:30.3045021Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:30.3047124Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:30.3049119Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:30.3049617Z   AWS_REGION: us-east-1
2025-08-14T23:54:30.3050174Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:30.3050835Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:30.3061085Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:30.3061868Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:30.3062700Z ##[endgroup]
2025-08-14T23:54:30.5292123Z ##[group]Run cat test/**/*_toprint.log || true
2025-08-14T23:54:30.5292854Z cat test/**/*_toprint.log || true
2025-08-14T23:54:30.5354029Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:30.5354825Z env:
2025-08-14T23:54:30.5355226Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:30.5355975Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:30.5357056Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:30.5358095Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:30.5359782Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:30.5361327Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:30.5361831Z   AWS_REGION: us-east-1
2025-08-14T23:54:30.5362460Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:30.5363134Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:30.5373771Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:30.5374663Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:30.5375652Z ##[endgroup]
2025-08-14T23:54:30.5697438Z cat: 'test/**/*_toprint.log': No such file or directory
2025-08-14T23:54:30.5924977Z Prepare all required actions
2025-08-14T23:54:30.5926530Z Getting action download info
2025-08-14T23:54:30.9973156Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2025-08-14T23:54:33.8371123Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02)
2025-08-14T23:54:34.7272734Z ##[group]Run ./.github/actions/upload-test-artifacts
2025-08-14T23:54:34.7273015Z with:
2025-08-14T23:54:34.7273191Z   use-gha: true
2025-08-14T23:54:34.7273468Z   file-suffix: test-distributed-3-3-linux.rocm.gpu.4_48128182504
2025-08-14T23:54:34.7273770Z   s3-bucket: gha-artifacts
2025-08-14T23:54:34.7273962Z env:
2025-08-14T23:54:34.7274118Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:34.7274431Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:34.7274892Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:34.7275408Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:34.7276128Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:34.7276779Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:34.7276991Z   AWS_REGION: us-east-1
2025-08-14T23:54:34.7277239Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:34.7277524Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:34.7281650Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:34.7281991Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:34.7282344Z ##[endgroup]
2025-08-14T23:54:34.7341115Z ##[group]Run actions/upload-artifact@v4
2025-08-14T23:54:34.7341386Z with:
2025-08-14T23:54:34.7341704Z   name: test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip
2025-08-14T23:54:34.7342108Z   retention-days: 14
2025-08-14T23:54:34.7342296Z   if-no-files-found: warn
2025-08-14T23:54:34.7342490Z   path: test/**/*.json
2025-08-14T23:54:34.7342675Z   compression-level: 6
2025-08-14T23:54:34.7343108Z   overwrite: false
2025-08-14T23:54:34.7343291Z   include-hidden-files: false
2025-08-14T23:54:34.7343489Z env:
2025-08-14T23:54:34.7343639Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:34.7343952Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:34.7344396Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:34.7344807Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:34.7345497Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:34.7346110Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:34.7346329Z   AWS_REGION: us-east-1
2025-08-14T23:54:34.7346591Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:34.7346870Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:34.7373795Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:34.7374245Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:34.7374626Z ##[endgroup]
2025-08-14T23:54:35.5664868Z With the provided path, there will be 6 files uploaded
2025-08-14T23:54:35.5665652Z Artifact name is valid!
2025-08-14T23:54:35.5667962Z Root directory input is valid!
2025-08-14T23:54:35.7888093Z Beginning upload of artifact content to blob storage
2025-08-14T23:54:36.1222827Z Uploaded bytes 43828
2025-08-14T23:54:36.1857141Z Finished uploading artifact content to blob storage!
2025-08-14T23:54:36.1862792Z SHA256 digest of uploaded artifact zip is be6cebf388b58dbb4ad9b4882222b0f246ba753e3d0f1c8f9fef01020f455032
2025-08-14T23:54:36.1865118Z Finalizing artifact upload
2025-08-14T23:54:36.3083167Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip.zip successfully finalized. Artifact ID 3770267715
2025-08-14T23:54:36.3085824Z Artifact test-jsons-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip has been successfully uploaded! Final size is 43828 bytes. Artifact ID is 3770267715
2025-08-14T23:54:36.3096587Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770267715
2025-08-14T23:54:36.3473869Z ##[group]Run actions/upload-artifact@v4
2025-08-14T23:54:36.3474463Z with:
2025-08-14T23:54:36.3475299Z   name: test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip
2025-08-14T23:54:36.3476283Z   retention-days: 14
2025-08-14T23:54:36.3476771Z   if-no-files-found: ignore
2025-08-14T23:54:36.3477316Z   path: test/**/*.xml
test/**/*.csv

2025-08-14T23:54:36.3477905Z   compression-level: 6
2025-08-14T23:54:36.3478396Z   overwrite: false
2025-08-14T23:54:36.3478878Z   include-hidden-files: false
2025-08-14T23:54:36.3479377Z env:
2025-08-14T23:54:36.3479802Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:36.3480612Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:36.3481814Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:36.3482905Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:36.3484779Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:36.3486810Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:36.3487353Z   AWS_REGION: us-east-1
2025-08-14T23:54:36.3488008Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:36.3488866Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:36.3499918Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:36.3500911Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:36.3501891Z ##[endgroup]
2025-08-14T23:54:37.2501698Z With the provided path, there will be 627 files uploaded
2025-08-14T23:54:37.2506312Z Artifact name is valid!
2025-08-14T23:54:37.2507516Z Root directory input is valid!
2025-08-14T23:54:37.4807722Z Beginning upload of artifact content to blob storage
2025-08-14T23:54:38.2839706Z Uploaded bytes 473369
2025-08-14T23:54:38.3469524Z Finished uploading artifact content to blob storage!
2025-08-14T23:54:38.3475230Z SHA256 digest of uploaded artifact zip is 27c50b8b5c784b16e3d9eb537973ffc8817c7c3e7133ee64690ee8800d9e34cb
2025-08-14T23:54:38.3478119Z Finalizing artifact upload
2025-08-14T23:54:38.4599191Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip.zip successfully finalized. Artifact ID 3770267876
2025-08-14T23:54:38.4601726Z Artifact test-reports-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip has been successfully uploaded! Final size is 473369 bytes. Artifact ID is 3770267876
2025-08-14T23:54:38.4616426Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770267876
2025-08-14T23:54:38.5061345Z ##[group]Run actions/upload-artifact@v4
2025-08-14T23:54:38.5061963Z with:
2025-08-14T23:54:38.5062681Z   name: logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip
2025-08-14T23:54:38.5063547Z   retention-days: 14
2025-08-14T23:54:38.5064020Z   if-no-files-found: ignore
2025-08-14T23:54:38.5064564Z   path: usage_log.txt
test/**/*.log

2025-08-14T23:54:38.5065119Z   compression-level: 6
2025-08-14T23:54:38.5065581Z   overwrite: false
2025-08-14T23:54:38.5066034Z   include-hidden-files: false
2025-08-14T23:54:38.5066537Z env:
2025-08-14T23:54:38.5066952Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:38.5067729Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:38.5068865Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:38.5070329Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:38.5072094Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:38.5073684Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:38.5074225Z   AWS_REGION: us-east-1
2025-08-14T23:54:38.5074811Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:38.5075539Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:38.5086057Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:38.5086918Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:38.5087814Z ##[endgroup]
2025-08-14T23:54:39.4343444Z Multiple search paths detected. Calculating the least common ancestor of all paths
2025-08-14T23:54:39.4345808Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact
2025-08-14T23:54:39.4347185Z With the provided path, there will be 92 files uploaded
2025-08-14T23:54:39.4348180Z Artifact name is valid!
2025-08-14T23:54:39.4348674Z Root directory input is valid!
2025-08-14T23:54:39.6479610Z Beginning upload of artifact content to blob storage
2025-08-14T23:54:40.4520398Z Uploaded bytes 647445
2025-08-14T23:54:40.5218084Z Finished uploading artifact content to blob storage!
2025-08-14T23:54:40.5223940Z SHA256 digest of uploaded artifact zip is 4cd95e1f7679a1b0de903e4260332a3ed80ff9e09ef8de2b98989c3327330f49
2025-08-14T23:54:40.5226729Z Finalizing artifact upload
2025-08-14T23:54:40.6250042Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip.zip successfully finalized. Artifact ID 3770268054
2025-08-14T23:54:40.6252327Z Artifact logs-runattempt1-test-distributed-3-3-linux.rocm.gpu.4_48128182504.zip has been successfully uploaded! Final size is 647445 bytes. Artifact ID is 3770268054
2025-08-14T23:54:40.6263554Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770268054
2025-08-14T23:54:40.6694972Z ##[group]Run # shellcheck disable=SC2156
2025-08-14T23:54:40.6695746Z # shellcheck disable=SC2156
2025-08-14T23:54:40.6697267Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2025-08-14T23:54:40.6766618Z shell: /usr/bin/bash -e {0}
2025-08-14T23:54:40.6767174Z env:
2025-08-14T23:54:40.6767618Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:40.6768442Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:40.6769603Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:40.6770731Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:40.6772505Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:40.6774130Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:40.6774677Z   AWS_REGION: us-east-1
2025-08-14T23:54:40.6775288Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:40.6776020Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:40.6786230Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:40.6787084Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:40.6787994Z ##[endgroup]
2025-08-14T23:54:41.0815904Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722
2025-08-14T23:54:41.0816911Z with:
2025-08-14T23:54:41.0817660Z   role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results
2025-08-14T23:54:41.0818563Z   role-duration-seconds: 18000
2025-08-14T23:54:41.0819130Z   aws-region: us-east-1
2025-08-14T23:54:41.0819736Z   audience: sts.amazonaws.com
2025-08-14T23:54:41.0820329Z env:
2025-08-14T23:54:41.0820807Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:41.0821742Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:41.0823094Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:41.0824370Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:41.0826321Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:41.0827876Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:41.0828413Z   AWS_REGION: us-east-1
2025-08-14T23:54:41.0829001Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:41.0829708Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:41.0840412Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:41.0841233Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:41.0842131Z ##[endgroup]
2025-08-14T23:54:41.4349441Z Assuming role with OIDC
2025-08-14T23:54:41.6708594Z Authenticated as assumedRoleId AROAUPVRELQNA5GQHA6IA:GitHubActions
2025-08-14T23:54:41.7584419Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main
2025-08-14T23:54:41.7585428Z with:
2025-08-14T23:54:41.7585969Z   benchmark-results-dir: test/test-reports
2025-08-14T23:54:41.7586624Z   dry-run: false
2025-08-14T23:54:41.7587128Z   schema-version: v3
2025-08-14T23:54:41.7588119Z   github-token: ***
2025-08-14T23:54:41.7588714Z env:
2025-08-14T23:54:41.7589271Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:41.7590235Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:41.7591474Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:41.7592689Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:41.7594606Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:41.7596370Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:41.7596965Z   AWS_REGION: us-east-1
2025-08-14T23:54:41.7598047Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:41.7598831Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:41.7612252Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:41.7613172Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:41.7614142Z ##[endgroup]
2025-08-14T23:54:41.7646670Z ##[group]Run set -eux
2025-08-14T23:54:41.7647175Z set -eux
2025-08-14T23:54:41.7647824Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-08-14T23:54:41.7648558Z 
2025-08-14T23:54:41.7648949Z DEVICE_NAME=""
2025-08-14T23:54:41.7649426Z DEVICE_TYPE=""
2025-08-14T23:54:41.7649851Z 
2025-08-14T23:54:41.7650288Z if command -v nvidia-smi; then
2025-08-14T23:54:41.7651067Z   # NB: I'm using PyTorch here to get the device name, however, it needs to
2025-08-14T23:54:41.7652033Z   # install the correct version of PyTorch manually for now. Any PyTorch
2025-08-14T23:54:41.7652946Z   # version is fine, I just use 2.7.1 to satify PYPIDEP linter
2025-08-14T23:54:41.7653692Z   python3 -mpip install torch==2.7.1
2025-08-14T23:54:41.7654283Z elif command -v rocminfo; then
2025-08-14T23:54:41.7655022Z   # NB: Installing torch on ROCm runner with pip here causes CI to fail
2025-08-14T23:54:41.7655949Z   # with a memoryview is too large error only on MI300 runners. Is pip
2025-08-14T23:54:41.7656883Z   # version on ROCm runner there too old? As a workaround, let's use the
2025-08-14T23:54:41.7657716Z   # GPU device name coming from rocminfo instead
2025-08-14T23:54:41.7658347Z   DEVICE_NAME=rocm
2025-08-14T23:54:41.7659171Z   DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs)
2025-08-14T23:54:41.7659989Z fi
2025-08-14T23:54:41.7660380Z 
2025-08-14T23:54:41.7660873Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV
2025-08-14T23:54:41.7661592Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV
2025-08-14T23:54:41.7722961Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:41.7723625Z env:
2025-08-14T23:54:41.7724038Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:41.7724774Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:41.7725968Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:41.7726983Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:41.7728561Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:41.7729981Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:41.7730909Z   AWS_REGION: us-east-1
2025-08-14T23:54:41.7731478Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:41.7732154Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:41.7743026Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:41.7743776Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:41.7744591Z ##[endgroup]
2025-08-14T23:54:41.7851755Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0
2025-08-14T23:54:42.0748170Z Defaulting to user installation because normal site-packages is not writeable
2025-08-14T23:54:42.2450582Z Requirement already satisfied: boto3==1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (1.35.33)
2025-08-14T23:54:42.2454613Z Requirement already satisfied: psutil==7.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (7.0.0)
2025-08-14T23:54:42.2460110Z Requirement already satisfied: pynvml==12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (12.0.0)
2025-08-14T23:54:42.2493768Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from boto3==1.35.33) (0.10.0)
2025-08-14T23:54:42.2499600Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (0.10.4)
2025-08-14T23:54:42.2503203Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (1.35.99)
2025-08-14T23:54:42.2657466Z Requirement already satisfied: nvidia-ml-py<13.0.0a0,>=12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from pynvml==12.0.0) (12.575.51)
2025-08-14T23:54:42.2706427Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.26.5)
2025-08-14T23:54:42.2710930Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.9.0.post0)
2025-08-14T23:54:42.2754128Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.16.0)
2025-08-14T23:54:42.5612854Z + DEVICE_NAME=
2025-08-14T23:54:42.5613429Z + DEVICE_TYPE=
2025-08-14T23:54:42.5613931Z + command -v nvidia-smi
2025-08-14T23:54:42.5614493Z + command -v rocminfo
2025-08-14T23:54:42.5615032Z + DEVICE_NAME=rocm
2025-08-14T23:54:42.5615518Z /opt/rocm/bin/rocminfo
2025-08-14T23:54:42.5629927Z ++ rocminfo
2025-08-14T23:54:42.5633257Z ++ grep 'Marketing Name'
2025-08-14T23:54:42.5648502Z ++ tail -n1
2025-08-14T23:54:42.5649145Z ++ awk -F: '{print $2}'
2025-08-14T23:54:42.5649703Z ++ xargs
2025-08-14T23:54:42.7200415Z + DEVICE_TYPE='AMD Instinct MI250X/MI250'
2025-08-14T23:54:42.7201189Z + echo DEVICE_NAME=rocm
2025-08-14T23:54:42.7201821Z + echo 'DEVICE_TYPE=AMD Instinct MI250X/MI250'
2025-08-14T23:54:42.7254596Z ##[group]Run set -eux
2025-08-14T23:54:42.7255164Z set -eux
2025-08-14T23:54:42.7255661Z 
2025-08-14T23:54:42.7256170Z if [[ -z "${GITHUB_TOKEN}" ]]; then
2025-08-14T23:54:42.7256896Z   echo "Missing github-token input"
2025-08-14T23:54:42.7257497Z   exit 1
2025-08-14T23:54:42.7257943Z fi
2025-08-14T23:54:42.7326199Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:42.7326965Z env:
2025-08-14T23:54:42.7327433Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:42.7328289Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:42.7329492Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:42.7330639Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:42.7332439Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:42.7334505Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:42.7335131Z   AWS_REGION: us-east-1
2025-08-14T23:54:42.7335799Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:42.7336553Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:42.7348880Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:42.7349757Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:42.7350677Z   DEVICE_NAME: rocm
2025-08-14T23:54:42.7351185Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:42.7352092Z   GITHUB_TOKEN: ***
2025-08-14T23:54:42.7352582Z ##[endgroup]
2025-08-14T23:54:42.7461392Z + [[ -z *** ]]
2025-08-14T23:54:42.7573650Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main
2025-08-14T23:54:42.7574479Z with:
2025-08-14T23:54:42.7575202Z   github-token: ***
2025-08-14T23:54:42.7575688Z env:
2025-08-14T23:54:42.7576131Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:42.7576962Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:42.7578155Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:42.7579737Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:42.7581536Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:42.7583137Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:42.7583698Z   AWS_REGION: us-east-1
2025-08-14T23:54:42.7584279Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:42.7584995Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:42.7597311Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:42.7598171Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:42.7599082Z   DEVICE_NAME: rocm
2025-08-14T23:54:42.7599604Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:42.7600199Z ##[endgroup]
2025-08-14T23:54:42.7631396Z ##[group]Run set -eux
2025-08-14T23:54:42.7631942Z set -eux
2025-08-14T23:54:42.7632405Z 
2025-08-14T23:54:42.7633333Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}"
2025-08-14T23:54:42.7701098Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:42.7701775Z env:
2025-08-14T23:54:42.7702173Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:42.7702926Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:42.7703981Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:42.7704950Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:42.7706529Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:42.7707943Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:42.7708456Z   AWS_REGION: us-east-1
2025-08-14T23:54:42.7708995Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:42.7709674Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:42.7720494Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:42.7721253Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:42.7722064Z   DEVICE_NAME: rocm
2025-08-14T23:54:42.7722530Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:42.7723259Z   GITHUB_TOKEN: ***
2025-08-14T23:54:42.7723690Z ##[endgroup]
2025-08-14T23:54:42.7823407Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16976370303 gpud535
2025-08-14T23:54:44.5887367Z setting job-id=48128182504
2025-08-14T23:54:44.5888628Z setting job-name=linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T23:54:44.6131601Z ##[group]Run set -eux
2025-08-14T23:54:44.6132225Z set -eux
2025-08-14T23:54:44.6132709Z 
2025-08-14T23:54:44.6133528Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \
2025-08-14T23:54:44.6134561Z   --schema-version "${SCHEMA_VERSION}" \
2025-08-14T23:54:44.6135283Z   --repo "${REPO}" \
2025-08-14T23:54:44.6135931Z   --head-branch "${HEAD_BRANCH}" \
2025-08-14T23:54:44.6136638Z   --head-sha "${HEAD_SHA}" \
2025-08-14T23:54:44.6137348Z   --workflow-id "${WORKFLOW_RUN_ID}" \
2025-08-14T23:54:44.6138075Z   --run-attempt "${RUN_ATTEMPT}" \
2025-08-14T23:54:44.6138761Z   --job-id "${JOB_ID}" \
2025-08-14T23:54:44.6139396Z   --job-name "${JOB_NAME}"
2025-08-14T23:54:44.6210442Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:44.6211248Z env:
2025-08-14T23:54:44.6211730Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:44.6212647Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:44.6213911Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:44.6215546Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:44.6217710Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:44.6219689Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:44.6220394Z   AWS_REGION: us-east-1
2025-08-14T23:54:44.6221189Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:44.6222137Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:44.6228638Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:44.6229521Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:44.6230622Z   DEVICE_NAME: rocm
2025-08-14T23:54:44.6231242Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:44.6231919Z   SCHEMA_VERSION: v3
2025-08-14T23:54:44.6232437Z   REPO: pytorch/pytorch
2025-08-14T23:54:44.6232985Z   HEAD_BRANCH: refs/heads/main
2025-08-14T23:54:44.6233629Z   HEAD_SHA: 1fc683cf17c8c673044538d10266c00f92987be2
2025-08-14T23:54:44.6234313Z   WORKFLOW_RUN_ID: 16976370303
2025-08-14T23:54:44.6234838Z   RUN_ATTEMPT: 1
2025-08-14T23:54:44.6235300Z   JOB_ID: 48128182504
2025-08-14T23:54:44.6236276Z   JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)
2025-08-14T23:54:44.6237304Z ##[endgroup]
2025-08-14T23:54:44.6345302Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 1fc683cf17c8c673044538d10266c00f92987be2 --workflow-id 16976370303 --run-attempt 1 --job-id 48128182504 --job-name 'linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)'
2025-08-14T23:54:44.6647835Z ##[group]Run set -eux
2025-08-14T23:54:44.6648412Z set -eux
2025-08-14T23:54:44.6648862Z 
2025-08-14T23:54:44.6649759Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py"
2025-08-14T23:54:44.6720987Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:44.6721754Z env:
2025-08-14T23:54:44.6722215Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:44.6723058Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:44.6724266Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:44.6725414Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:44.6727817Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:44.6729467Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:44.6730046Z   AWS_REGION: us-east-1
2025-08-14T23:54:44.6730679Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:44.6731426Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:44.6744663Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:44.6745708Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:44.6746627Z   DEVICE_NAME: rocm
2025-08-14T23:54:44.6747157Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:44.6747764Z ##[endgroup]
2025-08-14T23:54:44.6849972Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py
2025-08-14T23:54:45.4350696Z /var/home/pytorchci/.local/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:276: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.)
2025-08-14T23:54:45.4352938Z   cpu = _conversion_method_template(device=torch.device("cpu"))
2025-08-14T23:54:46.0225708Z ##[group]Run set -eux
2025-08-14T23:54:46.0226407Z set -eux
2025-08-14T23:54:46.0226961Z 
2025-08-14T23:54:46.0227605Z # TODO (huydhn): Implement this part
2025-08-14T23:54:46.0228475Z echo "dependencies={}" >> "${GITHUB_OUTPUT}"
2025-08-14T23:54:46.0302307Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:46.0303098Z env:
2025-08-14T23:54:46.0303593Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:46.0304476Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:46.0305960Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:46.0307360Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:46.0309384Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:46.0311103Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:46.0311703Z   AWS_REGION: us-east-1
2025-08-14T23:54:46.0312356Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:46.0313148Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:46.0326392Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:46.0327324Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:46.0328285Z   DEVICE_NAME: rocm
2025-08-14T23:54:46.0328833Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:46.0329470Z ##[endgroup]
2025-08-14T23:54:46.0436301Z + echo 'dependencies={}'
2025-08-14T23:54:46.0481947Z ##[group]Run set -eux
2025-08-14T23:54:46.0482524Z set -eux
2025-08-14T23:54:46.0482989Z 
2025-08-14T23:54:46.0483532Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then
2025-08-14T23:54:46.0484410Z   echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping"
2025-08-14T23:54:46.0485382Z   # We don't want the job to fail if the directory doesn't exist
2025-08-14T23:54:46.0486380Z   exit 0
2025-08-14T23:54:46.0486843Z fi
2025-08-14T23:54:46.0487282Z 
2025-08-14T23:54:46.0487753Z if [[ "${DRY_RUN}" == "true" ]]; then
2025-08-14T23:54:46.0488669Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-08-14T23:54:46.0489734Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-08-14T23:54:46.0490578Z     --metadata "${BENCHMARK_METADATA}" \
2025-08-14T23:54:46.0491278Z     --runners "${RUNNER_INFO}" \
2025-08-14T23:54:46.0491969Z     --dependencies "${DEPENDENCIES}" \
2025-08-14T23:54:46.0492599Z     --dry-run
2025-08-14T23:54:46.0493108Z else
2025-08-14T23:54:46.0493853Z   python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \
2025-08-14T23:54:46.0495288Z     --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \
2025-08-14T23:54:46.0496156Z     --metadata "${BENCHMARK_METADATA}" \
2025-08-14T23:54:46.0496830Z     --runners "${RUNNER_INFO}" \
2025-08-14T23:54:46.0497506Z     --dependencies "${DEPENDENCIES}"
2025-08-14T23:54:46.0498121Z fi
2025-08-14T23:54:46.0564699Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:46.0565454Z env:
2025-08-14T23:54:46.0566026Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:46.0566899Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:46.0568105Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:46.0569209Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:46.0571017Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:46.0572629Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:46.0573728Z   AWS_REGION: us-east-1
2025-08-14T23:54:46.0574462Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:46.0575347Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:46.0588939Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:46.0589815Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:46.0590734Z   DEVICE_NAME: rocm
2025-08-14T23:54:46.0591272Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:46.0591968Z   BENCHMARK_RESULTS_DIR: test/test-reports
2025-08-14T23:54:46.0592585Z   DRY_RUN: false
2025-08-14T23:54:46.0595102Z   BENCHMARK_METADATA: {"timestamp": 1755215684, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182504}
2025-08-14T23:54:46.0598458Z   RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpud535.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]
2025-08-14T23:54:46.0599823Z   DEPENDENCIES: {}
2025-08-14T23:54:46.0600302Z ##[endgroup]
2025-08-14T23:54:46.0707304Z + [[ ! -d test/test-reports ]]
2025-08-14T23:54:46.0708007Z + [[ false == \t\r\u\e ]]
2025-08-14T23:54:46.0713313Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1755215684, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 3, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182504}' --runners '[{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpud535.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]' --dependencies '{}'
2025-08-14T23:54:46.2430417Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'distributed/fsdp/test_fsdp_apply'}, {'test_file': 'distributed/fsdp/test_distributed_checkpoint'}, {'test_file': 'distributed/test_store'}, {'test_file': 'distributed/test_dynamo_distributed'}, {'test_file': 'distributed/test_inductor_collectives'}, {'test_file': 'distributed/tensor/test_dtensor_compile'}, {'test_file': 'distributed/test_c10d_functional_native'}, {'test_file': 'distributed/tensor/parallel/test_micro_pipeline_tp'}, {'test_file': 'distributed/tensor/debug/test_op_coverage'}, {'test_file': 'distributed/_tools/test_mod_tracker'}, {'test_file': 'distributed/checkpoint/test_nested_dict'}, {'test_file': 'distributed/flight_recorder/test_fr_analysis'}, {'test_file': 'distributed/test_compute_comm_reordering'}, {'test_file': 'distributed/checkpoint/test_consolidate_hf_safetensors'}, {'test_file': 'distributed/_tools/test_runtime_estimator'}, {'test_file': 'distributed/_tools/test_sac_ilp'}, {'test_file': 'distributed/pipelining/test_schedule_multiproc'}, {'test_file': 'distributed/_composable/test_contract'}, {'test_file': 'distributed/_shard/sharded_tensor/test_logger'}, {'test_file': 'distributed/pipelining/test_transformer'}, {'test_file': 'distributed/tensor/test_dtensor'}, {'test_file': 'distributed/tensor/test_attention'}, {'test_file': 'distributed/pipelining/test_pipe'}, {'test_file': 'distributed/_tools/test_fake_collectives'}, {'test_file': 'distributed/pipelining/test_backward'}, {'test_file': 'distributed/checkpoint/_experimental/test_barriers'}, {'test_file': 'distributed/checkpoint/_experimental/test_types'}, {'test_file': 'distributed/_tools/test_sac_estimator'}, {'test_file': 'distributed/tensor/test_xla_integration'}, {'test_file': 'distributed/checkpoint/test_dedup_tensors'}, {'test_file': 'distributed/pipelining/test_unflatten'}, {'test_file': 'distributed/checkpoint/test_format_utils'}, {'test_file': 'distributed/tensor/test_dtensor_ops'}, {'test_file': 'distributed/pipelining/test_microbatch'}, {'test_file': 'distributed/tensor/test_api'}, {'test_file': 'distributed/fsdp/test_fsdp_fx'}, {'test_file': 'distributed/test_nvshmem_triton'}, {'test_file': 'distributed/tensor/test_experimental_ops'}, {'test_file': 'distributed/checkpoint/test_hf_storage'}, {'test_file': 'distributed/tensor/test_common_rules'}, {'test_file': 'distributed/tensor/test_convolution_ops'}, {'test_file': 'distributed/tensor/test_matrix_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_reader'}, {'test_file': 'distributed/_composable/test_composability/test_pp_composability'}, {'test_file': 'distributed/tensor/test_tensor_ops'}, {'test_file': 'distributed/tensor/debug/test_comm_mode_features'}, {'test_file': 'distributed/tensor/experimental/test_register_sharding'}, {'test_file': 'distributed/checkpoint/_experimental/test_builder'}, {'test_file': 'distributed/tensor/test_init'}, {'test_file': 'distributed/optim/test_named_optimizer'}, {'test_file': 'distributed/_composable/test_replicate_with_fsdp'}, {'test_file': 'distributed/tensor/parallel/test_tp_style'}, {'test_file': 'distributed/_tools/test_memory_tracker'}, {'test_file': 'distributed/_composable/test_checkpoint'}, {'test_file': 'distributed/checkpoint/_experimental/test_staging'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_writer'}, {'test_file': 'distributed/tensor/test_pointwise_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpointer'}, {'test_file': 'distributed/checkpoint/fsdp/test_fsdp_dsd'}, {'test_file': 'distributed/tensor/experimental/test_local_map'}, {'test_file': 'distributed/checkpoint/e2e/test_fsdp_ep'}, {'test_file': 'distributed/test_functional_api'}, {'test_file': 'distributed/tensor/test_random_ops'}, {'test_file': 'distributed/_tools/test_fsdp2_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_traverse'}, {'test_file': 'distributed/checkpoint/test_fsspec'}, {'test_file': 'distributed/tensor/test_redistribute'}, {'test_file': 'distributed/checkpoint/test_async_process_executor'}, {'test_file': 'distributed/tensor/test_math_ops'}, {'test_file': 'distributed/test_backends'}, {'test_file': 'distributed/tensor/experimental/test_tp_transform'}, {'test_file': 'distributed/test_device_mesh'}, {'test_file': 'distributed/checkpoint/test_fsdp_optim_state'}, {'test_file': 'distributed/checkpoint/test_dtensor_resharding'}, {'test_file': 'distributed/test_symmetric_memory'}, {'test_file': 'distributed/checkpoint/e2e/test_fine_tuning'}, {'test_file': 'distributed/pipelining/test_stage'}, {'test_file': 'distributed/_tools/test_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_tp_checkpoint'}, {'test_file': 'distributed/pipelining/test_schedule'}, {'test_file': 'distributed/test_nvshmem'}, {'test_file': 'distributed/tensor/test_embedding_ops'}, {'test_file': 'distributed/checkpoint/test_save_load_api'}, {'test_file': 'distributed/test_control_collectives'}, {'test_file': 'distributed/checkpoint/test_hf_safetensor_e2e'}, {'test_file': 'distributed/checkpoint/e2e/test_e2e_save_and_load'}, {'test_file': 'distributed/checkpoint/test_planner'}, {'test_file': 'distributed/launcher/test_run'}, {'test_file': 'distributed/checkpoint/test_hsdp_checkpoint'}, {'test_file': 'distributed/checkpoint/test_dtensor_checkpoint'}, {'test_file': 'distributed/tensor/test_view_ops'}, {'test_file': 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion'}, {'test_file': 'distributed/tensor/test_optimizers'}, {'test_file': 'distributed/test_p2p_ipc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_ignore_params'}, {'test_file': 'distributed/checkpoint/test_compatibility'}, {'test_file': 'distributed/tensor/parallel/test_tp_examples'}, {'test_file': 'distributed/tensor/test_utils'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_overlap'}, {'test_file': 'distributed/tensor/parallel/test_tp_random_state'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_logging'}, {'test_file': 'distributed/optim/test_apply_optimizer_in_backward'}, {'test_file': 'distributed/_composable/test_composability/test_2d_composability'}, {'test_file': 'distributed/_composable/test_replicate_with_compiler'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_compile'}, {'test_file': 'distributed/checkpoint/test_fsdp_model_state'}, {'test_file': 'distributed/fsdp/test_shard_utils'}, {'test_file': 'distributed/elastic/test_control_plane'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_grad_scaler'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_process'}, {'test_file': 'distributed/tensor/parallel/test_parallelize_api'}, {'test_file': 'distributed/fsdp/test_fsdp_optim_state'}, {'test_file': 'distributed/tensor/test_op_strategy'}, {'test_file': 'distributed/checkpoint/test_checkpoint'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_'}, {'test_file': 'distributed/checkpoint/test_pg_transport'}, {'test_file': 'distributed/test_collective_utils'}, {'test_file': 'distributed/tensor/debug/test_comm_mode'}, {'test_file': 'distributed/fsdp/test_fsdp_pure_fp16'}, {'test_file': 'distributed/checkpoint/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_state_dict'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state'}, {'test_file': 'distributed/test_dist2'}, {'test_file': 'distributed/fsdp/test_fsdp_memory'}, {'test_file': 'distributed/_composable/test_replicate'}, {'test_file': 'distributed/test_fake_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_backward_prefetch'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_autograd'}, {'test_file': 'distributed/checkpoint/test_state_dict_stager'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_frozen'}, {'test_file': 'distributed/fsdp/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_overlap'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_init'}, {'test_file': 'distributed/fsdp/test_checkpoint_wrapper'}, {'test_file': 'distributed/fsdp/test_fsdp_grad_acc'}, {'test_file': 'distributed/fsdp/test_fsdp_flatten_params'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_mixed_precision'}, {'test_file': 'distributed/test_c10d_logger'}, {'test_file': 'distributed/test_c10d_object_collectives'}, {'test_file': 'distributed/test_serialization'}, {'test_file': 'distributed/checkpoint/test_state_dict'}, {'test_file': 'distributed/optim/test_zero_redundancy_optimizer'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_comm'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint_cpu'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_extensions'}, {'test_file': 'distributed/fsdp/test_fsdp_ignored_modules'}, {'test_file': 'distributed/fsdp/test_fsdp_freezing_weights'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint'}, {'test_file': 'distributed/test_nccl'}, {'test_file': 'distributed/fsdp/test_fsdp_checkpoint'}, {'test_file': 'distributed/test_distributed_spawn'}, {'test_file': 'distributed/fsdp/test_fsdp_unshard_params'}, {'test_file': 'distributed/test_multi_threaded_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_tp_integration'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_training'}, {'test_file': 'distributed/fsdp/test_fsdp_dtensor_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_fine_tune'}, {'test_file': 'distributed/fsdp/test_fsdp_hybrid_shard'}, {'test_file': 'distributed/fsdp/test_fsdp_comm_hooks'}, {'test_file': 'distributed/fsdp/test_fsdp_exec_order'}, {'test_file': 'distributed/fsdp/test_fsdp_misc'}, {'test_file': 'distributed/fsdp/test_wrap'}, {'test_file': 'distributed/fsdp/test_fsdp_clip_grad_norm'}, {'test_file': 'distributed/fsdp/test_fsdp_meta'}, {'test_file': 'distributed/_shard/sharding_spec/test_sharding_spec'}, {'test_file': 'distributed/fsdp/test_hsdp_dtensor_state_dict'}, {'test_file': 'distributed/checkpoint/test_state_dict_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_uneven'}, {'test_file': 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks'}, {'test_file': 'distributed/algorithms/quantization/test_quantization'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_comm'}, {'test_file': 'distributed/fsdp/test_fsdp_input'}, {'test_file': 'distributed/fsdp/test_fsdp_traversal'}, {'test_file': 'distributed/fsdp/test_fsdp_sharded_grad_scaler'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_wrapping'}, {'test_file': 'distributed/_shard/test_sharder'}, {'test_file': 'distributed/algorithms/test_join'}, {'test_file': 'distributed/test_data_parallel'}, {'test_file': 'distributed/_shard/sharding_plan/test_sharding_plan'}, {'test_file': 'distributed/nn/jit/test_instantiator'}, {'test_file': 'distributed/test_pg_wrapper'}, {'test_file': 'distributed/test_c10d_pypg'}, {'test_file': 'distributed/fsdp/test_fsdp_use_orig_params'}, {'test_file': 'distributed/test_composability'}, {'test_file': 'distributed/fsdp/test_fsdp_core'}, {'test_file': 'distributed/_shard/sharded_optim/test_sharded_optim'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_forward'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_tensor_ops'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_binary_cmp'}, {'test_file': 'distributed/fsdp/test_fsdp_mixed_precision'}, {'test_file': 'distributed/test_c10d_ucc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_memory'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_init'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard'}, {'test_file': 'distributed/test_c10d_gloo'}, {'test_file': 'distributed/test_c10d_common'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding_bag'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor'}, {'test_file': 'distributed/test_c10d_spawn_gloo'}, {'test_file': 'distributed/test_c10d_spawn_nccl'}, {'test_file': 'distributed/test_launcher'}, {'test_file': 'distributed/test_c10d_spawn_ucc'}, {'test_file': 'distributed/test_c10d_nccl'}, {'test_file': 'distributed/test_c10d_ops_nccl'}, {'test_file': 'distributed/elastic/events/lib_test'}, {'test_file': 'distributed/elastic/metrics/api_test'}, {'test_file': 'distributed/elastic/multiprocessing/api_test'}, {'test_file': 'distributed/elastic/timer/api_test'}, {'test_file': 'distributed/elastic/timer/local_timer_example'}, {'test_file': 'distributed/elastic/timer/local_timer_test'}, {'test_file': 'distributed/elastic/utils/distributed_test'}, {'test_file': 'distributed/elastic/utils/logging_test'}, {'test_file': 'distributed/elastic/utils/util_test'}, {'test_file': 'distributed/tensor/test_dtensor_dispatch_overhead'}], 'excluded': []} from test/test-reports/td_exclusions-b03b1fddde1e0ba587d1.json is not a benchmark record, skipping
2025-08-14T23:54:46.2513206Z   warn(f"{result} from {filepath} is not a benchmark record, skipping")
2025-08-14T23:54:46.2602426Z Prepare all required actions
2025-08-14T23:54:46.2603252Z Getting action download info
2025-08-14T23:54:46.2657931Z ##[group]Run ./.github/actions/teardown-rocm
2025-08-14T23:54:46.2658681Z env:
2025-08-14T23:54:46.2659203Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:46.2660168Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:46.2686208Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:46.2687324Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:46.2688941Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:46.2690374Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:46.2691416Z   AWS_REGION: us-east-1
2025-08-14T23:54:46.2691967Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:46.2692685Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:46.2704719Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:46.2705506Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:46.2706332Z   DEVICE_NAME: rocm
2025-08-14T23:54:46.2706798Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:46.2707339Z ##[endgroup]
2025-08-14T23:54:46.2736990Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T23:54:46.2738060Z # ignore expansion of "docker ps -q" since it could be empty
2025-08-14T23:54:46.2738913Z # shellcheck disable=SC2046
2025-08-14T23:54:46.2739643Z docker stop $(docker ps -q) || true
2025-08-14T23:54:46.2740370Z # Prune all stopped containers.
2025-08-14T23:54:46.2741053Z docker container prune -f
2025-08-14T23:54:46.2798424Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:54:46.2799108Z env:
2025-08-14T23:54:46.2799517Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:54:46.2800246Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:54:46.2801292Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:54:46.2802277Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:54:46.2803848Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:54:46.2805257Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:54:46.2805951Z   AWS_REGION: us-east-1
2025-08-14T23:54:46.2806487Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:54:46.2807153Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:54:46.2818123Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:54:46.2819049Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:54:46.2820023Z   DEVICE_NAME: rocm
2025-08-14T23:54:46.2820591Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:54:46.2821236Z ##[endgroup]
2025-08-14T23:54:57.1293919Z e9242eedfa92
2025-08-14T23:55:04.9243319Z Deleted Containers:
2025-08-14T23:55:04.9244171Z e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:55:04.9244823Z 
2025-08-14T23:55:04.9245094Z Total reclaimed space: 10.11GB
2025-08-14T23:55:04.9364716Z Prepare all required actions
2025-08-14T23:55:04.9424272Z ##[group]Run ./.github/actions/diskspace-cleanup
2025-08-14T23:55:04.9424714Z with:
2025-08-14T23:55:04.9425040Z   diskspace-cutoff: 70
2025-08-14T23:55:04.9425389Z env:
2025-08-14T23:55:04.9425692Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:55:04.9426291Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:55:04.9427057Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:55:04.9427640Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:55:04.9428857Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:55:04.9429715Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:55:04.9430004Z   AWS_REGION: us-east-1
2025-08-14T23:55:04.9430343Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:55:04.9430720Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:55:04.9436962Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:55:04.9437393Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:55:04.9437858Z   DEVICE_NAME: rocm
2025-08-14T23:55:04.9438123Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:55:04.9438426Z ##[endgroup]
2025-08-14T23:55:04.9455730Z ##[group]Run set -ex
2025-08-14T23:55:04.9456073Z set -ex
2025-08-14T23:55:04.9456682Z diskspace_cutoff=70
2025-08-14T23:55:04.9457143Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}')
2025-08-14T23:55:04.9457643Z if [ ! -d "$docker_root_dir" ]; then
2025-08-14T23:55:04.9458255Z     echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check."
2025-08-14T23:55:04.9458829Z     exit 0
2025-08-14T23:55:04.9459121Z fi
2025-08-14T23:55:04.9459609Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T23:55:04.9460627Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified"
2025-08-14T23:55:04.9461640Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then
2025-08-14T23:55:04.9462172Z     docker system prune -af
2025-08-14T23:55:04.9462828Z     diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //')
2025-08-14T23:55:04.9463432Z     if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then
2025-08-14T23:55:04.9463999Z         echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace."
2025-08-14T23:55:04.9464502Z         echo "$msg"
2025-08-14T23:55:04.9464778Z         exit 1
2025-08-14T23:55:04.9465033Z     else
2025-08-14T23:55:04.9465336Z         difference=$((diskspace - diskspace_new))
2025-08-14T23:55:04.9465746Z         echo "Diskspace saved: $difference percent"
2025-08-14T23:55:04.9466078Z     fi
2025-08-14T23:55:04.9466308Z fi
2025-08-14T23:55:04.9528541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2025-08-14T23:55:04.9529288Z env:
2025-08-14T23:55:04.9529743Z   GIT_DEFAULT_BRANCH: main
2025-08-14T23:55:04.9530589Z   RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts
2025-08-14T23:55:04.9532033Z   RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results
2025-08-14T23:55:04.9533401Z   RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs
2025-08-14T23:55:04.9535497Z   GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host
2025-08-14T23:55:04.9537003Z   AWS_DEFAULT_REGION: us-east-1
2025-08-14T23:55:04.9537321Z   AWS_REGION: us-east-1
2025-08-14T23:55:04.9537632Z   AWS_ACCESS_KEY_ID: ***
2025-08-14T23:55:04.9538008Z   AWS_SECRET_ACCESS_KEY: ***
2025-08-14T23:55:04.9546094Z   AWS_SESSION_TOKEN: ***
2025-08-14T23:55:04.9546951Z   CONTAINER_NAME: e9242eedfa92fdedd69f5b2970461fdbb2727e419ddceb51fce8fd947968b28a
2025-08-14T23:55:04.9547858Z   DEVICE_NAME: rocm
2025-08-14T23:55:04.9548378Z   DEVICE_TYPE: AMD Instinct MI250X/MI250
2025-08-14T23:55:04.9548983Z ##[endgroup]
2025-08-14T23:55:04.9654246Z + diskspace_cutoff=70
2025-08-14T23:55:04.9664091Z ++ docker info -f '{{.DockerRootDir}}'
2025-08-14T23:55:05.0649978Z + docker_root_dir=/media/4TB/docker-rootless
2025-08-14T23:55:05.0650692Z + '[' '!' -d /media/4TB/docker-rootless ']'
2025-08-14T23:55:05.0665874Z ++ df -H --output=pcent /media/4TB/docker-rootless
2025-08-14T23:55:05.0670486Z ++ sed -n 2p
2025-08-14T23:55:05.0672250Z ++ sed s/%//
2025-08-14T23:55:05.0675203Z ++ sed 's/ //'
2025-08-14T23:55:05.0715593Z + diskspace=14
2025-08-14T23:55:05.0718005Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified'
2025-08-14T23:55:05.0719431Z + [[ 14 -ge 70 ]]
2025-08-14T23:55:05.0814161Z Post job cleanup.
2025-08-14T23:55:05.0907580Z Post job cleanup.
2025-08-14T23:55:05.2338173Z Post job cleanup.
2025-08-14T23:55:05.2985998Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com
2025-08-14T23:55:05.3667162Z Post job cleanup.
2025-08-14T23:55:05.5212261Z Post job cleanup.
2025-08-14T23:55:05.5299500Z Post job cleanup.
2025-08-14T23:55:05.6496169Z [command]/usr/bin/git version
2025-08-14T23:55:05.6552763Z git version 2.34.1
2025-08-14T23:55:05.6585501Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/c78359d4-cacc-486f-9429-64cc54357986/.gitconfig'
2025-08-14T23:55:05.6597819Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/c78359d4-cacc-486f-9429-64cc54357986' before making global git config changes
2025-08-14T23:55:05.6599672Z Adding repository directory to the temporary git global config as a safe directory
2025-08-14T23:55:05.6601265Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch
2025-08-14T23:55:05.6666601Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2025-08-14T23:55:05.6736437Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :"
2025-08-14T23:55:05.7452627Z Entering 'android/libs/fbjni'
2025-08-14T23:55:05.7578907Z Entering 'third_party/FP16'
2025-08-14T23:55:05.7709091Z Entering 'third_party/FXdiv'
2025-08-14T23:55:05.7834510Z Entering 'third_party/NNPACK'
2025-08-14T23:55:05.7963781Z Entering 'third_party/NVTX'
2025-08-14T23:55:05.8096566Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T23:55:05.8221050Z Entering 'third_party/XNNPACK'
2025-08-14T23:55:05.8384279Z Entering 'third_party/aiter'
2025-08-14T23:55:05.8507957Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T23:55:05.8648503Z Entering 'third_party/benchmark'
2025-08-14T23:55:05.8774308Z Entering 'third_party/composable_kernel'
2025-08-14T23:55:05.8911935Z Entering 'third_party/cpp-httplib'
2025-08-14T23:55:05.9035769Z Entering 'third_party/cpuinfo'
2025-08-14T23:55:05.9157312Z Entering 'third_party/cudnn_frontend'
2025-08-14T23:55:05.9280529Z Entering 'third_party/cutlass'
2025-08-14T23:55:05.9421867Z Entering 'third_party/fbgemm'
2025-08-14T23:55:05.9543468Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T23:55:05.9660136Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T23:55:05.9791635Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T23:55:05.9908991Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T23:55:06.0048648Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T23:55:06.0165055Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T23:55:06.0281010Z Entering 'third_party/fbgemm/external/json'
2025-08-14T23:55:06.0407930Z Entering 'third_party/flash-attention'
2025-08-14T23:55:06.0531184Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T23:55:06.0664928Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T23:55:06.0806447Z Entering 'third_party/flatbuffers'
2025-08-14T23:55:06.0937677Z Entering 'third_party/fmt'
2025-08-14T23:55:06.1063814Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T23:55:06.1188409Z Entering 'third_party/gloo'
2025-08-14T23:55:06.1312579Z Entering 'third_party/googletest'
2025-08-14T23:55:06.1438990Z Entering 'third_party/ideep'
2025-08-14T23:55:06.1556132Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T23:55:06.1692744Z Entering 'third_party/ittapi'
2025-08-14T23:55:06.1815986Z Entering 'third_party/kineto'
2025-08-14T23:55:06.1936928Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T23:55:06.2048227Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T23:55:06.2168431Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T23:55:06.2289919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T23:55:06.2414442Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T23:55:06.2529919Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T23:55:06.2656315Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T23:55:06.2774429Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T23:55:06.2892094Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T23:55:06.3012355Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T23:55:06.3135387Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T23:55:06.3254450Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T23:55:06.3379630Z Entering 'third_party/kleidiai'
2025-08-14T23:55:06.3506200Z Entering 'third_party/mimalloc'
2025-08-14T23:55:06.3630253Z Entering 'third_party/nlohmann'
2025-08-14T23:55:06.3757012Z Entering 'third_party/onnx'
2025-08-14T23:55:06.3911992Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T23:55:06.4046701Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T23:55:06.4171264Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T23:55:06.4287618Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T23:55:06.4403330Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T23:55:06.4519853Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T23:55:06.4640971Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T23:55:06.4757045Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T23:55:06.4871499Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T23:55:06.4985775Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T23:55:06.5106142Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T23:55:06.5230613Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T23:55:06.5394871Z Entering 'third_party/pocketfft'
2025-08-14T23:55:06.5518413Z Entering 'third_party/protobuf'
2025-08-14T23:55:06.5648811Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T23:55:06.5765830Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T23:55:06.5893325Z Entering 'third_party/psimd'
2025-08-14T23:55:06.6019490Z Entering 'third_party/pthreadpool'
2025-08-14T23:55:06.6143899Z Entering 'third_party/pybind11'
2025-08-14T23:55:06.6269881Z Entering 'third_party/python-peachpy'
2025-08-14T23:55:06.6393225Z Entering 'third_party/sleef'
2025-08-14T23:55:06.6519373Z Entering 'third_party/tensorpipe'
2025-08-14T23:55:06.6639666Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T23:55:06.6757683Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T23:55:06.6874592Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T23:55:06.6990536Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T23:55:06.7102692Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T23:55:06.7278846Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2025-08-14T23:55:06.7333858Z http.https://github.com/.extraheader
2025-08-14T23:55:06.7356887Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2025-08-14T23:55:06.7441891Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :"
2025-08-14T23:55:06.8122686Z Entering 'android/libs/fbjni'
2025-08-14T23:55:06.8189908Z http.https://github.com/.extraheader
2025-08-14T23:55:06.8288288Z Entering 'third_party/FP16'
2025-08-14T23:55:06.8353943Z http.https://github.com/.extraheader
2025-08-14T23:55:06.8448588Z Entering 'third_party/FXdiv'
2025-08-14T23:55:06.8514504Z http.https://github.com/.extraheader
2025-08-14T23:55:06.8607632Z Entering 'third_party/NNPACK'
2025-08-14T23:55:06.8673904Z http.https://github.com/.extraheader
2025-08-14T23:55:06.8768785Z Entering 'third_party/NVTX'
2025-08-14T23:55:06.8834475Z http.https://github.com/.extraheader
2025-08-14T23:55:06.8934597Z Entering 'third_party/VulkanMemoryAllocator'
2025-08-14T23:55:06.9001567Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9097129Z Entering 'third_party/XNNPACK'
2025-08-14T23:55:06.9161490Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9290728Z Entering 'third_party/aiter'
2025-08-14T23:55:06.9355617Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9448343Z Entering 'third_party/aiter/3rdparty/composable_kernel'
2025-08-14T23:55:06.9508889Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9625607Z Entering 'third_party/benchmark'
2025-08-14T23:55:06.9689426Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9781819Z Entering 'third_party/composable_kernel'
2025-08-14T23:55:06.9845357Z http.https://github.com/.extraheader
2025-08-14T23:55:06.9957974Z Entering 'third_party/cpp-httplib'
2025-08-14T23:55:07.0022623Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0116362Z Entering 'third_party/cpuinfo'
2025-08-14T23:55:07.0180701Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0274181Z Entering 'third_party/cudnn_frontend'
2025-08-14T23:55:07.0337814Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0433032Z Entering 'third_party/cutlass'
2025-08-14T23:55:07.0497010Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0611969Z Entering 'third_party/fbgemm'
2025-08-14T23:55:07.0675761Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0771709Z Entering 'third_party/fbgemm/external/asmjit'
2025-08-14T23:55:07.0832477Z http.https://github.com/.extraheader
2025-08-14T23:55:07.0925885Z Entering 'third_party/fbgemm/external/composable_kernel'
2025-08-14T23:55:07.0987299Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1094801Z Entering 'third_party/fbgemm/external/cpuinfo'
2025-08-14T23:55:07.1155263Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1245050Z Entering 'third_party/fbgemm/external/cutlass'
2025-08-14T23:55:07.1306567Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1422459Z Entering 'third_party/fbgemm/external/googletest'
2025-08-14T23:55:07.1483941Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1578658Z Entering 'third_party/fbgemm/external/hipify_torch'
2025-08-14T23:55:07.1638093Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1729057Z Entering 'third_party/fbgemm/external/json'
2025-08-14T23:55:07.1791071Z http.https://github.com/.extraheader
2025-08-14T23:55:07.1896037Z Entering 'third_party/flash-attention'
2025-08-14T23:55:07.1961510Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2055368Z Entering 'third_party/flash-attention/csrc/composable_kernel'
2025-08-14T23:55:07.2116049Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2221860Z Entering 'third_party/flash-attention/csrc/cutlass'
2025-08-14T23:55:07.2282802Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2405794Z Entering 'third_party/flatbuffers'
2025-08-14T23:55:07.2472316Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2576161Z Entering 'third_party/fmt'
2025-08-14T23:55:07.2639938Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2734825Z Entering 'third_party/gemmlowp/gemmlowp'
2025-08-14T23:55:07.2802087Z http.https://github.com/.extraheader
2025-08-14T23:55:07.2900679Z Entering 'third_party/gloo'
2025-08-14T23:55:07.2965799Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3065144Z Entering 'third_party/googletest'
2025-08-14T23:55:07.3133808Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3230069Z Entering 'third_party/ideep'
2025-08-14T23:55:07.3297476Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3386049Z Entering 'third_party/ideep/mkl-dnn'
2025-08-14T23:55:07.3446314Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3558244Z Entering 'third_party/ittapi'
2025-08-14T23:55:07.3624152Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3717600Z Entering 'third_party/kineto'
2025-08-14T23:55:07.3782180Z http.https://github.com/.extraheader
2025-08-14T23:55:07.3871117Z Entering 'third_party/kineto/libkineto/third_party/dynolog'
2025-08-14T23:55:07.3932269Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4022099Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'
2025-08-14T23:55:07.4081322Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4174351Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'
2025-08-14T23:55:07.4234220Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4327716Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'
2025-08-14T23:55:07.4389164Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4481211Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'
2025-08-14T23:55:07.4541186Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4630249Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'
2025-08-14T23:55:07.4690873Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4790415Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog'
2025-08-14T23:55:07.4850497Z http.https://github.com/.extraheader
2025-08-14T23:55:07.4945605Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'
2025-08-14T23:55:07.5005378Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5094567Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json'
2025-08-14T23:55:07.5154492Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5248476Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'
2025-08-14T23:55:07.5308457Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5406207Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2025-08-14T23:55:07.5465954Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5557040Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2025-08-14T23:55:07.5616074Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5715165Z Entering 'third_party/kleidiai'
2025-08-14T23:55:07.5777907Z http.https://github.com/.extraheader
2025-08-14T23:55:07.5872432Z Entering 'third_party/mimalloc'
2025-08-14T23:55:07.5937351Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6032029Z Entering 'third_party/nlohmann'
2025-08-14T23:55:07.6098466Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6194839Z Entering 'third_party/onnx'
2025-08-14T23:55:07.6260486Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6387669Z Entering 'third_party/onnx/third_party/pybind11'
2025-08-14T23:55:07.6451499Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6555030Z Entering 'third_party/opentelemetry-cpp'
2025-08-14T23:55:07.6621203Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6715314Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark'
2025-08-14T23:55:07.6776229Z http.https://github.com/.extraheader
2025-08-14T23:55:07.6866830Z Entering 'third_party/opentelemetry-cpp/third_party/googletest'
2025-08-14T23:55:07.6926570Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7016374Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl'
2025-08-14T23:55:07.7076685Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7165643Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json'
2025-08-14T23:55:07.7227623Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7321448Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto'
2025-08-14T23:55:07.7383093Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7473018Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp'
2025-08-14T23:55:07.7533791Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7623118Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp'
2025-08-14T23:55:07.7682979Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7770035Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'
2025-08-14T23:55:07.7831324Z http.https://github.com/.extraheader
2025-08-14T23:55:07.7928533Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'
2025-08-14T23:55:07.7988597Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8087863Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg'
2025-08-14T23:55:07.8148286Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8288883Z Entering 'third_party/pocketfft'
2025-08-14T23:55:07.8354054Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8446451Z Entering 'third_party/protobuf'
2025-08-14T23:55:07.8511256Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8608642Z Entering 'third_party/protobuf/third_party/benchmark'
2025-08-14T23:55:07.8669876Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8759371Z Entering 'third_party/protobuf/third_party/googletest'
2025-08-14T23:55:07.8819576Z http.https://github.com/.extraheader
2025-08-14T23:55:07.8917286Z Entering 'third_party/psimd'
2025-08-14T23:55:07.8982286Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9076747Z Entering 'third_party/pthreadpool'
2025-08-14T23:55:07.9147689Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9247232Z Entering 'third_party/pybind11'
2025-08-14T23:55:07.9311019Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9404009Z Entering 'third_party/python-peachpy'
2025-08-14T23:55:07.9468194Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9561627Z Entering 'third_party/sleef'
2025-08-14T23:55:07.9626123Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9718873Z Entering 'third_party/tensorpipe'
2025-08-14T23:55:07.9783100Z http.https://github.com/.extraheader
2025-08-14T23:55:07.9871093Z Entering 'third_party/tensorpipe/third_party/googletest'
2025-08-14T23:55:07.9931365Z http.https://github.com/.extraheader
2025-08-14T23:55:08.0018178Z Entering 'third_party/tensorpipe/third_party/libnop'
2025-08-14T23:55:08.0076765Z http.https://github.com/.extraheader
2025-08-14T23:55:08.0164129Z Entering 'third_party/tensorpipe/third_party/libuv'
2025-08-14T23:55:08.0224552Z http.https://github.com/.extraheader
2025-08-14T23:55:08.0318170Z Entering 'third_party/tensorpipe/third_party/pybind11'
2025-08-14T23:55:08.0377780Z http.https://github.com/.extraheader
2025-08-14T23:55:08.0461738Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2025-08-14T23:55:08.0520817Z http.https://github.com/.extraheader
2025-08-14T23:55:08.0941368Z Cleaning up orphan processes