2025-08-14T21:20:39.6096222Z Current runner version: '2.327.1' 2025-08-14T21:20:39.6101468Z Runner name: 'gpu9498' 2025-08-14T21:20:39.6102149Z Runner group name: 'linux.rocm.gpu.group' 2025-08-14T21:20:39.6102918Z Machine name: 'gpu9498' 2025-08-14T21:20:39.6105433Z ##[group]GITHUB_TOKEN Permissions 2025-08-14T21:20:39.6107175Z Contents: read 2025-08-14T21:20:39.6107748Z Metadata: read 2025-08-14T21:20:39.6108197Z ##[endgroup] 2025-08-14T21:20:39.6109902Z Secret source: Actions 2025-08-14T21:20:39.6110575Z Prepare workflow directory 2025-08-14T21:20:40.2820833Z Prepare all required actions 2025-08-14T21:20:40.2870855Z Getting action download info 2025-08-14T21:20:40.6907116Z Download action repository 'pytorch/pytorch@main' (SHA:3be70dc30e893b552fc0f23ca06cd8f7949b6d08) 2025-08-14T21:20:45.9033125Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-08-14T21:20:46.7475599Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-08-14T21:20:47.5743726Z Download action repository 'pytorch/test-infra@main' (SHA:83f58f391e939c10dcb8cb6d745e4cefa3b98a84) 2025-08-14T21:20:48.8704450Z Download action repository 'actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-08-14T21:20:50.0061069Z Getting action download info 2025-08-14T21:20:50.1632274Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-08-14T21:20:50.9238733Z Getting action download info 2025-08-14T21:20:51.0875828Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-08-14T21:20:51.7996917Z Getting action download info 2025-08-14T21:20:51.9795206Z Uses: pytorch/pytorch/.github/workflows/_rocm-test.yml@refs/heads/main (1fc683cf17c8c673044538d10266c00f92987be2) 2025-08-14T21:20:51.9803033Z ##[group] Inputs 2025-08-14T21:20:51.9803652Z build-environment: linux-jammy-rocm-py3.10 2025-08-14T21:20:51.9806224Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]} 2025-08-14T21:20:51.9809309Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:20:51.9810556Z sync-tag: 2025-08-14T21:20:51.9811833Z timeout-minutes: 300 2025-08-14T21:20:51.9812280Z tests-to-include: 2025-08-14T21:20:51.9812675Z dashboard-tag: 2025-08-14T21:20:51.9814286Z disable-monitor: true 2025-08-14T21:20:51.9814831Z monitor-log-interval: 5 2025-08-14T21:20:51.9815336Z monitor-data-collect-interval: 1 2025-08-14T21:20:51.9815871Z ##[endgroup] 2025-08-14T21:20:51.9816769Z Complete job name: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:20:52.1526606Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-08-14T21:20:52.1527216Z with: 2025-08-14T21:20:52.1527382Z no-sudo: true 2025-08-14T21:20:52.1527581Z submodules: recursive 2025-08-14T21:20:52.1527786Z fetch-depth: 0 2025-08-14T21:20:52.1528097Z env: 2025-08-14T21:20:52.1528257Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:20:52.1528447Z ##[endgroup] 2025-08-14T21:20:52.1599835Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:20:52.1600712Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:20:52.1629805Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:20:52.1630422Z env: 2025-08-14T21:20:52.1630585Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:20:52.1630786Z ##[endgroup] 2025-08-14T21:20:52.1960594Z ##[group]Run # Use all available CPUs for fetching 2025-08-14T21:20:52.1961413Z # Use all available CPUs for fetching 2025-08-14T21:20:52.1962052Z cd "${GITHUB_WORKSPACE}" 2025-08-14T21:20:52.1962650Z git config --global fetch.parallel 0 2025-08-14T21:20:52.1963308Z git config --global submodule.fetchJobs 0 2025-08-14T21:20:52.1963882Z  2025-08-14T21:20:52.1964485Z # Clean workspace. The default checkout action should also do this, but 2025-08-14T21:20:52.1965293Z # do it here as well just in case 2025-08-14T21:20:52.1965823Z if [[ -d .git ]]; then 2025-08-14T21:20:52.1966316Z  if [ -z "${NO_SUDO}" ]; then 2025-08-14T21:20:52.1966827Z  sudo git clean -ffdx 2025-08-14T21:20:52.1967288Z  else 2025-08-14T21:20:52.1967698Z  git clean -ffdx 2025-08-14T21:20:52.1968119Z  fi 2025-08-14T21:20:52.1968468Z fi 2025-08-14T21:20:52.2019362Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:20:52.2019976Z env: 2025-08-14T21:20:52.2020309Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:20:52.2020695Z NO_SUDO: true 2025-08-14T21:20:52.2021143Z ##[endgroup] 2025-08-14T21:20:52.7788819Z Removing .additional_ci_files/ 2025-08-14T21:20:52.7789254Z Removing build/ 2025-08-14T21:20:52.7789489Z Removing dist/ 2025-08-14T21:20:52.7789703Z Removing test/test-reports/ 2025-08-14T21:20:52.7890796Z ##[group]Run actions/checkout@v4 2025-08-14T21:20:52.7891321Z with: 2025-08-14T21:20:52.7891757Z ref: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:20:52.7892332Z fetch-depth: 0 2025-08-14T21:20:52.7892728Z submodules: recursive 2025-08-14T21:20:52.7893172Z show-progress: false 2025-08-14T21:20:52.7893614Z repository: pytorch/pytorch 2025-08-14T21:20:52.7894361Z token: *** 2025-08-14T21:20:52.7894811Z ssh-strict: true 2025-08-14T21:20:52.7895279Z ssh-user: git 2025-08-14T21:20:52.7895791Z persist-credentials: true 2025-08-14T21:20:52.7896343Z clean: true 2025-08-14T21:20:52.7896837Z sparse-checkout-cone-mode: true 2025-08-14T21:20:52.7897423Z fetch-tags: false 2025-08-14T21:20:52.7897872Z lfs: false 2025-08-14T21:20:52.7898307Z set-safe-directory: true 2025-08-14T21:20:52.7898817Z env: 2025-08-14T21:20:52.7899201Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:20:52.7899521Z ##[endgroup] 2025-08-14T21:20:52.9263987Z Syncing repository: pytorch/pytorch 2025-08-14T21:20:52.9266396Z ##[group]Getting Git version info 2025-08-14T21:20:52.9267255Z Working directory is '/var/home/pytorchci/actions-runner/_work/pytorch/pytorch' 2025-08-14T21:20:52.9268385Z [command]/usr/bin/git version 2025-08-14T21:20:52.9268824Z git version 2.34.1 2025-08-14T21:20:52.9270228Z ##[endgroup] 2025-08-14T21:20:52.9276347Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/23192730-ab08-452b-948d-06e0ed69e6ad/.gitconfig' 2025-08-14T21:20:52.9278831Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/23192730-ab08-452b-948d-06e0ed69e6ad' before making global git config changes 2025-08-14T21:20:52.9280529Z Adding repository directory to the temporary git global config as a safe directory 2025-08-14T21:20:52.9281824Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2025-08-14T21:20:52.9346844Z [command]/usr/bin/git config --local --get remote.origin.url 2025-08-14T21:20:52.9395067Z https://github.com/pytorch/pytorch 2025-08-14T21:20:52.9417518Z ##[group]Removing previously created refs, to avoid conflicts 2025-08-14T21:20:52.9420190Z [command]/usr/bin/git rev-parse --symbolic-full-name --verify --quiet HEAD 2025-08-14T21:20:52.9468422Z HEAD 2025-08-14T21:20:52.9546206Z ##[endgroup] 2025-08-14T21:20:52.9550837Z [command]/usr/bin/git submodule status 2025-08-14T21:20:53.0329158Z 7e1e1fe3858c63c251c637ae41a20de425dde96f android/libs/fbjni (v0.1.0-12-g7e1e1fe) 2025-08-14T21:20:53.0560910Z 4dfe081cf6bcd15db339cf2680b9281b8451eeb3 third_party/FP16 (4dfe081) 2025-08-14T21:20:53.0785297Z b408327ac2a15ec3e43352421954f5b1967701d1 third_party/FXdiv (b408327) 2025-08-14T21:20:53.1037119Z c07e3a0400713d546e0dea2d5466dd22ea389c73 third_party/NNPACK (c07e3a0) 2025-08-14T21:20:53.1163869Z 2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07 third_party/NVTX (v3.1.0-263-g2942f16) 2025-08-14T21:20:53.1364635Z 1d8f600fd424278486eade7ed3e877c99f0846b1 third_party/VulkanMemoryAllocator (v2.1.0-982-g1d8f600) 2025-08-14T21:20:53.2230316Z 51a0103656eff6fc9bfd39a4597923c4b542c883 third_party/XNNPACK (remotes/origin/ds/ndk-1243-g51a0103656) 2025-08-14T21:20:53.2321174Z 01aae101b9e5e94d6c16a9514c9fb8df99c93150 third_party/aiter (v0.1.1-92-g01aae101) 2025-08-14T21:20:53.2390896Z 299e5928955cc62af9968370293b916f5130916f third_party/benchmark (v1.9.3) 2025-08-14T21:20:53.2619647Z 7fe50dc3da2069d6645d9deb8c017a876472a977 third_party/composable_kernel (rocm-6.4.2-459-g7fe50dc3d) 2025-08-14T21:20:53.3028982Z 3af7f2c16147f3fbc6e4d717032daf505dc1652c third_party/cpp-httplib (v0.20.1) 2025-08-14T21:20:53.3237523Z 5e3d2445e6a84d9599bee2bf78edbb4d80865e1d third_party/cpuinfo (5e3d244) 2025-08-14T21:20:53.3333851Z f937055efc6d414d11f4c6577e3977fe74f35fb6 third_party/cudnn_frontend (v0.5-52-gf937055) 2025-08-14T21:20:53.3573514Z e51efbfe18fe4f4cbb66ab814c55bf4aa0185491 third_party/cutlass (v4.1.0) 2025-08-14T21:20:53.3695369Z 21c7d30c526c0f1ad873ecc632dca6cfa8a69067 third_party/fbgemm (v1.3.0-rc1-165-g21c7d30c) 2025-08-14T21:20:53.3907934Z 979702c87a8713a8e0a5e9fee122b90d2ef13be5 third_party/flash-attention (v2.7.4) 2025-08-14T21:20:53.3976712Z a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757 third_party/flatbuffers (v24.12.23) 2025-08-14T21:20:53.4717234Z 40626af88bd7df9a5fb80be7b25ac85b122d6c21 third_party/fmt (11.2.0) 2025-08-14T21:20:53.5002862Z 3fb5c176c17c765a3492cd2f0321b0dab712f350 third_party/gemmlowp/gemmlowp (remotes/origin/revert-87-master-135-g3fb5c17) 2025-08-14T21:20:53.5307651Z c7b7b022c124d9643957d9bd55f57ac59fce8fa2 third_party/gloo (remotes/origin/gh/c-p-i-o/1/base-33-gc7b7b02) 2025-08-14T21:20:53.5780605Z 52eb8108c5bdec04579160ae17225d66034bd723 third_party/googletest (release-1.8.0-3544-g52eb8108) 2025-08-14T21:20:53.5984974Z 719d8e6cd7f7a0e01b155657526d693acf97c2b3 third_party/ideep (pytorch-rls-v3.7.1) 2025-08-14T21:20:53.6150234Z dec1d23ca65ab069d225dfe40dea14f455170959 third_party/ittapi (v3.25.5) 2025-08-14T21:20:53.6748805Z 5e7501833f1021ce6f618572d3baf657b6319658 third_party/kineto (remotes/origin/sraikund/test-98-g5e75018) 2025-08-14T21:20:53.6818327Z cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7 third_party/kleidiai (v1.8.0) 2025-08-14T21:20:53.6885727Z fbd8b99c2b828428947d70fdc046bb55609be93e third_party/mimalloc (v2.2.4) 2025-08-14T21:20:53.6955369Z 55f93686c01528224f448c19128836e7df245f72 third_party/nlohmann (v3.12.0) 2025-08-14T21:20:53.7591228Z e709452ef2bbc1d113faf678c24e6d3467696e83 third_party/onnx (v1.18.0) 2025-08-14T21:20:53.7660646Z a799f4aed9c94b765dcdaabaeab7d5e7e2310878 third_party/opentelemetry-cpp (v1.14.2) 2025-08-14T21:20:53.7736884Z 0fa0ef591e38c2758e3184c6c23e497b9f732ffa third_party/pocketfft (release_for_eigen-40-g0fa0ef5) 2025-08-14T21:20:53.8442264Z d1eca4e4b421cd2997495c4b4e65cea6be4e9b8a third_party/protobuf (v3.7.0-rc.2-1279-gd1eca4e4b) 2025-08-14T21:20:53.8657677Z 072586a71b55b7f8c584153d223e95687148a900 third_party/psimd (heads/master) 2025-08-14T21:20:53.8813440Z 4fe0e1e183925bf8cfa6aae24237e724a96479b8 third_party/pthreadpool (0.1-144-g4fe0e1e) 2025-08-14T21:20:53.8941440Z a2e59f0e7065404b44dfe92a28aca47ba1378dc4 third_party/pybind11 (v2.11.0-182-ga2e59f0e) 2025-08-14T21:20:53.9149194Z f45429b087dd7d5bc78bb40dc7cf06425c252d67 third_party/python-peachpy (remotes/origin/pre-generated) 2025-08-14T21:20:53.9348707Z 5a1d179df9cf652951b59010a2d2075372d67f68 third_party/sleef (3.8) 2025-08-14T21:20:53.9568161Z dacda0567d9f23d4bc503e1c4f84aa65f33ac38a third_party/tensorpipe (remotes/origin/HEAD) 2025-08-14T21:20:53.9598363Z ##[group]Cleaning the repository 2025-08-14T21:20:53.9605162Z [command]/usr/bin/git clean -ffdx 2025-08-14T21:20:54.0280547Z [command]/usr/bin/git reset --hard HEAD 2025-08-14T21:20:54.1676117Z HEAD is now at 6c05ea6475b [DTensor] add op support: aten.squeeze_.dim (#159532) 2025-08-14T21:20:54.1728332Z ##[endgroup] 2025-08-14T21:20:54.1731065Z ##[group]Disabling automatic garbage collection 2025-08-14T21:20:54.1741463Z [command]/usr/bin/git config --local gc.auto 0 2025-08-14T21:20:54.1817335Z ##[endgroup] 2025-08-14T21:20:54.1818266Z ##[group]Setting up auth 2025-08-14T21:20:54.1828672Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-14T21:20:54.1928286Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-14T21:20:54.2648799Z Entering 'android/libs/fbjni' 2025-08-14T21:20:54.2771552Z Entering 'third_party/FP16' 2025-08-14T21:20:54.2904387Z Entering 'third_party/FXdiv' 2025-08-14T21:20:54.3029233Z Entering 'third_party/NNPACK' 2025-08-14T21:20:54.3154920Z Entering 'third_party/NVTX' 2025-08-14T21:20:54.3283104Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:20:54.3407457Z Entering 'third_party/XNNPACK' 2025-08-14T21:20:54.3565732Z Entering 'third_party/aiter' 2025-08-14T21:20:54.3687636Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:20:54.3827970Z Entering 'third_party/benchmark' 2025-08-14T21:20:54.3951512Z Entering 'third_party/composable_kernel' 2025-08-14T21:20:54.4097202Z Entering 'third_party/cpp-httplib' 2025-08-14T21:20:54.4222160Z Entering 'third_party/cpuinfo' 2025-08-14T21:20:54.4345543Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:20:54.4468582Z Entering 'third_party/cutlass' 2025-08-14T21:20:54.4614889Z Entering 'third_party/fbgemm' 2025-08-14T21:20:54.4742078Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:20:54.4858601Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:20:54.4991971Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:20:54.5107817Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:20:54.5242954Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:20:54.5357185Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:20:54.5470779Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:20:54.5594492Z Entering 'third_party/flash-attention' 2025-08-14T21:20:54.5714278Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:20:54.5844479Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:20:54.5982509Z Entering 'third_party/flatbuffers' 2025-08-14T21:20:54.6109993Z Entering 'third_party/fmt' 2025-08-14T21:20:54.6232316Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:20:54.6354682Z Entering 'third_party/gloo' 2025-08-14T21:20:54.6477275Z Entering 'third_party/googletest' 2025-08-14T21:20:54.6598418Z Entering 'third_party/ideep' 2025-08-14T21:20:54.6713399Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:20:54.6848635Z Entering 'third_party/ittapi' 2025-08-14T21:20:54.6970042Z Entering 'third_party/kineto' 2025-08-14T21:20:54.7087199Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:20:54.7198500Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:20:54.7316090Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:20:54.7429395Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:20:54.7544310Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:20:54.7656497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:20:54.7779333Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:20:54.7893555Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:20:54.8009257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:20:54.8126067Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:20:54.8250434Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:20:54.8364975Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:20:54.8487727Z Entering 'third_party/kleidiai' 2025-08-14T21:20:54.8611876Z Entering 'third_party/mimalloc' 2025-08-14T21:20:54.8735805Z Entering 'third_party/nlohmann' 2025-08-14T21:20:54.8864143Z Entering 'third_party/onnx' 2025-08-14T21:20:54.9021163Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:20:54.9150932Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:20:54.9271060Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:20:54.9383391Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:20:54.9499658Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:20:54.9611473Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:20:54.9727928Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:20:54.9841585Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:20:54.9956496Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:20:55.0073609Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:20:55.0195621Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:20:55.0322722Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:20:55.0488948Z Entering 'third_party/pocketfft' 2025-08-14T21:20:55.0610375Z Entering 'third_party/protobuf' 2025-08-14T21:20:55.0736866Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:20:55.0853808Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:20:55.0977296Z Entering 'third_party/psimd' 2025-08-14T21:20:55.1099341Z Entering 'third_party/pthreadpool' 2025-08-14T21:20:55.1220811Z Entering 'third_party/pybind11' 2025-08-14T21:20:55.1342955Z Entering 'third_party/python-peachpy' 2025-08-14T21:20:55.1462783Z Entering 'third_party/sleef' 2025-08-14T21:20:55.1583238Z Entering 'third_party/tensorpipe' 2025-08-14T21:20:55.1700942Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:20:55.1814116Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:20:55.1926027Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:20:55.2038622Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:20:55.2150953Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:20:55.2322390Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-14T21:20:55.2403464Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-14T21:20:55.3079451Z Entering 'android/libs/fbjni' 2025-08-14T21:20:55.3201180Z Entering 'third_party/FP16' 2025-08-14T21:20:55.3321271Z Entering 'third_party/FXdiv' 2025-08-14T21:20:55.3446467Z Entering 'third_party/NNPACK' 2025-08-14T21:20:55.3566920Z Entering 'third_party/NVTX' 2025-08-14T21:20:55.3688541Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:20:55.3807452Z Entering 'third_party/XNNPACK' 2025-08-14T21:20:55.3960934Z Entering 'third_party/aiter' 2025-08-14T21:20:55.4078602Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:20:55.4223677Z Entering 'third_party/benchmark' 2025-08-14T21:20:55.4345935Z Entering 'third_party/composable_kernel' 2025-08-14T21:20:55.4490083Z Entering 'third_party/cpp-httplib' 2025-08-14T21:20:55.4610563Z Entering 'third_party/cpuinfo' 2025-08-14T21:20:55.4732692Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:20:55.4854126Z Entering 'third_party/cutlass' 2025-08-14T21:20:55.4993911Z Entering 'third_party/fbgemm' 2025-08-14T21:20:55.5116101Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:20:55.5229436Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:20:55.5359760Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:20:55.5473956Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:20:55.5609178Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:20:55.5725618Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:20:55.5836564Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:20:55.5959091Z Entering 'third_party/flash-attention' 2025-08-14T21:20:55.6079676Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:20:55.6209001Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:20:55.6347126Z Entering 'third_party/flatbuffers' 2025-08-14T21:20:55.6475256Z Entering 'third_party/fmt' 2025-08-14T21:20:55.6596865Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:20:55.6716527Z Entering 'third_party/gloo' 2025-08-14T21:20:55.6834536Z Entering 'third_party/googletest' 2025-08-14T21:20:55.6953137Z Entering 'third_party/ideep' 2025-08-14T21:20:55.7070419Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:20:55.7206379Z Entering 'third_party/ittapi' 2025-08-14T21:20:55.7327868Z Entering 'third_party/kineto' 2025-08-14T21:20:55.7450999Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:20:55.7558467Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:20:55.7677852Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:20:55.7790631Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:20:55.7904876Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:20:55.8013897Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:20:55.8133982Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:20:55.8249264Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:20:55.8369225Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:20:55.8481690Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:20:55.8608930Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:20:55.8727035Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:20:55.8847428Z Entering 'third_party/kleidiai' 2025-08-14T21:20:55.8971849Z Entering 'third_party/mimalloc' 2025-08-14T21:20:55.9091870Z Entering 'third_party/nlohmann' 2025-08-14T21:20:55.9216683Z Entering 'third_party/onnx' 2025-08-14T21:20:55.9370812Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:20:55.9499128Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:20:55.9619441Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:20:55.9733451Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:20:55.9848424Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:20:55.9968281Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:20:56.0081160Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:20:56.0193003Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:20:56.0305185Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:20:56.0412353Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:20:56.0529650Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:20:56.0646954Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:20:56.0805237Z Entering 'third_party/pocketfft' 2025-08-14T21:20:56.0924168Z Entering 'third_party/protobuf' 2025-08-14T21:20:56.1046384Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:20:56.1156667Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:20:56.1273468Z Entering 'third_party/psimd' 2025-08-14T21:20:56.1393604Z Entering 'third_party/pthreadpool' 2025-08-14T21:20:56.1514736Z Entering 'third_party/pybind11' 2025-08-14T21:20:56.1634651Z Entering 'third_party/python-peachpy' 2025-08-14T21:20:56.1752922Z Entering 'third_party/sleef' 2025-08-14T21:20:56.1872081Z Entering 'third_party/tensorpipe' 2025-08-14T21:20:56.1987660Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:20:56.2100895Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:20:56.2210178Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:20:56.2319509Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:20:56.2424359Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:20:56.2594621Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:20:56.2693350Z ##[endgroup] 2025-08-14T21:20:56.2694163Z ##[group]Fetching the repository 2025-08-14T21:20:56.2709565Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-08-14T21:20:57.0979695Z From https://github.com/pytorch/pytorch 2025-08-14T21:20:57.0980515Z - [deleted] (none) -> ciflow/inductor/155263 2025-08-14T21:20:57.1673111Z - [deleted] (none) -> ciflow/inductor/159530 2025-08-14T21:20:57.1673967Z - [deleted] (none) -> ciflow/inductor/160000 2025-08-14T21:20:57.1674726Z - [deleted] (none) -> ciflow/inductor/160605 2025-08-14T21:20:57.1675510Z - [deleted] (none) -> ciflow/inductor/160645 2025-08-14T21:20:57.1676979Z - [deleted] (none) -> ciflow/trunk/155263 2025-08-14T21:20:57.1678159Z - [deleted] (none) -> ciflow/trunk/155649 2025-08-14T21:20:57.1679619Z - [deleted] (none) -> ciflow/trunk/158137 2025-08-14T21:20:57.1681293Z - [deleted] (none) -> ciflow/trunk/159530 2025-08-14T21:20:57.1682193Z - [deleted] (none) -> ciflow/trunk/160000 2025-08-14T21:20:57.1684871Z - [deleted] (none) -> ciflow/trunk/160008 2025-08-14T21:20:57.1685511Z - [deleted] (none) -> ciflow/trunk/160562 2025-08-14T21:20:57.1686109Z - [deleted] (none) -> ciflow/trunk/160605 2025-08-14T21:20:57.1687496Z - [deleted] (none) -> ciflow/trunk/160636 2025-08-14T21:20:57.1689195Z - [deleted] (none) -> ciflow/trunk/160643 2025-08-14T21:20:57.1690307Z - [deleted] (none) -> ciflow/trunk/160645 2025-08-14T21:20:58.9915107Z 78538ad3e12..c4cb9008294 autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr 2025-08-14T21:20:59.0056875Z + 18b763f5271...4bc85c55ba9 exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model (forced update) 2025-08-14T21:20:59.0138967Z * [new branch] gh/eellison/813/base -> origin/gh/eellison/813/base 2025-08-14T21:20:59.0140994Z * [new branch] gh/eellison/813/head -> origin/gh/eellison/813/head 2025-08-14T21:20:59.0142135Z * [new branch] gh/eellison/813/orig -> origin/gh/eellison/813/orig 2025-08-14T21:20:59.0170960Z 03f2c0ce4ea..8dcf14991ff gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-08-14T21:20:59.0176144Z + bcac008b1bc...9a2c169f560 gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig (forced update) 2025-08-14T21:20:59.0183625Z 6646c3dfa98..e10ecdec4fc gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base 2025-08-14T21:20:59.0185932Z a0786e4dfb6..44585d2f482 gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head 2025-08-14T21:20:59.0192363Z + 7e281b54e69...9e777070cd7 gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig (forced update) 2025-08-14T21:20:59.0197541Z 0e2e4700677..c7bdd8517c7 gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base 2025-08-14T21:20:59.0202232Z bc418a1823f..c36bdef982a gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head 2025-08-14T21:20:59.0207502Z + 339f2ac02fc...65bcf0cf22c gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig (forced update) 2025-08-14T21:20:59.0212260Z 9f7ab8d153b..37addbd5732 gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base 2025-08-14T21:20:59.0217662Z cbb9ebfa7db..d6b43a6cd4e gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head 2025-08-14T21:20:59.0219806Z + 3030a7fb87f...0768ffd59f3 gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig (forced update) 2025-08-14T21:20:59.0222334Z 329c5274318..87732b2c98e gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base 2025-08-14T21:20:59.0227712Z a2ed4621e3c..5556c80b778 gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head 2025-08-14T21:20:59.0232507Z + c1fdf0419c3...c6b3f286785 gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig (forced update) 2025-08-14T21:20:59.0236703Z 14a3891c9ab..e4dcf56a427 gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base 2025-08-14T21:20:59.0242305Z f9eca810cbb..2f44712cf54 gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head 2025-08-14T21:20:59.0247010Z + e6b771a3a3a...77fbab074fc gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig (forced update) 2025-08-14T21:20:59.0252285Z 60127e9851d..3d434480221 gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base 2025-08-14T21:20:59.0257325Z 49a567502c2..347333b3995 gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head 2025-08-14T21:20:59.0262981Z + 37072dc6925...9ee0a5f1e67 gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig (forced update) 2025-08-14T21:20:59.0265923Z 02f81cd7750..72cea7eea1f gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base 2025-08-14T21:20:59.0268393Z 7f0483b79d9..64b4dccb023 gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head 2025-08-14T21:20:59.0272749Z + b39476f5c91...03b5add0bd3 gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig (forced update) 2025-08-14T21:20:59.0277512Z cc6f595d52d..2a152d3688b gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base 2025-08-14T21:20:59.0282680Z dcba0f797c9..9ba4773891f gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head 2025-08-14T21:20:59.0287259Z + a07cf993edd...dbe36858fef gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig (forced update) 2025-08-14T21:20:59.0355107Z ea5b50f2a44..42e61c43417 gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-08-14T21:20:59.0359310Z b87cd6dc372..bf07dcc2698 gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-08-14T21:20:59.0364639Z + 16c7cc80222...35ce16d727d gh/xmfan/274/orig -> origin/gh/xmfan/274/orig (forced update) 2025-08-14T21:20:59.0368892Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-08-14T21:20:59.0371619Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-08-14T21:20:59.0374195Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-08-14T21:20:59.0381340Z 7f5d2fe3aef..16b50f866ad gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-08-14T21:20:59.0385269Z 97d57cea8ee..14ac4b0cd8e gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-08-14T21:20:59.0389310Z + 0a0e2ec5444...196b71c5f3f gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig (forced update) 2025-08-14T21:20:59.0393602Z 4576f5bc30f..66605343d69 gh/yangw-dev/17/base -> origin/gh/yangw-dev/17/base 2025-08-14T21:20:59.0397800Z b56f5705a0e..44feacc104c gh/yangw-dev/17/head -> origin/gh/yangw-dev/17/head 2025-08-14T21:20:59.0402171Z + 44b8e361b49...cd4ff57277f gh/yangw-dev/17/orig -> origin/gh/yangw-dev/17/orig (forced update) 2025-08-14T21:20:59.0406368Z 4bb8d6ee236..8e0d31dd9a7 gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-08-14T21:20:59.0410563Z ff64e4ec359..7f2192ddf6c gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-08-14T21:20:59.0414637Z + 47aa0c2599e...c241d1d9683 gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig (forced update) 2025-08-14T21:20:59.0419719Z b74d09ea9cb..c6f893ae6a2 gh/ydwu4/253/base -> origin/gh/ydwu4/253/base 2025-08-14T21:20:59.0423959Z 343c95790d8..ec23555a649 gh/ydwu4/253/head -> origin/gh/ydwu4/253/head 2025-08-14T21:20:59.0428032Z + 46a1a2cb3f1...42603577470 gh/ydwu4/253/orig -> origin/gh/ydwu4/253/orig (forced update) 2025-08-14T21:20:59.0435158Z ccedc96dd99..4c947d82af4 gh/ydwu4/289/base -> origin/gh/ydwu4/289/base 2025-08-14T21:20:59.0439235Z 5635e90ad37..9faad049c4e gh/ydwu4/289/head -> origin/gh/ydwu4/289/head 2025-08-14T21:20:59.0443789Z + b9dc464b082...249b3f1a4cc gh/ydwu4/289/orig -> origin/gh/ydwu4/289/orig (forced update) 2025-08-14T21:20:59.0448016Z c6f4bbfe24a..56500da2a46 gh/ydwu4/290/base -> origin/gh/ydwu4/290/base 2025-08-14T21:20:59.0452572Z fb52b34d9d8..a9f4e34e95a gh/ydwu4/290/head -> origin/gh/ydwu4/290/head 2025-08-14T21:20:59.0457426Z + 4aee28d7cba...84588c9cecc gh/ydwu4/290/orig -> origin/gh/ydwu4/290/orig (forced update) 2025-08-14T21:20:59.0459734Z eafc551da2d..77fe4d9d877 gh/ydwu4/291/base -> origin/gh/ydwu4/291/base 2025-08-14T21:20:59.0462276Z 1eff640ef2a..884e299f170 gh/ydwu4/291/head -> origin/gh/ydwu4/291/head 2025-08-14T21:20:59.0464442Z + 535f628315d...561d588cae4 gh/ydwu4/291/orig -> origin/gh/ydwu4/291/orig (forced update) 2025-08-14T21:20:59.0466705Z 95767ac464a..047f5842de9 gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-08-14T21:20:59.0469176Z 855491e7d90..ca7a5d62924 gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-08-14T21:20:59.0471380Z + 281c90bb098...59233551ea9 gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig (forced update) 2025-08-14T21:20:59.0473976Z 90531c229e5..deb9cd690fc gh/ydwu4/293/base -> origin/gh/ydwu4/293/base 2025-08-14T21:20:59.0476223Z 0250adb10dc..5e052917f0f gh/ydwu4/293/head -> origin/gh/ydwu4/293/head 2025-08-14T21:20:59.0478141Z + 30b21ad5427...ffdc931dbb5 gh/ydwu4/293/orig -> origin/gh/ydwu4/293/orig (forced update) 2025-08-14T21:20:59.0480285Z 8d1de9cf2a4..37e3590e3f3 gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-08-14T21:20:59.0482521Z 3ec3a23927b..b43dcce75fd gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-08-14T21:20:59.0484490Z + b3ae71c07a2...b3c1981a98a gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig (forced update) 2025-08-14T21:20:59.0486875Z dda63ad5505..dcfb17c44e4 gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-08-14T21:20:59.0488242Z f5a3014ee02..19f4a9b32ef gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-08-14T21:20:59.0490159Z + 9b776485c6b...9ee4bfc6f03 gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig (forced update) 2025-08-14T21:20:59.0492447Z fc25c68f20f..c107eb8fa07 gh/ydwu4/301/base -> origin/gh/ydwu4/301/base 2025-08-14T21:20:59.0494444Z 2b889529039..d53b6450b00 gh/ydwu4/301/head -> origin/gh/ydwu4/301/head 2025-08-14T21:20:59.0496571Z + 3e899b2ce51...1512814970a gh/ydwu4/301/orig -> origin/gh/ydwu4/301/orig (forced update) 2025-08-14T21:20:59.0498763Z 2b889529039..b0a7fdb5401 gh/ydwu4/305/base -> origin/gh/ydwu4/305/base 2025-08-14T21:20:59.0500780Z 7eeafe602af..538a08b69f2 gh/ydwu4/305/head -> origin/gh/ydwu4/305/head 2025-08-14T21:20:59.0502924Z + 4d734258b6d...48e3105c227 gh/ydwu4/305/orig -> origin/gh/ydwu4/305/orig (forced update) 2025-08-14T21:20:59.0504800Z 867bd790e14..e8622caa50f gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-08-14T21:20:59.0506684Z 9973dd5f806..14d2597c77a gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-08-14T21:20:59.0508720Z + 1d368e3911c...66680d2c273 gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig (forced update) 2025-08-14T21:20:59.0510553Z 9973dd5f806..a42184b16d8 gh/ydwu4/307/base -> origin/gh/ydwu4/307/base 2025-08-14T21:20:59.0512405Z 9fa0484136c..a48d6f473d9 gh/ydwu4/307/head -> origin/gh/ydwu4/307/head 2025-08-14T21:20:59.0514445Z + 7e922ff99d8...dd60840ccfd gh/ydwu4/307/orig -> origin/gh/ydwu4/307/orig (forced update) 2025-08-14T21:20:59.0516509Z 12913a9a722..39133df24c2 gh/ydwu4/308/head -> origin/gh/ydwu4/308/head 2025-08-14T21:20:59.0518373Z + cee669623d0...279de9fc5c6 gh/ydwu4/308/orig -> origin/gh/ydwu4/308/orig (forced update) 2025-08-14T21:20:59.0520497Z * [new branch] gh/ydwu4/309/base -> origin/gh/ydwu4/309/base 2025-08-14T21:20:59.0522442Z * [new branch] gh/ydwu4/309/head -> origin/gh/ydwu4/309/head 2025-08-14T21:20:59.0523524Z * [new branch] gh/ydwu4/309/orig -> origin/gh/ydwu4/309/orig 2025-08-14T21:20:59.0524945Z * [new branch] gh/ydwu4/310/base -> origin/gh/ydwu4/310/base 2025-08-14T21:20:59.0526143Z * [new branch] gh/ydwu4/310/head -> origin/gh/ydwu4/310/head 2025-08-14T21:20:59.0527447Z * [new branch] gh/ydwu4/310/orig -> origin/gh/ydwu4/310/orig 2025-08-14T21:20:59.0529198Z * [new branch] gh/ydwu4/311/base -> origin/gh/ydwu4/311/base 2025-08-14T21:20:59.0530639Z * [new branch] gh/ydwu4/311/head -> origin/gh/ydwu4/311/head 2025-08-14T21:20:59.0531693Z * [new branch] gh/ydwu4/311/orig -> origin/gh/ydwu4/311/orig 2025-08-14T21:20:59.0540549Z 65053c03a3d..3be70dc30e8 main -> origin/main 2025-08-14T21:20:59.0546174Z + e33a33da492...8ac9fe6cbf8 mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 (forced update) 2025-08-14T21:20:59.0555238Z + fd5bdcd271d...0203223a156 update_submodule_FBGEMM -> origin/update_submodule_FBGEMM (forced update) 2025-08-14T21:20:59.0557101Z e4de93f6a3e..7e27347fd35 viable/strict -> origin/viable/strict 2025-08-14T21:20:59.0562820Z t [tag update] ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562 2025-08-14T21:20:59.0564817Z t [tag update] ciflow/inductor/153966 -> ciflow/inductor/153966 2025-08-14T21:20:59.0566096Z t [tag update] ciflow/inductor/154193 -> ciflow/inductor/154193 2025-08-14T21:20:59.0567449Z * [new tag] ciflow/inductor/155958 -> ciflow/inductor/155958 2025-08-14T21:20:59.0568490Z * [new tag] ciflow/inductor/156851 -> ciflow/inductor/156851 2025-08-14T21:20:59.0571082Z t [tag update] ciflow/inductor/159009 -> ciflow/inductor/159009 2025-08-14T21:20:59.0572091Z t [tag update] ciflow/inductor/159010 -> ciflow/inductor/159010 2025-08-14T21:20:59.0572985Z * [new tag] ciflow/inductor/159365 -> ciflow/inductor/159365 2025-08-14T21:20:59.0574296Z t [tag update] ciflow/inductor/159366 -> ciflow/inductor/159366 2025-08-14T21:20:59.0575672Z t [tag update] ciflow/inductor/159367 -> ciflow/inductor/159367 2025-08-14T21:20:59.0576914Z t [tag update] ciflow/inductor/159368 -> ciflow/inductor/159368 2025-08-14T21:20:59.0578344Z t [tag update] ciflow/inductor/159483 -> ciflow/inductor/159483 2025-08-14T21:20:59.0579976Z t [tag update] ciflow/inductor/159864 -> ciflow/inductor/159864 2025-08-14T21:20:59.0581274Z t [tag update] ciflow/inductor/159865 -> ciflow/inductor/159865 2025-08-14T21:20:59.0582773Z t [tag update] ciflow/inductor/159875 -> ciflow/inductor/159875 2025-08-14T21:20:59.0584044Z t [tag update] ciflow/inductor/159902 -> ciflow/inductor/159902 2025-08-14T21:20:59.0585598Z t [tag update] ciflow/inductor/159923 -> ciflow/inductor/159923 2025-08-14T21:20:59.0588013Z t [tag update] ciflow/inductor/160201 -> ciflow/inductor/160201 2025-08-14T21:20:59.0589932Z t [tag update] ciflow/inductor/160374 -> ciflow/inductor/160374 2025-08-14T21:20:59.0591798Z t [tag update] ciflow/inductor/160461 -> ciflow/inductor/160461 2025-08-14T21:20:59.0593178Z t [tag update] ciflow/inductor/160467 -> ciflow/inductor/160467 2025-08-14T21:20:59.0594432Z t [tag update] ciflow/inductor/160476 -> ciflow/inductor/160476 2025-08-14T21:20:59.0595900Z t [tag update] ciflow/inductor/160483 -> ciflow/inductor/160483 2025-08-14T21:20:59.0597867Z t [tag update] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-08-14T21:20:59.0599719Z t [tag update] ciflow/inductor/160540 -> ciflow/inductor/160540 2025-08-14T21:20:59.0601278Z t [tag update] ciflow/inductor/160548 -> ciflow/inductor/160548 2025-08-14T21:20:59.0602558Z t [tag update] ciflow/inductor/160583 -> ciflow/inductor/160583 2025-08-14T21:20:59.0604118Z t [tag update] ciflow/inductor/160592 -> ciflow/inductor/160592 2025-08-14T21:20:59.0605690Z t [tag update] ciflow/inductor/160635 -> ciflow/inductor/160635 2025-08-14T21:20:59.0606904Z * [new tag] ciflow/inductor/160668 -> ciflow/inductor/160668 2025-08-14T21:20:59.0607777Z * [new tag] ciflow/inductor/160669 -> ciflow/inductor/160669 2025-08-14T21:20:59.0608833Z * [new tag] ciflow/inductor/160670 -> ciflow/inductor/160670 2025-08-14T21:20:59.0610072Z * [new tag] ciflow/inductor/160671 -> ciflow/inductor/160671 2025-08-14T21:20:59.0611243Z * [new tag] ciflow/inductor/160677 -> ciflow/inductor/160677 2025-08-14T21:20:59.0612253Z * [new tag] ciflow/inductor/160679 -> ciflow/inductor/160679 2025-08-14T21:20:59.0614254Z t [tag update] ciflow/periodic/160201 -> ciflow/periodic/160201 2025-08-14T21:20:59.0616879Z t [tag update] ciflow/trunk/154193 -> ciflow/trunk/154193 2025-08-14T21:20:59.0617936Z * [new tag] ciflow/trunk/155958 -> ciflow/trunk/155958 2025-08-14T21:20:59.0618768Z * [new tag] ciflow/trunk/156851 -> ciflow/trunk/156851 2025-08-14T21:20:59.0619796Z * [new tag] ciflow/trunk/157152 -> ciflow/trunk/157152 2025-08-14T21:20:59.0620785Z * [new tag] ciflow/trunk/158810 -> ciflow/trunk/158810 2025-08-14T21:20:59.0621692Z * [new tag] ciflow/trunk/158812 -> ciflow/trunk/158812 2025-08-14T21:20:59.0623475Z t [tag update] ciflow/trunk/158863 -> ciflow/trunk/158863 2025-08-14T21:20:59.0624478Z t [tag update] ciflow/trunk/158864 -> ciflow/trunk/158864 2025-08-14T21:20:59.0625705Z t [tag update] ciflow/trunk/158883 -> ciflow/trunk/158883 2025-08-14T21:20:59.0627071Z t [tag update] ciflow/trunk/158965 -> ciflow/trunk/158965 2025-08-14T21:20:59.0628468Z t [tag update] ciflow/trunk/159562 -> ciflow/trunk/159562 2025-08-14T21:20:59.0629582Z * [new tag] ciflow/trunk/159682 -> ciflow/trunk/159682 2025-08-14T21:20:59.0631081Z t [tag update] ciflow/trunk/159923 -> ciflow/trunk/159923 2025-08-14T21:20:59.0632427Z * [new tag] ciflow/trunk/160219 -> ciflow/trunk/160219 2025-08-14T21:20:59.0633783Z * [new tag] ciflow/trunk/160430 -> ciflow/trunk/160430 2025-08-14T21:20:59.0635138Z t [tag update] ciflow/trunk/160592 -> ciflow/trunk/160592 2025-08-14T21:20:59.0636010Z * [new tag] ciflow/trunk/160656 -> ciflow/trunk/160656 2025-08-14T21:20:59.0637444Z t [tag update] ciflow/vllm/160116 -> ciflow/vllm/160116 2025-08-14T21:20:59.0638832Z t [tag update] ciflow/vllm/160583 -> ciflow/vllm/160583 2025-08-14T21:20:59.0640397Z t [tag update] ciflow/win-arm64/159562 -> ciflow/win-arm64/159562 2025-08-14T21:20:59.0643718Z * [new tag] trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 -> trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 2025-08-14T21:20:59.0645227Z * [new tag] trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 -> trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 2025-08-14T21:20:59.0646719Z * [new tag] trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 -> trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 2025-08-14T21:20:59.0648209Z * [new tag] trunk/1fc683cf17c8c673044538d10266c00f92987be2 -> trunk/1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:20:59.0649665Z * [new tag] trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 -> trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 2025-08-14T21:20:59.0651159Z * [new tag] trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 -> trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 2025-08-14T21:20:59.0652660Z * [new tag] trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 -> trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 2025-08-14T21:20:59.0654146Z * [new tag] trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 -> trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 2025-08-14T21:20:59.0656035Z * [new tag] trunk/8d6d3246316e1767a57d5e855acd6208da753b75 -> trunk/8d6d3246316e1767a57d5e855acd6208da753b75 2025-08-14T21:20:59.0657931Z * [new tag] trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 -> trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 2025-08-14T21:20:59.0660307Z * [new tag] trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 -> trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 2025-08-14T21:20:59.0661976Z * [new tag] trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb -> trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb 2025-08-14T21:20:59.1694716Z [command]/usr/bin/git rev-parse --verify --quiet 1fc683cf17c8c673044538d10266c00f92987be2^{object} 2025-08-14T21:20:59.1806137Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:20:59.1820763Z ##[endgroup] 2025-08-14T21:20:59.1821572Z ##[group]Determining the checkout info 2025-08-14T21:20:59.1822420Z ##[endgroup] 2025-08-14T21:20:59.1830230Z [command]/usr/bin/git sparse-checkout disable 2025-08-14T21:20:59.2233625Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-08-14T21:20:59.2309854Z ##[group]Checking out the ref 2025-08-14T21:20:59.2317665Z [command]/usr/bin/git checkout --progress --force 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:20:59.4809903Z Previous HEAD position was 6c05ea6475b [DTensor] add op support: aten.squeeze_.dim (#159532) 2025-08-14T21:20:59.4828275Z HEAD is now at 1fc683cf17c [Inductor] Allow indexing a flexible layout for extract_input_node_reduction_ranges (#160645) 2025-08-14T21:20:59.4939251Z ##[endgroup] 2025-08-14T21:20:59.4940101Z ##[group]Setting up auth for fetching submodules 2025-08-14T21:20:59.4951009Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:20:59.5045387Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-08-14T21:20:59.5123425Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-08-14T21:20:59.5198692Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-08-14T21:20:59.5265934Z ##[endgroup] 2025-08-14T21:20:59.5266622Z ##[group]Fetching submodules 2025-08-14T21:20:59.5273729Z [command]/usr/bin/git submodule sync --recursive 2025-08-14T21:20:59.5996637Z Synchronizing submodule url for 'android/libs/fbjni' 2025-08-14T21:20:59.6106687Z Synchronizing submodule url for 'third_party/FP16' 2025-08-14T21:20:59.6215197Z Synchronizing submodule url for 'third_party/FXdiv' 2025-08-14T21:20:59.6320751Z Synchronizing submodule url for 'third_party/NNPACK' 2025-08-14T21:20:59.6427792Z Synchronizing submodule url for 'third_party/NVTX' 2025-08-14T21:20:59.6538714Z Synchronizing submodule url for 'third_party/VulkanMemoryAllocator' 2025-08-14T21:20:59.6646050Z Synchronizing submodule url for 'third_party/XNNPACK' 2025-08-14T21:20:59.6784657Z Synchronizing submodule url for 'third_party/aiter' 2025-08-14T21:20:59.6886833Z Synchronizing submodule url for 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:20:59.7012589Z Synchronizing submodule url for 'third_party/benchmark' 2025-08-14T21:20:59.7120052Z Synchronizing submodule url for 'third_party/composable_kernel' 2025-08-14T21:20:59.7251493Z Synchronizing submodule url for 'third_party/cpp-httplib' 2025-08-14T21:20:59.7357380Z Synchronizing submodule url for 'third_party/cpuinfo' 2025-08-14T21:20:59.7472267Z Synchronizing submodule url for 'third_party/cudnn_frontend' 2025-08-14T21:20:59.7582154Z Synchronizing submodule url for 'third_party/cutlass' 2025-08-14T21:20:59.7708955Z Synchronizing submodule url for 'third_party/fbgemm' 2025-08-14T21:20:59.7813338Z Synchronizing submodule url for 'third_party/fbgemm/external/asmjit' 2025-08-14T21:20:59.7908757Z Synchronizing submodule url for 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:20:59.8019642Z Synchronizing submodule url for 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:20:59.8120896Z Synchronizing submodule url for 'third_party/fbgemm/external/cutlass' 2025-08-14T21:20:59.8240349Z Synchronizing submodule url for 'third_party/fbgemm/external/googletest' 2025-08-14T21:20:59.8333993Z Synchronizing submodule url for 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:20:59.8429781Z Synchronizing submodule url for 'third_party/fbgemm/external/json' 2025-08-14T21:20:59.8540896Z Synchronizing submodule url for 'third_party/flash-attention' 2025-08-14T21:20:59.8643620Z Synchronizing submodule url for 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:20:59.8752339Z Synchronizing submodule url for 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:20:59.8876703Z Synchronizing submodule url for 'third_party/flatbuffers' 2025-08-14T21:20:59.8991338Z Synchronizing submodule url for 'third_party/fmt' 2025-08-14T21:20:59.9102837Z Synchronizing submodule url for 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:20:59.9210989Z Synchronizing submodule url for 'third_party/gloo' 2025-08-14T21:20:59.9319239Z Synchronizing submodule url for 'third_party/googletest' 2025-08-14T21:20:59.9429423Z Synchronizing submodule url for 'third_party/ideep' 2025-08-14T21:20:59.9528593Z Synchronizing submodule url for 'third_party/ideep/mkl-dnn' 2025-08-14T21:20:59.9656185Z Synchronizing submodule url for 'third_party/ittapi' 2025-08-14T21:20:59.9761929Z Synchronizing submodule url for 'third_party/kineto' 2025-08-14T21:20:59.9861793Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:20:59.9954302Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:00.0057312Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:00.0157208Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:00.0256716Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:00.0347421Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:00.0451733Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:00.0547891Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:00.0645514Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:00.0748421Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:00.0851556Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:00.0945567Z Synchronizing submodule url for 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:00.1049835Z Synchronizing submodule url for 'third_party/kleidiai' 2025-08-14T21:21:00.1155694Z Synchronizing submodule url for 'third_party/mimalloc' 2025-08-14T21:21:00.1262454Z Synchronizing submodule url for 'third_party/nlohmann' 2025-08-14T21:21:00.1370656Z Synchronizing submodule url for 'third_party/onnx' 2025-08-14T21:21:00.1505509Z Synchronizing submodule url for 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:00.1621097Z Synchronizing submodule url for 'third_party/opentelemetry-cpp' 2025-08-14T21:21:00.1721644Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:00.1823529Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:00.1920984Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:00.2016457Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:00.2115812Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:00.2216693Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:00.2312095Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:00.2403480Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:00.2502910Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:00.2604139Z Synchronizing submodule url for 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:00.2749277Z Synchronizing submodule url for 'third_party/pocketfft' 2025-08-14T21:21:00.2853364Z Synchronizing submodule url for 'third_party/protobuf' 2025-08-14T21:21:00.2957707Z Synchronizing submodule url for 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:00.3055388Z Synchronizing submodule url for 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:00.3162533Z Synchronizing submodule url for 'third_party/psimd' 2025-08-14T21:21:00.3270791Z Synchronizing submodule url for 'third_party/pthreadpool' 2025-08-14T21:21:00.3380163Z Synchronizing submodule url for 'third_party/pybind11' 2025-08-14T21:21:00.3484830Z Synchronizing submodule url for 'third_party/python-peachpy' 2025-08-14T21:21:00.3592153Z Synchronizing submodule url for 'third_party/sleef' 2025-08-14T21:21:00.3699554Z Synchronizing submodule url for 'third_party/tensorpipe' 2025-08-14T21:21:00.3796781Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:00.3897387Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:00.3992539Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:00.4088042Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:00.4179245Z Synchronizing submodule url for 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:00.4326351Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-08-14T21:21:00.5610913Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-08-14T21:21:00.6237424Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-08-14T21:21:00.6847843Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-08-14T21:21:00.7474154Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-08-14T21:21:00.8147048Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-08-14T21:21:00.8799321Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-08-14T21:21:00.9816474Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-08-14T21:21:01.0691197Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-08-14T21:21:01.1632642Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-08-14T21:21:01.2344530Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-08-14T21:21:01.3466032Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-08-14T21:21:01.4172942Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c' 2025-08-14T21:21:01.4836492Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-08-14T21:21:01.5547992Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-08-14T21:21:01.6410499Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491' 2025-08-14T21:21:01.7311715Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067' 2025-08-14T21:21:01.7886948Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-08-14T21:21:01.8618701Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9' 2025-08-14T21:21:01.9266774Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-08-14T21:21:02.0031094Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad' 2025-08-14T21:21:02.0666979Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:21:02.1236305Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c' 2025-08-14T21:21:02.1919563Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-08-14T21:21:02.2658299Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-08-14T21:21:02.3575915Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-08-14T21:21:02.4376818Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-08-14T21:21:02.5171370Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-08-14T21:21:02.5844950Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-08-14T21:21:02.6478225Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-08-14T21:21:02.7136404Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2' 2025-08-14T21:21:02.7783403Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:21:02.8437679Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-08-14T21:21:02.9429512Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-08-14T21:21:03.0110700Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-08-14T21:21:03.0810826Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658' 2025-08-14T21:21:03.1459201Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-08-14T21:21:03.2086974Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-08-14T21:21:03.2698278Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-08-14T21:21:03.3328610Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-08-14T21:21:03.3929667Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-08-14T21:21:03.4496900Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-08-14T21:21:03.5104333Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-08-14T21:21:03.5719768Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-08-14T21:21:03.6428702Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-08-14T21:21:03.7012647Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-08-14T21:21:03.7623136Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-08-14T21:21:03.8222892Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-08-14T21:21:03.8900351Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-08-14T21:21:03.9596568Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-08-14T21:21:04.0346781Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-08-14T21:21:04.1376808Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-08-14T21:21:04.2108841Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:21:04.2896545Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-08-14T21:21:04.3477133Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-08-14T21:21:04.4071584Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-08-14T21:21:04.4656775Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-08-14T21:21:04.5346000Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-08-14T21:21:04.5946149Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-08-14T21:21:04.6511766Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-08-14T21:21:04.7118215Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-08-14T21:21:04.7755004Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-08-14T21:21:04.8373035Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-08-14T21:21:04.9330517Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-08-14T21:21:05.0050855Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-08-14T21:21:05.1119181Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-08-14T21:21:05.1703593Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-08-14T21:21:05.2317723Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-08-14T21:21:05.2942230Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-08-14T21:21:05.3561343Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-08-14T21:21:05.4262310Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:21:05.4893367Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-08-14T21:21:05.5539199Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-08-14T21:21:05.6225271Z Submodule path 'third_party/tensorpipe': checked out 'dacda0567d9f23d4bc503e1c4f84aa65f33ac38a' 2025-08-14T21:21:05.6798402Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-08-14T21:21:05.7353290Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-08-14T21:21:05.8266100Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-08-14T21:21:05.8903980Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-08-14T21:21:05.9449593Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-08-14T21:21:05.9686901Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-08-14T21:21:06.0371027Z Entering 'android/libs/fbjni' 2025-08-14T21:21:06.0489594Z Entering 'third_party/FP16' 2025-08-14T21:21:06.0600965Z Entering 'third_party/FXdiv' 2025-08-14T21:21:06.0713700Z Entering 'third_party/NNPACK' 2025-08-14T21:21:06.0823722Z Entering 'third_party/NVTX' 2025-08-14T21:21:06.0937876Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:21:06.1050742Z Entering 'third_party/XNNPACK' 2025-08-14T21:21:06.1193061Z Entering 'third_party/aiter' 2025-08-14T21:21:06.1302434Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:21:06.1428602Z Entering 'third_party/benchmark' 2025-08-14T21:21:06.1539333Z Entering 'third_party/composable_kernel' 2025-08-14T21:21:06.1665377Z Entering 'third_party/cpp-httplib' 2025-08-14T21:21:06.1774137Z Entering 'third_party/cpuinfo' 2025-08-14T21:21:06.1883460Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:21:06.1991269Z Entering 'third_party/cutlass' 2025-08-14T21:21:06.2118148Z Entering 'third_party/fbgemm' 2025-08-14T21:21:06.2230423Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:21:06.2333540Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:21:06.2450957Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:21:06.2552237Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:21:06.2671962Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:21:06.2774473Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:21:06.2874104Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:21:06.2986677Z Entering 'third_party/flash-attention' 2025-08-14T21:21:06.3095155Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:21:06.3211967Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:21:06.3339167Z Entering 'third_party/flatbuffers' 2025-08-14T21:21:06.3452478Z Entering 'third_party/fmt' 2025-08-14T21:21:06.3560436Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:21:06.3668166Z Entering 'third_party/gloo' 2025-08-14T21:21:06.3778717Z Entering 'third_party/googletest' 2025-08-14T21:21:06.3886793Z Entering 'third_party/ideep' 2025-08-14T21:21:06.3992081Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:21:06.4112873Z Entering 'third_party/ittapi' 2025-08-14T21:21:06.4220239Z Entering 'third_party/kineto' 2025-08-14T21:21:06.4323262Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:21:06.4420085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:06.4522911Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:06.4626789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:06.4727992Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:06.4828456Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:06.4937040Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:06.5036167Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:06.5137639Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:06.5237457Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:06.5344034Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:06.5445344Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:06.5553850Z Entering 'third_party/kleidiai' 2025-08-14T21:21:06.5665960Z Entering 'third_party/mimalloc' 2025-08-14T21:21:06.5774311Z Entering 'third_party/nlohmann' 2025-08-14T21:21:06.5884220Z Entering 'third_party/onnx' 2025-08-14T21:21:06.6026652Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:06.6142291Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:21:06.6250948Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:06.6351108Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:06.6451502Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:06.6551269Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:06.6654016Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:06.6751863Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:06.6850511Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:06.6946211Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:06.7050210Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:06.7157557Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:06.7303507Z Entering 'third_party/pocketfft' 2025-08-14T21:21:06.7417904Z Entering 'third_party/protobuf' 2025-08-14T21:21:06.7533506Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:06.7635798Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:06.7747171Z Entering 'third_party/psimd' 2025-08-14T21:21:06.7859497Z Entering 'third_party/pthreadpool' 2025-08-14T21:21:06.7968944Z Entering 'third_party/pybind11' 2025-08-14T21:21:06.8081347Z Entering 'third_party/python-peachpy' 2025-08-14T21:21:06.8191738Z Entering 'third_party/sleef' 2025-08-14T21:21:06.8302779Z Entering 'third_party/tensorpipe' 2025-08-14T21:21:06.8409927Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:06.8513298Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:06.8612736Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:06.8713973Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:06.8809376Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:06.8954645Z ##[endgroup] 2025-08-14T21:21:06.8955738Z ##[group]Persisting credentials for submodules 2025-08-14T21:21:06.8966651Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-08-14T21:21:06.9644999Z Entering 'android/libs/fbjni' 2025-08-14T21:21:06.9713712Z url.https://github.com/.insteadof 2025-08-14T21:21:06.9714861Z url.https://github.com/.insteadof 2025-08-14T21:21:06.9809321Z Entering 'third_party/FP16' 2025-08-14T21:21:06.9875694Z url.https://github.com/.insteadof 2025-08-14T21:21:06.9971269Z url.https://github.com/.insteadof 2025-08-14T21:21:06.9972139Z Entering 'third_party/FXdiv' 2025-08-14T21:21:07.0037531Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0038362Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0135895Z Entering 'third_party/NNPACK' 2025-08-14T21:21:07.0201427Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0202588Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0301087Z Entering 'third_party/NVTX' 2025-08-14T21:21:07.0367111Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0368080Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0463792Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:21:07.0531273Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0532311Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0625155Z Entering 'third_party/XNNPACK' 2025-08-14T21:21:07.0691920Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0692958Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0820020Z Entering 'third_party/aiter' 2025-08-14T21:21:07.0886459Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0887391Z url.https://github.com/.insteadof 2025-08-14T21:21:07.0978800Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:21:07.1040804Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1078976Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1156763Z Entering 'third_party/benchmark' 2025-08-14T21:21:07.1223199Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1223980Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1319436Z Entering 'third_party/composable_kernel' 2025-08-14T21:21:07.1383628Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1384305Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1497588Z Entering 'third_party/cpp-httplib' 2025-08-14T21:21:07.1562574Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1564004Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1654651Z Entering 'third_party/cpuinfo' 2025-08-14T21:21:07.1718615Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1719280Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1812049Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:21:07.1874849Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1875514Z url.https://github.com/.insteadof 2025-08-14T21:21:07.1967701Z Entering 'third_party/cutlass' 2025-08-14T21:21:07.2031527Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2032175Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2144677Z Entering 'third_party/fbgemm' 2025-08-14T21:21:07.2210336Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2211006Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2307778Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:21:07.2369817Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2370496Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2459600Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:21:07.2522604Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2523343Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2631506Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:21:07.2690089Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2690783Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2779966Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:21:07.2838995Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2839708Z url.https://github.com/.insteadof 2025-08-14T21:21:07.2949520Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:21:07.3007401Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3008034Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3095559Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:21:07.3156970Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3157655Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3245046Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:21:07.3304278Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3305005Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3403441Z Entering 'third_party/flash-attention' 2025-08-14T21:21:07.3467910Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3468589Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3556395Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:21:07.3615367Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3616024Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3718272Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:21:07.3776252Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3776960Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3890750Z Entering 'third_party/flatbuffers' 2025-08-14T21:21:07.3954462Z url.https://github.com/.insteadof 2025-08-14T21:21:07.3955183Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4054020Z Entering 'third_party/fmt' 2025-08-14T21:21:07.4116730Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4117401Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4207869Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:21:07.4271330Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4272067Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4367276Z Entering 'third_party/gloo' 2025-08-14T21:21:07.4429993Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4430745Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4524747Z Entering 'third_party/googletest' 2025-08-14T21:21:07.4588072Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4588750Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4681853Z Entering 'third_party/ideep' 2025-08-14T21:21:07.4744829Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4745531Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4831287Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:21:07.4892548Z url.https://github.com/.insteadof 2025-08-14T21:21:07.4893973Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5003844Z Entering 'third_party/ittapi' 2025-08-14T21:21:07.5068684Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5069362Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5171371Z Entering 'third_party/kineto' 2025-08-14T21:21:07.5235885Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5236568Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5329310Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:21:07.5390728Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5391451Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5478590Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:07.5540297Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5541882Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5634943Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:07.5695950Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5696696Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5788735Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:07.5849545Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5850335Z url.https://github.com/.insteadof 2025-08-14T21:21:07.5942505Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:07.6003966Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6004693Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6093015Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:07.6154120Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6154781Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6251367Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:07.6311359Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6312070Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6399835Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:07.6461864Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6462608Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6555845Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:07.6615696Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6616357Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6709970Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:07.6767512Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6768213Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6865102Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:07.6924328Z url.https://github.com/.insteadof 2025-08-14T21:21:07.6925026Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7016098Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:07.7075935Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7076655Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7175263Z Entering 'third_party/kleidiai' 2025-08-14T21:21:07.7238892Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7239614Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7336478Z Entering 'third_party/mimalloc' 2025-08-14T21:21:07.7401845Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7402582Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7495447Z Entering 'third_party/nlohmann' 2025-08-14T21:21:07.7561028Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7563255Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7659930Z Entering 'third_party/onnx' 2025-08-14T21:21:07.7726789Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7727462Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7857075Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:07.7919796Z url.https://github.com/.insteadof 2025-08-14T21:21:07.7920640Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8029997Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:21:07.8093991Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8094638Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8188240Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:07.8249556Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8250222Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8342177Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:07.8402212Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8402883Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8491470Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:07.8551160Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8552441Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8639922Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:07.8700677Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8701362Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8791056Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:07.8850817Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8851489Z url.https://github.com/.insteadof 2025-08-14T21:21:07.8940878Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:07.9000178Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9000958Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9088843Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:07.9148355Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9149036Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9235306Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:07.9293039Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9293695Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9386962Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:07.9446975Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9447636Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9545113Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:07.9606462Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9607161Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9746663Z Entering 'third_party/pocketfft' 2025-08-14T21:21:07.9810552Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9811231Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9902966Z Entering 'third_party/protobuf' 2025-08-14T21:21:07.9966482Z url.https://github.com/.insteadof 2025-08-14T21:21:07.9967148Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0064903Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:08.0125123Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0125791Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0214983Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:08.0275060Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0275803Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0374886Z Entering 'third_party/psimd' 2025-08-14T21:21:08.0440023Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0441162Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0535485Z Entering 'third_party/pthreadpool' 2025-08-14T21:21:08.0600777Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0601494Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0695172Z Entering 'third_party/pybind11' 2025-08-14T21:21:08.0758989Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0759747Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0853366Z Entering 'third_party/python-peachpy' 2025-08-14T21:21:08.0918453Z url.https://github.com/.insteadof 2025-08-14T21:21:08.0919185Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1014355Z Entering 'third_party/sleef' 2025-08-14T21:21:08.1078801Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1080440Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1174397Z Entering 'third_party/tensorpipe' 2025-08-14T21:21:08.1237894Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1238574Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1327865Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:08.1387696Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1388416Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1478384Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:08.1536110Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1536842Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1623899Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:08.1684029Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1685486Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1775517Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:08.1833401Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1834105Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1919496Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:08.1979120Z url.https://github.com/.insteadof 2025-08-14T21:21:08.1979874Z url.https://github.com/.insteadof 2025-08-14T21:21:08.2122334Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-08-14T21:21:08.2806405Z Entering 'android/libs/fbjni' 2025-08-14T21:21:08.2911534Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-08-14T21:21:08.2969636Z Entering 'third_party/FP16' 2025-08-14T21:21:08.3072338Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-08-14T21:21:08.3135635Z Entering 'third_party/FXdiv' 2025-08-14T21:21:08.3237952Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-08-14T21:21:08.3296151Z Entering 'third_party/NNPACK' 2025-08-14T21:21:08.3398733Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-08-14T21:21:08.3458167Z Entering 'third_party/NVTX' 2025-08-14T21:21:08.3559162Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-08-14T21:21:08.3620401Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:21:08.3722666Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-08-14T21:21:08.3783331Z Entering 'third_party/XNNPACK' 2025-08-14T21:21:08.3885364Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-08-14T21:21:08.3977565Z Entering 'third_party/aiter' 2025-08-14T21:21:08.4078818Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-08-14T21:21:08.4136657Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:21:08.4232366Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-08-14T21:21:08.4310559Z Entering 'third_party/benchmark' 2025-08-14T21:21:08.4413182Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:21:08.4471966Z Entering 'third_party/composable_kernel' 2025-08-14T21:21:08.4573811Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-08-14T21:21:08.4652919Z Entering 'third_party/cpp-httplib' 2025-08-14T21:21:08.4757145Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-08-14T21:21:08.4813884Z Entering 'third_party/cpuinfo' 2025-08-14T21:21:08.4917032Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-08-14T21:21:08.4976796Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:21:08.5078654Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-08-14T21:21:08.5139150Z Entering 'third_party/cutlass' 2025-08-14T21:21:08.5243830Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-08-14T21:21:08.5324006Z Entering 'third_party/fbgemm' 2025-08-14T21:21:08.5429228Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-08-14T21:21:08.5489930Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:21:08.5587004Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-08-14T21:21:08.5642296Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:21:08.5740289Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-08-14T21:21:08.5810431Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:21:08.5907864Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-08-14T21:21:08.5962936Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:21:08.6078134Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-08-14T21:21:08.6152157Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:21:08.6248235Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-08-14T21:21:08.6303052Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:21:08.6403523Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-08-14T21:21:08.6456572Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:21:08.6555405Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-08-14T21:21:08.6621184Z Entering 'third_party/flash-attention' 2025-08-14T21:21:08.6721879Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-08-14T21:21:08.6779019Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:21:08.6877501Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-08-14T21:21:08.6944563Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:21:08.7040353Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-08-14T21:21:08.7117415Z Entering 'third_party/flatbuffers' 2025-08-14T21:21:08.7216702Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-08-14T21:21:08.7280183Z Entering 'third_party/fmt' 2025-08-14T21:21:08.7377638Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:21:08.7434821Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:21:08.7533405Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-08-14T21:21:08.7590614Z Entering 'third_party/gloo' 2025-08-14T21:21:08.7689587Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-08-14T21:21:08.7746491Z Entering 'third_party/googletest' 2025-08-14T21:21:08.7849828Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:21:08.7908903Z Entering 'third_party/ideep' 2025-08-14T21:21:08.8009061Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-08-14T21:21:08.8062046Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:21:08.8158691Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-08-14T21:21:08.8236991Z Entering 'third_party/ittapi' 2025-08-14T21:21:08.8338318Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-08-14T21:21:08.8395429Z Entering 'third_party/kineto' 2025-08-14T21:21:08.8497489Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-08-14T21:21:08.8551459Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:21:08.8656346Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-08-14T21:21:08.8706173Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:08.8806902Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-08-14T21:21:08.8863981Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:08.8968796Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-08-14T21:21:08.9023030Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:08.9122641Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:21:08.9177063Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:08.9274229Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-08-14T21:21:08.9323776Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:08.9425174Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-08-14T21:21:08.9486548Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:08.9584551Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-08-14T21:21:08.9638179Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:08.9734697Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:21:08.9787502Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:08.9883307Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-08-14T21:21:08.9938261Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:09.0035519Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-08-14T21:21:09.0094053Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:09.0188688Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-08-14T21:21:09.0240787Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:09.0336509Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-08-14T21:21:09.0396708Z Entering 'third_party/kleidiai' 2025-08-14T21:21:09.0497399Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-08-14T21:21:09.0555433Z Entering 'third_party/mimalloc' 2025-08-14T21:21:09.0655710Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-08-14T21:21:09.0712535Z Entering 'third_party/nlohmann' 2025-08-14T21:21:09.0812048Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-08-14T21:21:09.0872814Z Entering 'third_party/onnx' 2025-08-14T21:21:09.0973087Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-08-14T21:21:09.1062619Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:09.1161865Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:21:09.1227934Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:21:09.1329097Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-08-14T21:21:09.1385554Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:09.1480781Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:21:09.1533533Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:09.1628970Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:21:09.1681150Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:09.1777785Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-08-14T21:21:09.1828634Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:09.1927147Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-08-14T21:21:09.1981672Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:09.2077065Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-08-14T21:21:09.2127767Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:09.2222645Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-08-14T21:21:09.2277003Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:09.2372727Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-08-14T21:21:09.2420835Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:09.2517744Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-08-14T21:21:09.2575517Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:09.2670625Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-08-14T21:21:09.2731430Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:09.2829962Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-08-14T21:21:09.2930612Z Entering 'third_party/pocketfft' 2025-08-14T21:21:09.3030801Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-08-14T21:21:09.3086516Z Entering 'third_party/protobuf' 2025-08-14T21:21:09.3187627Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-08-14T21:21:09.3247755Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:09.3342738Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:21:09.3394804Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:09.3489136Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:21:09.3550819Z Entering 'third_party/psimd' 2025-08-14T21:21:09.3651612Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-08-14T21:21:09.3708645Z Entering 'third_party/pthreadpool' 2025-08-14T21:21:09.3808367Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-08-14T21:21:09.3864618Z Entering 'third_party/pybind11' 2025-08-14T21:21:09.3961788Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:21:09.4019947Z Entering 'third_party/python-peachpy' 2025-08-14T21:21:09.4123537Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-08-14T21:21:09.4181853Z Entering 'third_party/sleef' 2025-08-14T21:21:09.4281427Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-08-14T21:21:09.4340134Z Entering 'third_party/tensorpipe' 2025-08-14T21:21:09.4440533Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-08-14T21:21:09.4495211Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:09.4590854Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:21:09.4643787Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:09.4739891Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-08-14T21:21:09.4792127Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:09.4888971Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-08-14T21:21:09.4941277Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:09.5036976Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:21:09.5084386Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:09.5185632Z file:/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-08-14T21:21:09.5769438Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-08-14T21:21:09.6450659Z Entering 'android/libs/fbjni' 2025-08-14T21:21:09.6561020Z Entering 'third_party/FP16' 2025-08-14T21:21:09.6671114Z Entering 'third_party/FXdiv' 2025-08-14T21:21:09.6780701Z Entering 'third_party/NNPACK' 2025-08-14T21:21:09.6890159Z Entering 'third_party/NVTX' 2025-08-14T21:21:09.7000719Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:21:09.7109559Z Entering 'third_party/XNNPACK' 2025-08-14T21:21:09.7250596Z Entering 'third_party/aiter' 2025-08-14T21:21:09.7356746Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:21:09.7482859Z Entering 'third_party/benchmark' 2025-08-14T21:21:09.7590561Z Entering 'third_party/composable_kernel' 2025-08-14T21:21:09.7719713Z Entering 'third_party/cpp-httplib' 2025-08-14T21:21:09.7826881Z Entering 'third_party/cpuinfo' 2025-08-14T21:21:09.7936209Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:21:09.8060207Z Entering 'third_party/cutlass' 2025-08-14T21:21:09.8187896Z Entering 'third_party/fbgemm' 2025-08-14T21:21:09.8300600Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:21:09.8405385Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:21:09.8526027Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:21:09.8629550Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:21:09.8750272Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:21:09.8852971Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:21:09.8953874Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:21:09.9069906Z Entering 'third_party/flash-attention' 2025-08-14T21:21:09.9179193Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:21:09.9294618Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:21:09.9419518Z Entering 'third_party/flatbuffers' 2025-08-14T21:21:09.9534986Z Entering 'third_party/fmt' 2025-08-14T21:21:09.9649799Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:21:09.9755152Z Entering 'third_party/gloo' 2025-08-14T21:21:09.9867098Z Entering 'third_party/googletest' 2025-08-14T21:21:09.9976735Z Entering 'third_party/ideep' 2025-08-14T21:21:10.0092560Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:21:10.0210355Z Entering 'third_party/ittapi' 2025-08-14T21:21:10.0320173Z Entering 'third_party/kineto' 2025-08-14T21:21:10.0428454Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:21:10.0526010Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:10.0631777Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:10.0736844Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:10.0838558Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:10.0938549Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:10.1048127Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:10.1150560Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:10.1251683Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:10.1355205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:10.1461750Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:10.1562668Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:10.1670464Z Entering 'third_party/kleidiai' 2025-08-14T21:21:10.1778677Z Entering 'third_party/mimalloc' 2025-08-14T21:21:10.1884444Z Entering 'third_party/nlohmann' 2025-08-14T21:21:10.1993481Z Entering 'third_party/onnx' 2025-08-14T21:21:10.2132225Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:10.2245161Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:21:10.2352339Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:10.2450437Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:10.2548082Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:10.2644877Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:10.2746993Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:10.2844517Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:10.2942363Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:10.3036456Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:10.3138518Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:10.3244053Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:10.3388058Z Entering 'third_party/pocketfft' 2025-08-14T21:21:10.3494815Z Entering 'third_party/protobuf' 2025-08-14T21:21:10.3605634Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:10.3703624Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:10.3809430Z Entering 'third_party/psimd' 2025-08-14T21:21:10.3916574Z Entering 'third_party/pthreadpool' 2025-08-14T21:21:10.4022688Z Entering 'third_party/pybind11' 2025-08-14T21:21:10.4130407Z Entering 'third_party/python-peachpy' 2025-08-14T21:21:10.4237218Z Entering 'third_party/sleef' 2025-08-14T21:21:10.4343785Z Entering 'third_party/tensorpipe' 2025-08-14T21:21:10.4446789Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:10.4543647Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:10.4640831Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:10.4738407Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:10.4831245Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:10.4988527Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-08-14T21:21:10.5680732Z Entering 'android/libs/fbjni' 2025-08-14T21:21:10.5794128Z Entering 'third_party/FP16' 2025-08-14T21:21:10.5904851Z Entering 'third_party/FXdiv' 2025-08-14T21:21:10.6017885Z Entering 'third_party/NNPACK' 2025-08-14T21:21:10.6127640Z Entering 'third_party/NVTX' 2025-08-14T21:21:10.6241171Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:21:10.6354954Z Entering 'third_party/XNNPACK' 2025-08-14T21:21:10.6498014Z Entering 'third_party/aiter' 2025-08-14T21:21:10.6604223Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:21:10.6732796Z Entering 'third_party/benchmark' 2025-08-14T21:21:10.6843598Z Entering 'third_party/composable_kernel' 2025-08-14T21:21:10.6971874Z Entering 'third_party/cpp-httplib' 2025-08-14T21:21:10.7080880Z Entering 'third_party/cpuinfo' 2025-08-14T21:21:10.7193240Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:21:10.7305858Z Entering 'third_party/cutlass' 2025-08-14T21:21:10.7434707Z Entering 'third_party/fbgemm' 2025-08-14T21:21:10.7544174Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:21:10.7648814Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:21:10.7768259Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:21:10.7873529Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:21:10.7995375Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:21:10.8097869Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:21:10.8199102Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:21:10.8312503Z Entering 'third_party/flash-attention' 2025-08-14T21:21:10.8421730Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:21:10.8542314Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:21:10.8671280Z Entering 'third_party/flatbuffers' 2025-08-14T21:21:10.8788615Z Entering 'third_party/fmt' 2025-08-14T21:21:10.8899478Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:21:10.9010555Z Entering 'third_party/gloo' 2025-08-14T21:21:10.9119658Z Entering 'third_party/googletest' 2025-08-14T21:21:10.9227920Z Entering 'third_party/ideep' 2025-08-14T21:21:10.9330622Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:21:10.9452686Z Entering 'third_party/ittapi' 2025-08-14T21:21:10.9558726Z Entering 'third_party/kineto' 2025-08-14T21:21:10.9668153Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:21:10.9768074Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:21:10.9873497Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:21:10.9976308Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:21:11.0079195Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:21:11.0178205Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:21:11.0290050Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:21:11.0396945Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:21:11.0499902Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:21:11.0604080Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:21:11.0713358Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:21:11.0818005Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:21:11.0929385Z Entering 'third_party/kleidiai' 2025-08-14T21:21:11.1045882Z Entering 'third_party/mimalloc' 2025-08-14T21:21:11.1158830Z Entering 'third_party/nlohmann' 2025-08-14T21:21:11.1275376Z Entering 'third_party/onnx' 2025-08-14T21:21:11.1423731Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:21:11.1544196Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:21:11.1654235Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:21:11.1756999Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:21:11.1861934Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:21:11.1964158Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:21:11.2071166Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:21:11.2172687Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:21:11.2273610Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:21:11.2370473Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:21:11.2474459Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:21:11.2584058Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:21:11.2732056Z Entering 'third_party/pocketfft' 2025-08-14T21:21:11.2840322Z Entering 'third_party/protobuf' 2025-08-14T21:21:11.2952427Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:21:11.3056069Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:21:11.3168207Z Entering 'third_party/psimd' 2025-08-14T21:21:11.3275908Z Entering 'third_party/pthreadpool' 2025-08-14T21:21:11.3389071Z Entering 'third_party/pybind11' 2025-08-14T21:21:11.3501419Z Entering 'third_party/python-peachpy' 2025-08-14T21:21:11.3613020Z Entering 'third_party/sleef' 2025-08-14T21:21:11.3720585Z Entering 'third_party/tensorpipe' 2025-08-14T21:21:11.3827815Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:21:11.3928338Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:21:11.4029431Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:21:11.4129557Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:21:11.4227423Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:21:11.4373948Z ##[endgroup] 2025-08-14T21:21:11.4520522Z [command]/usr/bin/git log -1 --format=%H 2025-08-14T21:21:11.4630982Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:21:11.5047549Z Prepare all required actions 2025-08-14T21:21:11.5048487Z Getting action download info 2025-08-14T21:21:11.6779752Z ##[group]Run ./.github/actions/setup-rocm 2025-08-14T21:21:11.6780223Z env: 2025-08-14T21:21:11.6780536Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.6780917Z ##[endgroup] 2025-08-14T21:21:11.6819288Z ##[group]Run dpkg -l | grep -E " rocm" 2025-08-14T21:21:11.6819797Z dpkg -l | grep -E " rocm" 2025-08-14T21:21:11.6864839Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:11.6865412Z env: 2025-08-14T21:21:11.6865741Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.6866116Z ##[endgroup] 2025-08-14T21:21:11.7301456Z ii rocm 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) software stack meta package 2025-08-14T21:21:11.7302658Z ii rocm-cmake 0.14.0.60303-74~22.04 amd64 rocm-cmake built using CMake 2025-08-14T21:21:11.7303825Z ii rocm-core 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7304982Z ii rocm-dbgapi 0.77.0.60303-74~22.04 amd64 Library to provide AMD GPU debugger API 2025-08-14T21:21:11.7306374Z ii rocm-debug-agent 2.0.3.60303-74~22.04 amd64 Radeon Open Compute Debug Agent (ROCdebug-agent) 2025-08-14T21:21:11.7307883Z ii rocm-developer-tools 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7309339Z ii rocm-device-libs 1.0.0.60303-74~22.04 amd64 Radeon Open Compute - device libraries 2025-08-14T21:21:11.7310505Z ii rocm-gdb 15.2.60303-74~22.04 amd64 ROCgdb 2025-08-14T21:21:11.7311200Z ii rocm-hip-libraries 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7311927Z ii rocm-hip-runtime 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7312613Z ii rocm-hip-runtime-dev 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7313222Z ii rocm-hip-sdk 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7314001Z ii rocm-language-runtime 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7314832Z ii rocm-llvm 18.0.0.25012.60303-74~22.04 amd64 ROCm core compiler 2025-08-14T21:21:11.7315559Z ii rocm-ml-libraries 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7316271Z ii rocm-ml-sdk 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7316904Z ii rocm-opencl 2.0.0.60303-74~22.04 amd64 clr built using CMake 2025-08-14T21:21:11.7317498Z ii rocm-opencl-dev 2.0.0.60303-74~22.04 amd64 clr built using CMake 2025-08-14T21:21:11.7318196Z ii rocm-opencl-runtime 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7318947Z ii rocm-opencl-sdk 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7320103Z ii rocm-openmp-sdk 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) OpenMP Software development Kit. 2025-08-14T21:21:11.7321395Z ii rocm-smi-lib 7.4.0.60303-74~22.04 amd64 AMD System Management libraries 2025-08-14T21:21:11.7322060Z ii rocm-utils 6.3.3.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime software stack 2025-08-14T21:21:11.7322746Z ii rocminfo 1.0.0.60303-74~22.04 amd64 Radeon Open Compute (ROCm) Runtime rocminfo tool 2025-08-14T21:21:11.7348818Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:21:11.7349799Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:21:11.7350526Z # shellcheck disable=SC2046 2025-08-14T21:21:11.7351096Z docker stop $(docker ps -q) || true 2025-08-14T21:21:11.7351707Z # Prune all stopped containers. 2025-08-14T21:21:11.7352272Z docker container prune -f 2025-08-14T21:21:11.7400717Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:11.7401441Z env: 2025-08-14T21:21:11.7401849Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.7402333Z ##[endgroup] 2025-08-14T21:21:11.8140277Z "docker stop" requires at least 1 argument. 2025-08-14T21:21:11.8140957Z See 'docker stop --help'. 2025-08-14T21:21:11.8141267Z 2025-08-14T21:21:11.8141543Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-08-14T21:21:11.8142014Z 2025-08-14T21:21:11.8142199Z Stop one or more running containers 2025-08-14T21:21:11.8462510Z Total reclaimed space: 0B 2025-08-14T21:21:11.8552469Z ##[group]Run cat /etc/os-release || true 2025-08-14T21:21:11.8553115Z cat /etc/os-release || true 2025-08-14T21:21:11.8553735Z cat /etc/apt/sources.list.d/rocm.list || true 2025-08-14T21:21:11.8554384Z cat /opt/rocm/.info/version || true 2025-08-14T21:21:11.8554946Z whoami 2025-08-14T21:21:11.8608132Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:11.8608816Z env: 2025-08-14T21:21:11.8609184Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.8609634Z ##[endgroup] 2025-08-14T21:21:11.8717284Z PRETTY_NAME="Ubuntu 22.04.4 LTS" 2025-08-14T21:21:11.8718108Z NAME="Ubuntu" 2025-08-14T21:21:11.8718622Z VERSION_ID="22.04" 2025-08-14T21:21:11.8719332Z VERSION="22.04.4 LTS (Jammy Jellyfish)" 2025-08-14T21:21:11.8720174Z VERSION_CODENAME=jammy 2025-08-14T21:21:11.8720984Z ID=ubuntu 2025-08-14T21:21:11.8721369Z ID_LIKE=debian 2025-08-14T21:21:11.8721928Z HOME_URL="https://www.ubuntu.com/" 2025-08-14T21:21:11.8722607Z SUPPORT_URL="https://help.ubuntu.com/" 2025-08-14T21:21:11.8723461Z BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2025-08-14T21:21:11.8724731Z PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2025-08-14T21:21:11.8725760Z UBUNTU_CODENAME=jammy 2025-08-14T21:21:11.8745583Z deb [arch=amd64 signed-by=/etc/apt/keyrings/rocm.gpg] https://repo.radeon.com/rocm/apt/6.3.3 jammy main 2025-08-14T21:21:11.8773479Z 6.3.3-74 2025-08-14T21:21:11.8828439Z pytorchci 2025-08-14T21:21:11.8877678Z ##[group]Run dpkg -l | grep -E " amdgpu" 2025-08-14T21:21:11.8878328Z dpkg -l | grep -E " amdgpu" 2025-08-14T21:21:11.8930355Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:11.8931018Z env: 2025-08-14T21:21:11.8931393Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.8931823Z ##[endgroup] 2025-08-14T21:21:11.9372235Z ii amdgpu-core 1:6.3.60303-2119913.22.04 all Core meta package for unified amdgpu driver. 2025-08-14T21:21:11.9373546Z ii amdgpu-dkms 1:6.10.5.60303-2119913.22.04 all amdgpu driver in DKMS format. 2025-08-14T21:21:11.9374858Z ii amdgpu-dkms-firmware 1:6.10.5.60303-2119913.22.04 all firmware blobs used by amdgpu driver in DKMS format 2025-08-14T21:21:11.9377141Z ii amdgpu-install 6.3.60303-2119913.22.04 all AMDGPU driver repository and installer 2025-08-14T21:21:11.9429235Z ##[group]Run rocm-smi 2025-08-14T21:21:11.9429722Z rocm-smi 2025-08-14T21:21:11.9483382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:11.9484095Z env: 2025-08-14T21:21:11.9484527Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:11.9485003Z ##[endgroup] 2025-08-14T21:21:12.1454252Z 2025-08-14T21:21:12.1454346Z 2025-08-14T21:21:12.1454986Z ========================================= ROCm System Management Interface ========================================= 2025-08-14T21:21:12.1456006Z =================================================== Concise Info =================================================== 2025-08-14T21:21:12.1457104Z Device Node IDs Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2025-08-14T21:21:12.1458787Z  (DID, GUID) (Edge) (Avg) (Mem, Compute, ID)  2025-08-14T21:21:12.1459638Z ==================================================================================================================== 2025-08-14T21:21:12.1460961Z 0 4 0x740c, 57586 36.0°C 92.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2025-08-14T21:21:12.1462147Z 1 5 0x740c, 45873 40.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2025-08-14T21:21:12.1463285Z 2 2 0x740c, 14571 35.0°C 92.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2025-08-14T21:21:12.1464394Z 3 3 0x740c, 27432 27.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2025-08-14T21:21:12.1465510Z 4 8 0x740c, 30939 33.0°C 86.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2025-08-14T21:21:12.1466646Z 5 9 0x740c, 8466 37.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2025-08-14T21:21:12.1467755Z 6 6 0x740c, 41154 35.0°C 92.0W N/A, N/A, 0 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2025-08-14T21:21:12.1468874Z 7 7 0x740c, 63755 30.0°C N/A N/A, N/A, 0 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2025-08-14T21:21:12.1469677Z ==================================================================================================================== 2025-08-14T21:21:12.1470392Z =============================================== End of ROCm SMI Log ================================================ 2025-08-14T21:21:12.1618605Z ##[group]Run rocminfo 2025-08-14T21:21:12.1619071Z rocminfo 2025-08-14T21:21:12.1665861Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.1666527Z env: 2025-08-14T21:21:12.1666898Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.1667365Z ##[endgroup] 2025-08-14T21:21:12.3027616Z ROCk module version 6.10.5 is loaded 2025-08-14T21:21:12.3028378Z ===================== 2025-08-14T21:21:12.3028861Z HSA System Attributes 2025-08-14T21:21:12.3029322Z ===================== 2025-08-14T21:21:12.3029781Z Runtime Version: 1.14 2025-08-14T21:21:12.3030258Z Runtime Ext Version: 1.6 2025-08-14T21:21:12.3030761Z System Timestamp Freq.: 1000.000000MHz 2025-08-14T21:21:12.3031607Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2025-08-14T21:21:12.3032594Z Machine Model: LARGE 2025-08-14T21:21:12.3033322Z System Endianness: LITTLE 2025-08-14T21:21:12.3033966Z Mwaitx: DISABLED 2025-08-14T21:21:12.3034444Z DMAbuf Support: YES 2025-08-14T21:21:12.3034753Z 2025-08-14T21:21:12.3034902Z ========== 2025-08-14T21:21:12.3035317Z HSA Agents 2025-08-14T21:21:12.3036506Z ========== 2025-08-14T21:21:12.3037373Z ******* 2025-08-14T21:21:12.3037848Z Agent 1 2025-08-14T21:21:12.3038295Z ******* 2025-08-14T21:21:12.3038790Z Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:21:12.3039537Z Uuid: CPU-XX 2025-08-14T21:21:12.3040419Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:21:12.3041255Z Vendor Name: CPU 2025-08-14T21:21:12.3042038Z Feature: None specified 2025-08-14T21:21:12.3042807Z Profile: FULL_PROFILE 2025-08-14T21:21:12.3043579Z Float Round Mode: NEAR 2025-08-14T21:21:12.3044248Z Max Queue Number: 0(0x0) 2025-08-14T21:21:12.3044918Z Queue Min Size: 0(0x0) 2025-08-14T21:21:12.3045581Z Queue Max Size: 0(0x0) 2025-08-14T21:21:12.3046228Z Queue Type: MULTI 2025-08-14T21:21:12.3046833Z Node: 0 2025-08-14T21:21:12.3047434Z Device Type: CPU 2025-08-14T21:21:12.3048024Z Cache Info: 2025-08-14T21:21:12.3048495Z L1: 32768(0x8000) KB 2025-08-14T21:21:12.3049102Z Chip ID: 0(0x0) 2025-08-14T21:21:12.3049740Z ASIC Revision: 0(0x0) 2025-08-14T21:21:12.3050414Z Cacheline Size: 64(0x40) 2025-08-14T21:21:12.3051098Z Max Clock Freq. (MHz): 2000 2025-08-14T21:21:12.3051715Z BDFID: 0 2025-08-14T21:21:12.3052347Z Internal Node ID: 0 2025-08-14T21:21:12.3053014Z Compute Unit: 64 2025-08-14T21:21:12.3053673Z SIMDs per CU: 0 2025-08-14T21:21:12.3054328Z Shader Engines: 0 2025-08-14T21:21:12.3055016Z Shader Arrs. per Eng.: 0 2025-08-14T21:21:12.3055733Z WatchPts on Addr. Ranges:1 2025-08-14T21:21:12.3056476Z Memory Properties: 2025-08-14T21:21:12.3057021Z Features: None 2025-08-14T21:21:12.3057544Z Pool Info: 2025-08-14T21:21:12.3058064Z Pool 1 2025-08-14T21:21:12.3058644Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3059324Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:21:12.3059970Z Allocatable: TRUE 2025-08-14T21:21:12.3060677Z Alloc Granule: 4KB 2025-08-14T21:21:12.3061408Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3062128Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3062831Z Accessible by all: TRUE 2025-08-14T21:21:12.3063419Z Pool 2 2025-08-14T21:21:12.3063964Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3064619Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:21:12.3065297Z Allocatable: TRUE 2025-08-14T21:21:12.3065983Z Alloc Granule: 4KB 2025-08-14T21:21:12.3066701Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3067434Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3068760Z Accessible by all: TRUE 2025-08-14T21:21:12.3069383Z Pool 3 2025-08-14T21:21:12.3069926Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2025-08-14T21:21:12.3070582Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:21:12.3071220Z Allocatable: TRUE 2025-08-14T21:21:12.3071914Z Alloc Granule: 4KB 2025-08-14T21:21:12.3072621Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3073326Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3074029Z Accessible by all: TRUE 2025-08-14T21:21:12.3074618Z Pool 4 2025-08-14T21:21:12.3075145Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3075799Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:21:12.3076601Z Allocatable: TRUE 2025-08-14T21:21:12.3077781Z Alloc Granule: 4KB 2025-08-14T21:21:12.3078610Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3079407Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3080096Z Accessible by all: TRUE 2025-08-14T21:21:12.3080814Z ISA Info: 2025-08-14T21:21:12.3081226Z ******* 2025-08-14T21:21:12.3081626Z Agent 2 2025-08-14T21:21:12.3082081Z ******* 2025-08-14T21:21:12.3082632Z Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:21:12.3083395Z Uuid: CPU-XX 2025-08-14T21:21:12.3084142Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:21:12.3084863Z Vendor Name: CPU 2025-08-14T21:21:12.3085518Z Feature: None specified 2025-08-14T21:21:12.3086177Z Profile: FULL_PROFILE 2025-08-14T21:21:12.3086837Z Float Round Mode: NEAR 2025-08-14T21:21:12.3087522Z Max Queue Number: 0(0x0) 2025-08-14T21:21:12.3088188Z Queue Min Size: 0(0x0) 2025-08-14T21:21:12.3088826Z Queue Max Size: 0(0x0) 2025-08-14T21:21:12.3089573Z Queue Type: MULTI 2025-08-14T21:21:12.3090282Z Node: 1 2025-08-14T21:21:12.3091018Z Device Type: CPU 2025-08-14T21:21:12.3091690Z Cache Info: 2025-08-14T21:21:12.3092252Z L1: 32768(0x8000) KB 2025-08-14T21:21:12.3092977Z Chip ID: 0(0x0) 2025-08-14T21:21:12.3093720Z ASIC Revision: 0(0x0) 2025-08-14T21:21:12.3094430Z Cacheline Size: 64(0x40) 2025-08-14T21:21:12.3095090Z Max Clock Freq. (MHz): 2000 2025-08-14T21:21:12.3095715Z BDFID: 0 2025-08-14T21:21:12.3096342Z Internal Node ID: 1 2025-08-14T21:21:12.3097019Z Compute Unit: 64 2025-08-14T21:21:12.3097666Z SIMDs per CU: 0 2025-08-14T21:21:12.3098315Z Shader Engines: 0 2025-08-14T21:21:12.3099008Z Shader Arrs. per Eng.: 0 2025-08-14T21:21:12.3099782Z WatchPts on Addr. Ranges:1 2025-08-14T21:21:12.3101259Z Memory Properties: 2025-08-14T21:21:12.3101796Z Features: None 2025-08-14T21:21:12.3102306Z Pool Info: 2025-08-14T21:21:12.3102723Z Pool 1 2025-08-14T21:21:12.3103268Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3103941Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:21:12.3104569Z Allocatable: TRUE 2025-08-14T21:21:12.3105249Z Alloc Granule: 4KB 2025-08-14T21:21:12.3105953Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3106686Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3107382Z Accessible by all: TRUE 2025-08-14T21:21:12.3107988Z Pool 2 2025-08-14T21:21:12.3108552Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3109190Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:21:12.3109839Z Allocatable: TRUE 2025-08-14T21:21:12.3110508Z Alloc Granule: 4KB 2025-08-14T21:21:12.3111223Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3111971Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3112690Z Accessible by all: TRUE 2025-08-14T21:21:12.3113313Z Pool 3 2025-08-14T21:21:12.3113944Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2025-08-14T21:21:12.3114621Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:21:12.3115254Z Allocatable: TRUE 2025-08-14T21:21:12.3115969Z Alloc Granule: 4KB 2025-08-14T21:21:12.3116669Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3117393Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3118094Z Accessible by all: TRUE 2025-08-14T21:21:12.3118696Z Pool 4 2025-08-14T21:21:12.3119227Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3119858Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:21:12.3120820Z Allocatable: TRUE 2025-08-14T21:21:12.3121494Z Alloc Granule: 4KB 2025-08-14T21:21:12.3122243Z Alloc Recommended Granule:4KB 2025-08-14T21:21:12.3122979Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3123716Z Accessible by all: TRUE 2025-08-14T21:21:12.3124324Z ISA Info: 2025-08-14T21:21:12.3124741Z ******* 2025-08-14T21:21:12.3125154Z Agent 3 2025-08-14T21:21:12.3125540Z ******* 2025-08-14T21:21:12.3126050Z Name: gfx90a 2025-08-14T21:21:12.3126774Z Uuid: GPU-bb978094b96db57c 2025-08-14T21:21:12.3127576Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3128379Z Vendor Name: AMD 2025-08-14T21:21:12.3129156Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3129938Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3130726Z Float Round Mode: NEAR 2025-08-14T21:21:12.3131538Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3132987Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3133656Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3134292Z Queue Type: MULTI 2025-08-14T21:21:12.3134896Z Node: 2 2025-08-14T21:21:12.3135507Z Device Type: GPU 2025-08-14T21:21:12.3136081Z Cache Info: 2025-08-14T21:21:12.3136546Z L1: 16(0x10) KB 2025-08-14T21:21:12.3137107Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3137683Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3138307Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3138974Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3139643Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3140279Z BDFID: 12544 2025-08-14T21:21:12.3140913Z Internal Node ID: 2 2025-08-14T21:21:12.3141568Z Compute Unit: 104 2025-08-14T21:21:12.3142219Z SIMDs per CU: 4 2025-08-14T21:21:12.3142866Z Shader Engines: 8 2025-08-14T21:21:12.3143545Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3144233Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3144944Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3145567Z Memory Properties: 2025-08-14T21:21:12.3146086Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3146832Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3147666Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3148488Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3149220Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3149760Z x 1024(0x400) 2025-08-14T21:21:12.3150311Z y 1024(0x400) 2025-08-14T21:21:12.3150835Z z 1024(0x400) 2025-08-14T21:21:12.3151435Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3152100Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3152773Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3153358Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3153838Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3154375Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3154938Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3155576Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3166985Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3168012Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3168866Z IOMMU Support:: None 2025-08-14T21:21:12.3169595Z Pool Info: 2025-08-14T21:21:12.3170127Z Pool 1 2025-08-14T21:21:12.3170813Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3171645Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3172390Z Allocatable: TRUE 2025-08-14T21:21:12.3173079Z Alloc Granule: 4KB 2025-08-14T21:21:12.3174222Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3175260Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3175951Z Accessible by all: FALSE 2025-08-14T21:21:12.3176556Z Pool 2 2025-08-14T21:21:12.3177103Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3177807Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3178597Z Allocatable: TRUE 2025-08-14T21:21:12.3179398Z Alloc Granule: 4KB 2025-08-14T21:21:12.3180256Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3181100Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3181808Z Accessible by all: FALSE 2025-08-14T21:21:12.3182407Z Pool 3 2025-08-14T21:21:12.3182950Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3183589Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3184219Z Allocatable: TRUE 2025-08-14T21:21:12.3184904Z Alloc Granule: 4KB 2025-08-14T21:21:12.3185607Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3186338Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3187042Z Accessible by all: FALSE 2025-08-14T21:21:12.3187626Z Pool 4 2025-08-14T21:21:12.3188142Z Segment: GROUP 2025-08-14T21:21:12.3188751Z Size: 64(0x40) KB 2025-08-14T21:21:12.3189389Z Allocatable: FALSE 2025-08-14T21:21:12.3190086Z Alloc Granule: 0KB 2025-08-14T21:21:12.3190797Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3191521Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3192212Z Accessible by all: FALSE 2025-08-14T21:21:12.3192813Z ISA Info: 2025-08-14T21:21:12.3193231Z ISA 1 2025-08-14T21:21:12.3193786Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3194515Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3195227Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3195917Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3196640Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3197325Z Fast f16: TRUE 2025-08-14T21:21:12.3197978Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3198619Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3199165Z x 1024(0x400) 2025-08-14T21:21:12.3199724Z y 1024(0x400) 2025-08-14T21:21:12.3200607Z z 1024(0x400) 2025-08-14T21:21:12.3201253Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3201859Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3202354Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3202958Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3203505Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3204504Z FBarrier Max Size: 32 2025-08-14T21:21:12.3205419Z ******* 2025-08-14T21:21:12.3205829Z Agent 4 2025-08-14T21:21:12.3206283Z ******* 2025-08-14T21:21:12.3206810Z Name: gfx90a 2025-08-14T21:21:12.3207559Z Uuid: GPU-f4a8e4f76a6ff8b4 2025-08-14T21:21:12.3208358Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3209188Z Vendor Name: AMD 2025-08-14T21:21:12.3209956Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3210733Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3211532Z Float Round Mode: NEAR 2025-08-14T21:21:12.3212240Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3212835Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3213412Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3213987Z Queue Type: MULTI 2025-08-14T21:21:12.3214507Z Node: 3 2025-08-14T21:21:12.3215049Z Device Type: GPU 2025-08-14T21:21:12.3215552Z Cache Info: 2025-08-14T21:21:12.3215973Z L1: 16(0x10) KB 2025-08-14T21:21:12.3216476Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3216986Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3217566Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3218155Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3218756Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3219310Z BDFID: 13312 2025-08-14T21:21:12.3219891Z Internal Node ID: 3 2025-08-14T21:21:12.3220633Z Compute Unit: 104 2025-08-14T21:21:12.3221194Z SIMDs per CU: 4 2025-08-14T21:21:12.3221779Z Shader Engines: 8 2025-08-14T21:21:12.3222366Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3222988Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3223611Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3224182Z Memory Properties: 2025-08-14T21:21:12.3224607Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3225144Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3225753Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3226465Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3227122Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3227657Z x 1024(0x400) 2025-08-14T21:21:12.3228224Z y 1024(0x400) 2025-08-14T21:21:12.3228764Z z 1024(0x400) 2025-08-14T21:21:12.3229384Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3229985Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3230575Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3231108Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3231524Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3232011Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3232477Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3233502Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3234169Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3234792Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3235401Z IOMMU Support:: None 2025-08-14T21:21:12.3235908Z Pool Info: 2025-08-14T21:21:12.3236294Z Pool 1 2025-08-14T21:21:12.3236776Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3237374Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3237960Z Allocatable: TRUE 2025-08-14T21:21:12.3238558Z Alloc Granule: 4KB 2025-08-14T21:21:12.3239193Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3239846Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3240565Z Accessible by all: FALSE 2025-08-14T21:21:12.3241084Z Pool 2 2025-08-14T21:21:12.3241563Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3242142Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3242707Z Allocatable: TRUE 2025-08-14T21:21:12.3243307Z Alloc Granule: 4KB 2025-08-14T21:21:12.3243917Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3244556Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3245169Z Accessible by all: FALSE 2025-08-14T21:21:12.3245702Z Pool 3 2025-08-14T21:21:12.3246214Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3246885Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3247547Z Allocatable: TRUE 2025-08-14T21:21:12.3248243Z Alloc Granule: 4KB 2025-08-14T21:21:12.3248977Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3249712Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3250442Z Accessible by all: FALSE 2025-08-14T21:21:12.3275283Z Pool 4 2025-08-14T21:21:12.3275825Z Segment: GROUP 2025-08-14T21:21:12.3276440Z Size: 64(0x40) KB 2025-08-14T21:21:12.3277026Z Allocatable: FALSE 2025-08-14T21:21:12.3277628Z Alloc Granule: 0KB 2025-08-14T21:21:12.3278281Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3278914Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3279516Z Accessible by all: FALSE 2025-08-14T21:21:12.3280042Z ISA Info: 2025-08-14T21:21:12.3280538Z ISA 1 2025-08-14T21:21:12.3281033Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3281686Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3282295Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3282914Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3283529Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3284103Z Fast f16: TRUE 2025-08-14T21:21:12.3285350Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3285942Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3286541Z x 1024(0x400) 2025-08-14T21:21:12.3287130Z y 1024(0x400) 2025-08-14T21:21:12.3287681Z z 1024(0x400) 2025-08-14T21:21:12.3288299Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3288930Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3289442Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3290013Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3290579Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3291211Z FBarrier Max Size: 32 2025-08-14T21:21:12.3291839Z ******* 2025-08-14T21:21:12.3292264Z Agent 5 2025-08-14T21:21:12.3292651Z ******* 2025-08-14T21:21:12.3293048Z Name: gfx90a 2025-08-14T21:21:12.3293599Z Uuid: GPU-5dd3a7090d90a096 2025-08-14T21:21:12.3294184Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3294787Z Vendor Name: AMD 2025-08-14T21:21:12.3295352Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3295918Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3296501Z Float Round Mode: NEAR 2025-08-14T21:21:12.3297079Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3297650Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3298203Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3298794Z Queue Type: MULTI 2025-08-14T21:21:12.3299324Z Node: 4 2025-08-14T21:21:12.3299847Z Device Type: GPU 2025-08-14T21:21:12.3300347Z Cache Info: 2025-08-14T21:21:12.3300765Z L1: 16(0x10) KB 2025-08-14T21:21:12.3301268Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3301774Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3302338Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3302919Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3303502Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3304042Z BDFID: 4352 2025-08-14T21:21:12.3304603Z Internal Node ID: 4 2025-08-14T21:21:12.3305183Z Compute Unit: 104 2025-08-14T21:21:12.3305738Z SIMDs per CU: 4 2025-08-14T21:21:12.3306303Z Shader Engines: 8 2025-08-14T21:21:12.3306892Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3307492Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3308118Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3308655Z Memory Properties: 2025-08-14T21:21:12.3309079Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3309608Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3310266Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3310852Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3311873Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3312340Z x 1024(0x400) 2025-08-14T21:21:12.3312806Z y 1024(0x400) 2025-08-14T21:21:12.3313268Z z 1024(0x400) 2025-08-14T21:21:12.3313778Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3314365Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3314937Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3315465Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3315876Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3316393Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3316957Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3317638Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3318406Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3319124Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3319793Z IOMMU Support:: None 2025-08-14T21:21:12.3320600Z Pool Info: 2025-08-14T21:21:12.3320977Z Pool 1 2025-08-14T21:21:12.3321463Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3322031Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3322602Z Allocatable: TRUE 2025-08-14T21:21:12.3323194Z Alloc Granule: 4KB 2025-08-14T21:21:12.3323824Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3324460Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3325078Z Accessible by all: FALSE 2025-08-14T21:21:12.3325596Z Pool 2 2025-08-14T21:21:12.3326055Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3326741Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3327419Z Allocatable: TRUE 2025-08-14T21:21:12.3328126Z Alloc Granule: 4KB 2025-08-14T21:21:12.3328863Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3329603Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3330320Z Accessible by all: FALSE 2025-08-14T21:21:12.3330930Z Pool 3 2025-08-14T21:21:12.3331487Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3332143Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3332775Z Allocatable: TRUE 2025-08-14T21:21:12.3333367Z Alloc Granule: 4KB 2025-08-14T21:21:12.3333981Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3334603Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3335198Z Accessible by all: FALSE 2025-08-14T21:21:12.3335708Z Pool 4 2025-08-14T21:21:12.3336142Z Segment: GROUP 2025-08-14T21:21:12.3336682Z Size: 64(0x40) KB 2025-08-14T21:21:12.3337227Z Allocatable: FALSE 2025-08-14T21:21:12.3337819Z Alloc Granule: 0KB 2025-08-14T21:21:12.3338915Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3339965Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3340684Z Accessible by all: FALSE 2025-08-14T21:21:12.3341305Z ISA Info: 2025-08-14T21:21:12.3341677Z ISA 1 2025-08-14T21:21:12.3342150Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3342793Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3343406Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3344007Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3344631Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3345202Z Fast f16: TRUE 2025-08-14T21:21:12.3345778Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3346346Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3346824Z x 1024(0x400) 2025-08-14T21:21:12.3347312Z y 1024(0x400) 2025-08-14T21:21:12.3347774Z z 1024(0x400) 2025-08-14T21:21:12.3348294Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3348817Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3349254Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3349734Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3350212Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3350752Z FBarrier Max Size: 32 2025-08-14T21:21:12.3351257Z ******* 2025-08-14T21:21:12.3351604Z Agent 6 2025-08-14T21:21:12.3351957Z ******* 2025-08-14T21:21:12.3352353Z Name: gfx90a 2025-08-14T21:21:12.3352886Z Uuid: GPU-2b7505fff509489f 2025-08-14T21:21:12.3353463Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3354060Z Vendor Name: AMD 2025-08-14T21:21:12.3354626Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3355196Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3355774Z Float Round Mode: NEAR 2025-08-14T21:21:12.3356363Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3356929Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3357487Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3358054Z Queue Type: MULTI 2025-08-14T21:21:12.3358582Z Node: 5 2025-08-14T21:21:12.3359114Z Device Type: GPU 2025-08-14T21:21:12.3359617Z Cache Info: 2025-08-14T21:21:12.3360035Z L1: 16(0x10) KB 2025-08-14T21:21:12.3360623Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3361124Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3361665Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3362254Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3362835Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3363370Z BDFID: 5120 2025-08-14T21:21:12.3363923Z Internal Node ID: 5 2025-08-14T21:21:12.3365102Z Compute Unit: 104 2025-08-14T21:21:12.3365679Z SIMDs per CU: 4 2025-08-14T21:21:12.3366256Z Shader Engines: 8 2025-08-14T21:21:12.3366959Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3367670Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3368399Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3369035Z Memory Properties: 2025-08-14T21:21:12.3369526Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3370165Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3370864Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3371570Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3372220Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3372744Z x 1024(0x400) 2025-08-14T21:21:12.3373215Z y 1024(0x400) 2025-08-14T21:21:12.3373669Z z 1024(0x400) 2025-08-14T21:21:12.3374187Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3374768Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3375351Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3375859Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3376278Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3376761Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3377238Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3377790Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3378540Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3379279Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3379971Z IOMMU Support:: None 2025-08-14T21:21:12.3380576Z Pool Info: 2025-08-14T21:21:12.3381004Z Pool 1 2025-08-14T21:21:12.3381536Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3382112Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3382672Z Allocatable: TRUE 2025-08-14T21:21:12.3383262Z Alloc Granule: 4KB 2025-08-14T21:21:12.3383883Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3384516Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3385117Z Accessible by all: FALSE 2025-08-14T21:21:12.3385660Z Pool 2 2025-08-14T21:21:12.3386129Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3386697Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3387262Z Allocatable: TRUE 2025-08-14T21:21:12.3387848Z Alloc Granule: 4KB 2025-08-14T21:21:12.3388467Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3389086Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3389693Z Accessible by all: FALSE 2025-08-14T21:21:12.3390221Z Pool 3 2025-08-14T21:21:12.3390672Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3391223Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3392233Z Allocatable: TRUE 2025-08-14T21:21:12.3392841Z Alloc Granule: 4KB 2025-08-14T21:21:12.3393464Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3394117Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3394753Z Accessible by all: FALSE 2025-08-14T21:21:12.3395277Z Pool 4 2025-08-14T21:21:12.3395729Z Segment: GROUP 2025-08-14T21:21:12.3396270Z Size: 64(0x40) KB 2025-08-14T21:21:12.3396846Z Allocatable: FALSE 2025-08-14T21:21:12.3397440Z Alloc Granule: 0KB 2025-08-14T21:21:12.3398074Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3398743Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3399352Z Accessible by all: FALSE 2025-08-14T21:21:12.3399898Z ISA Info: 2025-08-14T21:21:12.3400344Z ISA 1 2025-08-14T21:21:12.3400846Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3401495Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3402151Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3402791Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3403421Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3404014Z Fast f16: TRUE 2025-08-14T21:21:12.3404603Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3405180Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3405680Z x 1024(0x400) 2025-08-14T21:21:12.3406178Z y 1024(0x400) 2025-08-14T21:21:12.3406719Z z 1024(0x400) 2025-08-14T21:21:12.3407364Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3408004Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3408521Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3409121Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3409699Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3410362Z FBarrier Max Size: 32 2025-08-14T21:21:12.3410967Z ******* 2025-08-14T21:21:12.3411406Z Agent 7 2025-08-14T21:21:12.3411834Z ******* 2025-08-14T21:21:12.3412324Z Name: gfx90a 2025-08-14T21:21:12.3412941Z Uuid: GPU-2d3a0c893d332691 2025-08-14T21:21:12.3413534Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3414150Z Vendor Name: AMD 2025-08-14T21:21:12.3414731Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3415326Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3415918Z Float Round Mode: NEAR 2025-08-14T21:21:12.3416532Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3417117Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3417688Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3418316Z Queue Type: MULTI 2025-08-14T21:21:12.3419614Z Node: 6 2025-08-14T21:21:12.3420285Z Device Type: GPU 2025-08-14T21:21:12.3420894Z Cache Info: 2025-08-14T21:21:12.3421398Z L1: 16(0x10) KB 2025-08-14T21:21:12.3421910Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3422426Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3422998Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3423579Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3424170Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3424709Z BDFID: 44544 2025-08-14T21:21:12.3425270Z Internal Node ID: 6 2025-08-14T21:21:12.3425864Z Compute Unit: 104 2025-08-14T21:21:12.3426435Z SIMDs per CU: 4 2025-08-14T21:21:12.3427012Z Shader Engines: 8 2025-08-14T21:21:12.3427607Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3428224Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3428833Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3429373Z Memory Properties: 2025-08-14T21:21:12.3429812Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3430355Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3430979Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3431585Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3432168Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3432622Z x 1024(0x400) 2025-08-14T21:21:12.3433126Z y 1024(0x400) 2025-08-14T21:21:12.3433585Z z 1024(0x400) 2025-08-14T21:21:12.3434118Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3434723Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3435314Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3435851Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3436267Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3436762Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3437236Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3437796Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3438438Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3439077Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3439680Z IOMMU Support:: None 2025-08-14T21:21:12.3440412Z Pool Info: 2025-08-14T21:21:12.3440817Z Pool 1 2025-08-14T21:21:12.3441292Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3441876Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3442455Z Allocatable: TRUE 2025-08-14T21:21:12.3443071Z Alloc Granule: 4KB 2025-08-14T21:21:12.3443710Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3444340Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3444949Z Accessible by all: FALSE 2025-08-14T21:21:12.3445467Z Pool 2 2025-08-14T21:21:12.3446565Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3447264Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3447920Z Allocatable: TRUE 2025-08-14T21:21:12.3448621Z Alloc Granule: 4KB 2025-08-14T21:21:12.3449342Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3450080Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3450791Z Accessible by all: FALSE 2025-08-14T21:21:12.3451416Z Pool 3 2025-08-14T21:21:12.3451970Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3452613Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3453187Z Allocatable: TRUE 2025-08-14T21:21:12.3453798Z Alloc Granule: 4KB 2025-08-14T21:21:12.3454419Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3455037Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3455645Z Accessible by all: FALSE 2025-08-14T21:21:12.3456168Z Pool 4 2025-08-14T21:21:12.3456602Z Segment: GROUP 2025-08-14T21:21:12.3457134Z Size: 64(0x40) KB 2025-08-14T21:21:12.3457685Z Allocatable: FALSE 2025-08-14T21:21:12.3458336Z Alloc Granule: 0KB 2025-08-14T21:21:12.3459064Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3459802Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3460536Z Accessible by all: FALSE 2025-08-14T21:21:12.3461150Z ISA Info: 2025-08-14T21:21:12.3461559Z ISA 1 2025-08-14T21:21:12.3462035Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3462682Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3463298Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3463907Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3464532Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3465123Z Fast f16: TRUE 2025-08-14T21:21:12.3465696Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3466251Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3466735Z x 1024(0x400) 2025-08-14T21:21:12.3467232Z y 1024(0x400) 2025-08-14T21:21:12.3467705Z z 1024(0x400) 2025-08-14T21:21:12.3468229Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3468761Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3469194Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3469677Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3470153Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3470685Z FBarrier Max Size: 32 2025-08-14T21:21:12.3471195Z ******* 2025-08-14T21:21:12.3471541Z Agent 8 2025-08-14T21:21:12.3471883Z ******* 2025-08-14T21:21:12.3472279Z Name: gfx90a 2025-08-14T21:21:12.3473344Z Uuid: GPU-8f10eb83a29cc55d 2025-08-14T21:21:12.3473947Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3474539Z Vendor Name: AMD 2025-08-14T21:21:12.3475117Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3475688Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3476271Z Float Round Mode: NEAR 2025-08-14T21:21:12.3476854Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3477446Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3478020Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3478576Z Queue Type: MULTI 2025-08-14T21:21:12.3479105Z Node: 7 2025-08-14T21:21:12.3479652Z Device Type: GPU 2025-08-14T21:21:12.3480157Z Cache Info: 2025-08-14T21:21:12.3480652Z L1: 16(0x10) KB 2025-08-14T21:21:12.3481132Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3481632Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3482182Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3482766Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3483338Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3483881Z BDFID: 45824 2025-08-14T21:21:12.3484424Z Internal Node ID: 7 2025-08-14T21:21:12.3485001Z Compute Unit: 104 2025-08-14T21:21:12.3485572Z SIMDs per CU: 4 2025-08-14T21:21:12.3486149Z Shader Engines: 8 2025-08-14T21:21:12.3486805Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3487518Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3488250Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3488881Z Memory Properties: 2025-08-14T21:21:12.3489377Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3490004Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3490727Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3491429Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3492072Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3492598Z x 1024(0x400) 2025-08-14T21:21:12.3493116Z y 1024(0x400) 2025-08-14T21:21:12.3493588Z z 1024(0x400) 2025-08-14T21:21:12.3494102Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3494691Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3495273Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3495787Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3496199Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3496679Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3497155Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3497706Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3498391Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3499130Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3500506Z IOMMU Support:: None 2025-08-14T21:21:12.3501129Z Pool Info: 2025-08-14T21:21:12.3501564Z Pool 1 2025-08-14T21:21:12.3502055Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3502629Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3503204Z Allocatable: TRUE 2025-08-14T21:21:12.3503792Z Alloc Granule: 4KB 2025-08-14T21:21:12.3504421Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3505051Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3505650Z Accessible by all: FALSE 2025-08-14T21:21:12.3506172Z Pool 2 2025-08-14T21:21:12.3506648Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3507263Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3507825Z Allocatable: TRUE 2025-08-14T21:21:12.3508432Z Alloc Granule: 4KB 2025-08-14T21:21:12.3509070Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3509697Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3510330Z Accessible by all: FALSE 2025-08-14T21:21:12.3510852Z Pool 3 2025-08-14T21:21:12.3511334Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3511894Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3512475Z Allocatable: TRUE 2025-08-14T21:21:12.3513084Z Alloc Granule: 4KB 2025-08-14T21:21:12.3513723Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3514359Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3514972Z Accessible by all: FALSE 2025-08-14T21:21:12.3515502Z Pool 4 2025-08-14T21:21:12.3515950Z Segment: GROUP 2025-08-14T21:21:12.3516501Z Size: 64(0x40) KB 2025-08-14T21:21:12.3517061Z Allocatable: FALSE 2025-08-14T21:21:12.3517649Z Alloc Granule: 0KB 2025-08-14T21:21:12.3518280Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3518898Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3519513Z Accessible by all: FALSE 2025-08-14T21:21:12.3520056Z ISA Info: 2025-08-14T21:21:12.3520714Z ISA 1 2025-08-14T21:21:12.3521208Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3521849Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3522481Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3523089Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3523728Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3524309Z Fast f16: TRUE 2025-08-14T21:21:12.3524917Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3525523Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3526022Z x 1024(0x400) 2025-08-14T21:21:12.3526900Z y 1024(0x400) 2025-08-14T21:21:12.3527803Z z 1024(0x400) 2025-08-14T21:21:12.3528468Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3529112Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3529668Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3530250Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3530843Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3531523Z FBarrier Max Size: 32 2025-08-14T21:21:12.3532147Z ******* 2025-08-14T21:21:12.3532487Z Agent 9 2025-08-14T21:21:12.3532684Z ******* 2025-08-14T21:21:12.3532926Z Name: gfx90a 2025-08-14T21:21:12.3533195Z Uuid: GPU-2987fc4d26ca2e7a 2025-08-14T21:21:12.3533497Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3533792Z Vendor Name: AMD 2025-08-14T21:21:12.3534070Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3534364Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3534643Z Float Round Mode: NEAR 2025-08-14T21:21:12.3534942Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3535224Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3535510Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3535782Z Queue Type: MULTI 2025-08-14T21:21:12.3536048Z Node: 8 2025-08-14T21:21:12.3536323Z Device Type: GPU 2025-08-14T21:21:12.3536580Z Cache Info: 2025-08-14T21:21:12.3536797Z L1: 16(0x10) KB 2025-08-14T21:21:12.3537033Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3537288Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3537548Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3537832Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3538116Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3538375Z BDFID: 36352 2025-08-14T21:21:12.3538652Z Internal Node ID: 8 2025-08-14T21:21:12.3538942Z Compute Unit: 104 2025-08-14T21:21:12.3539217Z SIMDs per CU: 4 2025-08-14T21:21:12.3539503Z Shader Engines: 8 2025-08-14T21:21:12.3539795Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3540095Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3540388Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3540655Z Memory Properties: 2025-08-14T21:21:12.3540858Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3541125Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3541406Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3541704Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3541974Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3542193Z x 1024(0x400) 2025-08-14T21:21:12.3542431Z y 1024(0x400) 2025-08-14T21:21:12.3542896Z z 1024(0x400) 2025-08-14T21:21:12.3543160Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3543445Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3543752Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3543999Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3544210Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3544458Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3544688Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3544973Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3545280Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3545585Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3545871Z IOMMU Support:: None 2025-08-14T21:21:12.3546144Z Pool Info: 2025-08-14T21:21:12.3546335Z Pool 1 2025-08-14T21:21:12.3546567Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3546861Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3547140Z Allocatable: TRUE 2025-08-14T21:21:12.3547436Z Alloc Granule: 4KB 2025-08-14T21:21:12.3547738Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3548058Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3548362Z Accessible by all: FALSE 2025-08-14T21:21:12.3548614Z Pool 2 2025-08-14T21:21:12.3548859Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3549147Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3549446Z Allocatable: TRUE 2025-08-14T21:21:12.3549735Z Alloc Granule: 4KB 2025-08-14T21:21:12.3550053Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3550369Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3550664Z Accessible by all: FALSE 2025-08-14T21:21:12.3550945Z Pool 3 2025-08-14T21:21:12.3551169Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3551443Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3551708Z Allocatable: TRUE 2025-08-14T21:21:12.3551998Z Alloc Granule: 4KB 2025-08-14T21:21:12.3552299Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3552611Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3552912Z Accessible by all: FALSE 2025-08-14T21:21:12.3553160Z Pool 4 2025-08-14T21:21:12.3553387Z Segment: GROUP 2025-08-14T21:21:12.3553642Z Size: 64(0x40) KB 2025-08-14T21:21:12.3553921Z Allocatable: FALSE 2025-08-14T21:21:12.3554203Z Alloc Granule: 0KB 2025-08-14T21:21:12.3554512Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3554820Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3555105Z Accessible by all: FALSE 2025-08-14T21:21:12.3555359Z ISA Info: 2025-08-14T21:21:12.3555764Z ISA 1 2025-08-14T21:21:12.3556008Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3556314Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3556617Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3556928Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3557234Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3557520Z Fast f16: TRUE 2025-08-14T21:21:12.3557798Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3558077Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3558306Z x 1024(0x400) 2025-08-14T21:21:12.3558552Z y 1024(0x400) 2025-08-14T21:21:12.3558807Z z 1024(0x400) 2025-08-14T21:21:12.3559061Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3559340Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3559557Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3559803Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3560034Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3560367Z FBarrier Max Size: 32 2025-08-14T21:21:12.3560623Z ******* 2025-08-14T21:21:12.3560798Z Agent 10 2025-08-14T21:21:12.3560978Z ******* 2025-08-14T21:21:12.3561173Z Name: gfx90a 2025-08-14T21:21:12.3561448Z Uuid: GPU-a88b5a95f52381e5 2025-08-14T21:21:12.3561737Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:21:12.3562052Z Vendor Name: AMD 2025-08-14T21:21:12.3562334Z Feature: KERNEL_DISPATCH 2025-08-14T21:21:12.3562626Z Profile: BASE_PROFILE 2025-08-14T21:21:12.3562916Z Float Round Mode: NEAR 2025-08-14T21:21:12.3563202Z Max Queue Number: 128(0x80) 2025-08-14T21:21:12.3563491Z Queue Min Size: 64(0x40) 2025-08-14T21:21:12.3563767Z Queue Max Size: 131072(0x20000) 2025-08-14T21:21:12.3564056Z Queue Type: MULTI 2025-08-14T21:21:12.3564312Z Node: 9 2025-08-14T21:21:12.3564587Z Device Type: GPU 2025-08-14T21:21:12.3564845Z Cache Info: 2025-08-14T21:21:12.3565048Z L1: 16(0x10) KB 2025-08-14T21:21:12.3565291Z L2: 8192(0x2000) KB 2025-08-14T21:21:12.3565529Z Chip ID: 29708(0x740c) 2025-08-14T21:21:12.3565807Z ASIC Revision: 1(0x1) 2025-08-14T21:21:12.3566089Z Cacheline Size: 128(0x80) 2025-08-14T21:21:12.3566385Z Max Clock Freq. (MHz): 1700 2025-08-14T21:21:12.3566654Z BDFID: 37632 2025-08-14T21:21:12.3566917Z Internal Node ID: 9 2025-08-14T21:21:12.3567201Z Compute Unit: 104 2025-08-14T21:21:12.3567465Z SIMDs per CU: 4 2025-08-14T21:21:12.3567749Z Shader Engines: 8 2025-08-14T21:21:12.3568317Z Shader Arrs. per Eng.: 1 2025-08-14T21:21:12.3568641Z WatchPts on Addr. Ranges:4 2025-08-14T21:21:12.3568953Z Coherent Host Access: FALSE 2025-08-14T21:21:12.3569271Z Memory Properties: 2025-08-14T21:21:12.3569488Z Features: KERNEL_DISPATCH 2025-08-14T21:21:12.3569742Z Fast F16 Operation: TRUE 2025-08-14T21:21:12.3570031Z Wavefront Size: 64(0x40) 2025-08-14T21:21:12.3570312Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3570569Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3570779Z x 1024(0x400) 2025-08-14T21:21:12.3571009Z y 1024(0x400) 2025-08-14T21:21:12.3571231Z z 1024(0x400) 2025-08-14T21:21:12.3571488Z Max Waves Per CU: 32(0x20) 2025-08-14T21:21:12.3571771Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:21:12.3572048Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3572308Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3572510Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3572745Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3572981Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3573244Z Max fbarriers/Workgrp: 32 2025-08-14T21:21:12.3573554Z Packet Processor uCode:: 92 2025-08-14T21:21:12.3573847Z SDMA engine uCode:: 9 2025-08-14T21:21:12.3574130Z IOMMU Support:: None 2025-08-14T21:21:12.3574371Z Pool Info: 2025-08-14T21:21:12.3574563Z Pool 1 2025-08-14T21:21:12.3574818Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:21:12.3575106Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3575412Z Allocatable: TRUE 2025-08-14T21:21:12.3575720Z Alloc Granule: 4KB 2025-08-14T21:21:12.3576041Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3576365Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3576674Z Accessible by all: FALSE 2025-08-14T21:21:12.3576940Z Pool 2 2025-08-14T21:21:12.3577167Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:21:12.3577461Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3577742Z Allocatable: TRUE 2025-08-14T21:21:12.3578038Z Alloc Granule: 4KB 2025-08-14T21:21:12.3578339Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3578650Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3578954Z Accessible by all: FALSE 2025-08-14T21:21:12.3579203Z Pool 3 2025-08-14T21:21:12.3579430Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:21:12.3579693Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:21:12.3579970Z Allocatable: TRUE 2025-08-14T21:21:12.3580258Z Alloc Granule: 4KB 2025-08-14T21:21:12.3580563Z Alloc Recommended Granule:2048KB 2025-08-14T21:21:12.3581092Z Alloc Alignment: 4KB 2025-08-14T21:21:12.3581401Z Accessible by all: FALSE 2025-08-14T21:21:12.3581667Z Pool 4 2025-08-14T21:21:12.3581882Z Segment: GROUP 2025-08-14T21:21:12.3582153Z Size: 64(0x40) KB 2025-08-14T21:21:12.3582418Z Allocatable: FALSE 2025-08-14T21:21:12.3582711Z Alloc Granule: 0KB 2025-08-14T21:21:12.3583011Z Alloc Recommended Granule:0KB 2025-08-14T21:21:12.3583321Z Alloc Alignment: 0KB 2025-08-14T21:21:12.3583609Z Accessible by all: FALSE 2025-08-14T21:21:12.3583855Z ISA Info: 2025-08-14T21:21:12.3584037Z ISA 1 2025-08-14T21:21:12.3584278Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:21:12.3584592Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:21:12.3584888Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:21:12.3585195Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3585506Z Default Rounding Mode: NEAR 2025-08-14T21:21:12.3585790Z Fast f16: TRUE 2025-08-14T21:21:12.3586069Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:21:12.3586332Z Workgroup Max Size per Dimension: 2025-08-14T21:21:12.3586557Z x 1024(0x400) 2025-08-14T21:21:12.3586788Z y 1024(0x400) 2025-08-14T21:21:12.3587018Z z 1024(0x400) 2025-08-14T21:21:12.3587299Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:21:12.3587552Z Grid Max Size per Dimension: 2025-08-14T21:21:12.3587759Z x 4294967295(0xffffffff) 2025-08-14T21:21:12.3587994Z y 4294967295(0xffffffff) 2025-08-14T21:21:12.3588232Z z 4294967295(0xffffffff) 2025-08-14T21:21:12.3588494Z FBarrier Max Size: 32 2025-08-14T21:21:12.3588754Z *** Done *** 2025-08-14T21:21:12.3608464Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2025-08-14T21:21:12.3608805Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2025-08-14T21:21:12.3609358Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2025-08-14T21:21:12.3609885Z if [[ $ngpu -eq 0 ]]; then 2025-08-14T21:21:12.3610176Z  echo "Error: Failed to detect any GPUs on the runner" 2025-08-14T21:21:12.3610450Z  echo "$msg" 2025-08-14T21:21:12.3610626Z  exit 1 2025-08-14T21:21:12.3610798Z fi 2025-08-14T21:21:12.3634693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.3635001Z env: 2025-08-14T21:21:12.3635186Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.3635387Z ##[endgroup] 2025-08-14T21:21:12.5338382Z ##[group]Run pytorch/pytorch/.github/actions/diskspace-cleanup@main 2025-08-14T21:21:12.5338785Z with: 2025-08-14T21:21:12.5338999Z diskspace-cutoff: 70 2025-08-14T21:21:12.5339221Z env: 2025-08-14T21:21:12.5339422Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.5339656Z ##[endgroup] 2025-08-14T21:21:12.5420850Z ##[group]Run set -ex 2025-08-14T21:21:12.5421154Z set -ex 2025-08-14T21:21:12.5421403Z diskspace_cutoff=70 2025-08-14T21:21:12.5421746Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2025-08-14T21:21:12.5422746Z if [ ! -d "$docker_root_dir" ]; then 2025-08-14T21:21:12.5423896Z  echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check." 2025-08-14T21:21:12.5424932Z  exit 0 2025-08-14T21:21:12.5425393Z fi 2025-08-14T21:21:12.5426152Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2025-08-14T21:21:12.5427765Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2025-08-14T21:21:12.5429133Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2025-08-14T21:21:12.5429810Z  docker system prune -af 2025-08-14T21:21:12.5430699Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2025-08-14T21:21:12.5431705Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2025-08-14T21:21:12.5432786Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2025-08-14T21:21:12.5433707Z  echo "$msg" 2025-08-14T21:21:12.5434147Z  exit 1 2025-08-14T21:21:12.5434554Z  else 2025-08-14T21:21:12.5435024Z  difference=$((diskspace - diskspace_new)) 2025-08-14T21:21:12.5435722Z  echo "Diskspace saved: $difference percent" 2025-08-14T21:21:12.5436289Z  fi 2025-08-14T21:21:12.5436651Z fi 2025-08-14T21:21:12.5485213Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.5485897Z env: 2025-08-14T21:21:12.5486273Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.5486707Z ##[endgroup] 2025-08-14T21:21:12.5572401Z + diskspace_cutoff=70 2025-08-14T21:21:12.5581533Z ++ docker info -f '{{.DockerRootDir}}' 2025-08-14T21:21:12.6547983Z + docker_root_dir=/media/4TB/docker-rootless 2025-08-14T21:21:12.6548816Z + '[' '!' -d /media/4TB/docker-rootless ']' 2025-08-14T21:21:12.6564653Z ++ df -H --output=pcent /media/4TB/docker-rootless 2025-08-14T21:21:12.6567524Z ++ sed -n 2p 2025-08-14T21:21:12.6572338Z ++ sed s/%// 2025-08-14T21:21:12.6574467Z ++ sed 's/ //' 2025-08-14T21:21:12.6616778Z + diskspace=35 2025-08-14T21:21:12.6617997Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2025-08-14T21:21:12.6619190Z + [[ 35 -ge 70 ]] 2025-08-14T21:21:12.6676256Z ##[group]Run RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts" 2025-08-14T21:21:12.6677096Z RUNNER_ARTIFACT_DIR="${RUNNER_TEMP}/artifacts" 2025-08-14T21:21:12.6677766Z rm -rf "${RUNNER_ARTIFACT_DIR}" 2025-08-14T21:21:12.6678359Z mkdir -p "${RUNNER_ARTIFACT_DIR}" 2025-08-14T21:21:12.6679140Z echo "RUNNER_ARTIFACT_DIR=${RUNNER_ARTIFACT_DIR}" >> "${GITHUB_ENV}" 2025-08-14T21:21:12.6679868Z  2025-08-14T21:21:12.6680855Z RUNNER_TEST_RESULTS_DIR="${RUNNER_TEMP}/test-results" 2025-08-14T21:21:12.6681634Z rm -rf "${RUNNER_TEST_RESULTS_DIR}" 2025-08-14T21:21:12.6682245Z mkdir -p "${RUNNER_TEST_RESULTS_DIR}" 2025-08-14T21:21:12.6683086Z echo "RUNNER_TEST_RESULTS_DIR=${RUNNER_TEST_RESULTS_DIR}" >> "${GITHUB_ENV}" 2025-08-14T21:21:12.6683862Z  2025-08-14T21:21:12.6684267Z RUNNER_DOCS_DIR="${RUNNER_TEMP}/docs" 2025-08-14T21:21:12.6684824Z rm -rf "${RUNNER_DOCS_DIR}" 2025-08-14T21:21:12.6685357Z mkdir -p "${RUNNER_DOCS_DIR}" 2025-08-14T21:21:12.6686039Z echo "RUNNER_DOCS_DIR=${RUNNER_DOCS_DIR}" >> "${GITHUB_ENV}" 2025-08-14T21:21:12.6732389Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.6733049Z env: 2025-08-14T21:21:12.6733406Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.6733844Z ##[endgroup] 2025-08-14T21:21:12.7053010Z ##[group]Run env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:21:12.7054823Z env | grep '^GITHUB' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:21:12.7055733Z env | grep '^CI' >> "${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:21:12.7106948Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.7107796Z env: 2025-08-14T21:21:12.7108177Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.7108915Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:12.7110048Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:12.7111120Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:12.7111829Z ##[endgroup] 2025-08-14T21:21:12.7330000Z ##[group]Run # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2025-08-14T21:21:12.7331325Z # All GPUs are visible to the runner; visibility, if needed, will be set by run_test.py. 2025-08-14T21:21:12.7332330Z # Add render group for container creation. 2025-08-14T21:21:12.7333086Z render_gid=`cat /etc/group | grep render | cut -d: -f3` 2025-08-14T21:21:12.7334026Z # Ensure GPU isolation if pod is part of kubernetes setup with DEVICE_FLAG. 2025-08-14T21:21:12.7334959Z if [ -f "/etc/podinfo/gha-render-devices" ]; then 2025-08-14T21:21:12.7335717Z  DEVICE_FLAG=$(cat /etc/podinfo/gha-render-devices) 2025-08-14T21:21:12.7336366Z else 2025-08-14T21:21:12.7336799Z  DEVICE_FLAG="--device /dev/dri" 2025-08-14T21:21:12.7337307Z fi 2025-08-14T21:21:12.7338153Z # The --group-add daemon and --group-add bin are needed in the Ubuntu 24.04 and Almalinux OSs respectively. 2025-08-14T21:21:12.7339465Z # This is due to the device files (/dev/kfd & /dev/dri) being owned by video group on bare metal. 2025-08-14T21:21:12.7340693Z # This video group ID maps to subgid 1 inside the docker image due to the /etc/subgid entries. 2025-08-14T21:21:12.7341992Z # The group name corresponding to group ID 1 can change depending on the OS, so both are necessary. 2025-08-14T21:21:12.7344305Z echo "GPU_FLAG=--device=/dev/mem --device=/dev/kfd $DEVICE_FLAG --group-add video --group-add $render_gid --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host" >> "${GITHUB_ENV}" 2025-08-14T21:21:12.7392382Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.7393064Z env: 2025-08-14T21:21:12.7393426Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.7394173Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:12.7395311Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:12.7396367Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:12.7397101Z ##[endgroup] 2025-08-14T21:21:12.7613698Z ##[group]Run ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2025-08-14T21:21:12.7614605Z ngpu=$(rocminfo | grep -c -E 'Name:.*\sgfx') 2025-08-14T21:21:12.7615370Z if [[ $ngpu -lt 4 ]]; then 2025-08-14T21:21:12.7616336Z  echo "Error: only $ngpu GPU(s) detected, at least 4 GPUs are needed for distributed jobs" 2025-08-14T21:21:12.7617169Z  exit 1 2025-08-14T21:21:12.7617558Z fi 2025-08-14T21:21:12.7664246Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:12.7664909Z env: 2025-08-14T21:21:12.7665288Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.7666018Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:12.7667141Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:12.7668192Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:12.7670076Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:12.7672090Z ##[endgroup] 2025-08-14T21:21:12.9456575Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 2025-08-14T21:21:12.9457548Z with: 2025-08-14T21:21:12.9458196Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_s3_and_ecr_read_only 2025-08-14T21:21:12.9458987Z aws-region: us-east-1 2025-08-14T21:21:12.9459453Z role-duration-seconds: 18000 2025-08-14T21:21:12.9460003Z audience: sts.amazonaws.com 2025-08-14T21:21:12.9460540Z env: 2025-08-14T21:21:12.9460940Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:12.9461809Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:12.9463131Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:12.9464126Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:12.9465980Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:12.9467491Z ##[endgroup] 2025-08-14T21:21:13.2943302Z Assuming role with OIDC 2025-08-14T21:21:13.5305467Z Authenticated as assumedRoleId AROAUPVRELQNLLCOPFEJR:GitHubActions 2025-08-14T21:21:13.6157552Z ##[group]Run aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076 2025-08-14T21:21:13.6158540Z with: 2025-08-14T21:21:13.6158985Z mask-password: true 2025-08-14T21:21:13.6159530Z registry-type: private 2025-08-14T21:21:13.6160021Z skip-logout: false 2025-08-14T21:21:13.6160879Z env: 2025-08-14T21:21:13.6161304Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:13.6162158Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:13.6163396Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:13.6164552Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:13.6166440Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:13.6168101Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:13.6168719Z AWS_REGION: us-east-1 2025-08-14T21:21:13.6169819Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:13.6170568Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:13.6181960Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:13.6182457Z ##[endgroup] 2025-08-14T21:21:14.0092984Z Logging into registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.4868978Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-08-14T21:21:14.4869784Z with: 2025-08-14T21:21:14.4870966Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.4872320Z use-custom-docker-registry: true 2025-08-14T21:21:14.4872826Z docker-build-dir: .ci/docker 2025-08-14T21:21:14.4873308Z docker-build-script: ./build.sh 2025-08-14T21:21:14.4873792Z working-directory: . 2025-08-14T21:21:14.4874357Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.4875004Z force-push: false 2025-08-14T21:21:14.4875376Z env: 2025-08-14T21:21:14.4875725Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:14.4876417Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:14.4877433Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:14.4878443Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:14.4880069Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:14.4882151Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:14.4882601Z AWS_REGION: us-east-1 2025-08-14T21:21:14.4883173Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:14.4883774Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:14.4893512Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:14.4893933Z ##[endgroup] 2025-08-14T21:21:14.4930186Z ##[group]Run set -ex 2025-08-14T21:21:14.4930674Z set -ex 2025-08-14T21:21:14.4931044Z  2025-08-14T21:21:14.4931682Z # If the docker build directory or the build script doesn't exist, the action will 2025-08-14T21:21:14.4932819Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-08-14T21:21:14.4933771Z # job could then download the pre-built image as usual 2025-08-14T21:21:14.4934897Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-08-14T21:21:14.4935910Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4936424Z else 2025-08-14T21:21:14.4936831Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4937541Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4938219Z  2025-08-14T21:21:14.4939102Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-08-14T21:21:14.4940105Z  exit 0 2025-08-14T21:21:14.4940457Z fi 2025-08-14T21:21:14.4940794Z  2025-08-14T21:21:14.4941323Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-08-14T21:21:14.4942280Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-08-14T21:21:14.4943127Z  # use it as it is, but first let's extract the tag 2025-08-14T21:21:14.4943895Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-08-14T21:21:14.4944720Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4945481Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4946122Z else 2025-08-14T21:21:14.4946548Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-08-14T21:21:14.4947151Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-08-14T21:21:14.4947764Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-08-14T21:21:14.4948287Z  fi 2025-08-14T21:21:14.4949385Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-08-14T21:21:14.4950369Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4951381Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4952489Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.4953169Z fi 2025-08-14T21:21:14.4996727Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:14.4997356Z env: 2025-08-14T21:21:14.4997722Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:14.4998418Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:14.4999459Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:14.5000536Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:14.5002144Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:14.5003586Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:14.5004404Z AWS_REGION: us-east-1 2025-08-14T21:21:14.5004944Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:14.5005568Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:14.5015292Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:14.5015728Z REPO_NAME: pytorch 2025-08-14T21:21:14.5016912Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5018195Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:21:14.5018669Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-08-14T21:21:14.5019281Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.5019924Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-08-14T21:21:14.5020387Z CUSTOM_TAG_PREFIX: 2025-08-14T21:21:14.5020768Z ##[endgroup] 2025-08-14T21:21:14.5096240Z + [[ -d .ci/docker ]] 2025-08-14T21:21:14.5096842Z + [[ -f .ci/docker/./build.sh ]] 2025-08-14T21:21:14.5097443Z + [[ true == \t\r\u\e ]] 2025-08-14T21:21:14.5097984Z + echo skip=false 2025-08-14T21:21:14.5105351Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-08-14T21:21:14.5121099Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5123525Z ++ awk -F '[:,]' '{print $2}' 2025-08-14T21:21:14.5181320Z + DOCKER_TAG=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5182948Z + echo docker-tag=pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5184843Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5248925Z ##[group]Run set +e 2025-08-14T21:21:14.5249624Z set +e 2025-08-14T21:21:14.5250187Z set -x 2025-08-14T21:21:14.5250661Z  2025-08-14T21:21:14.5251154Z login() { 2025-08-14T21:21:14.5252196Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:21:14.5253311Z } 2025-08-14T21:21:14.5253776Z  2025-08-14T21:21:14.5254193Z retry () { 2025-08-14T21:21:14.5254678Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:21:14.5255245Z } 2025-08-14T21:21:14.5255614Z  2025-08-14T21:21:14.5256030Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:21:14.5256576Z  2025-08-14T21:21:14.5256971Z START_TIME=$(date +%s) 2025-08-14T21:21:14.5257973Z # Wait up to 120 minutes 2025-08-14T21:21:14.5258692Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-08-14T21:21:14.5259648Z  # Check if image already exists, if it does then skip building it 2025-08-14T21:21:14.5260723Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-08-14T21:21:14.5261498Z  exit 0 2025-08-14T21:21:14.5262017Z  fi 2025-08-14T21:21:14.5262470Z  2025-08-14T21:21:14.5263183Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-08-14T21:21:14.5264370Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-08-14T21:21:14.5265540Z  # latter, it will wait for the Docker images to become available before continuing 2025-08-14T21:21:14.5266462Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-08-14T21:21:14.5267186Z  # It's a Docker build job, let's build the image 2025-08-14T21:21:14.5267827Z  break 2025-08-14T21:21:14.5268250Z  else 2025-08-14T21:21:14.5268839Z  # It's a regular build job, wait for the image to become available 2025-08-14T21:21:14.5270004Z  sleep 300 2025-08-14T21:21:14.5270455Z  fi 2025-08-14T21:21:14.5270847Z done 2025-08-14T21:21:14.5271238Z  2025-08-14T21:21:14.5271884Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-08-14T21:21:14.5272922Z # be empty. The default action would be to continue rebuild the image 2025-08-14T21:21:14.5273846Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-08-14T21:21:14.5274647Z  # if we're on the base branch then use the parent commit 2025-08-14T21:21:14.5275353Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-08-14T21:21:14.5275915Z else 2025-08-14T21:21:14.5276512Z  # otherwise we're on a PR, so use the most recent base commit 2025-08-14T21:21:14.5277361Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-08-14T21:21:14.5278003Z fi 2025-08-14T21:21:14.5278375Z  2025-08-14T21:21:14.5278797Z if [[ -z "${MERGE_BASE}" ]]; then 2025-08-14T21:21:14.5279442Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.5280032Z  2025-08-14T21:21:14.5281194Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-08-14T21:21:14.5282273Z  exit 0 2025-08-14T21:21:14.5282750Z fi 2025-08-14T21:21:14.5283189Z  2025-08-14T21:21:14.5283827Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-08-14T21:21:14.5285023Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-08-14T21:21:14.5286050Z  exit 1 2025-08-14T21:21:14.5286441Z fi 2025-08-14T21:21:14.5286810Z  2025-08-14T21:21:14.5287446Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-08-14T21:21:14.5288599Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-08-14T21:21:14.5289752Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-08-14T21:21:14.5291190Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-08-14T21:21:14.5292809Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-08-14T21:21:14.5293806Z fi 2025-08-14T21:21:14.5294219Z  2025-08-14T21:21:14.5294688Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:21:14.5348339Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:14.5349023Z env: 2025-08-14T21:21:14.5349431Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:14.5350585Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:14.5351801Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:14.5352941Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:14.5354813Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:14.5356452Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:14.5356978Z AWS_REGION: us-east-1 2025-08-14T21:21:14.5357651Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:14.5358409Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:14.5370034Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:14.5370495Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:21:14.5371042Z BASE_REVISION: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:21:14.5372401Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5373969Z DOCKER_TAG: pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:14.5375275Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.5375898Z DOCKER_PUSH: 2025-08-14T21:21:14.5376273Z ##[endgroup] 2025-08-14T21:21:14.5459440Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.5460185Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:14.5467454Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:21:14.5471262Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:15.8009755Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2025-08-14T21:21:15.8010989Z Configure a credential helper to remove this warning. See 2025-08-14T21:21:15.8012125Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2025-08-14T21:21:15.8012849Z 2025-08-14T21:21:15.8013403Z Login Succeeded 2025-08-14T21:21:15.8079432Z ++ date +%s 2025-08-14T21:21:15.8107456Z + START_TIME=1755206475 2025-08-14T21:21:15.8117557Z ++ date +%s 2025-08-14T21:21:15.8146195Z + [[ 1755199275 -lt 1755206475 ]] 2025-08-14T21:21:15.8147852Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:16.8377739Z { 2025-08-14T21:21:16.8378291Z "schemaVersion": 2, 2025-08-14T21:21:16.8379184Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-08-14T21:21:16.8380168Z "config": { 2025-08-14T21:21:16.8380938Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-08-14T21:21:16.8381769Z "size": 28646, 2025-08-14T21:21:16.8382585Z "digest": "sha256:460acde6d8cbae0e961d7e3597d10525c881798e91edaf1ab0df7a07e0905a24" 2025-08-14T21:21:16.8383527Z }, 2025-08-14T21:21:16.8383891Z "layers": [ 2025-08-14T21:21:16.8384404Z { 2025-08-14T21:21:16.8385143Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8385921Z "size": 30448173, 2025-08-14T21:21:16.8386694Z "digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495" 2025-08-14T21:21:16.8387545Z }, 2025-08-14T21:21:16.8387881Z { 2025-08-14T21:21:16.8388452Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8389184Z "size": 1554, 2025-08-14T21:21:16.8389899Z "digest": "sha256:c7b4a852a45516e27a9256df90878663d770f96d271d6155d43be78cc5225eef" 2025-08-14T21:21:16.8390711Z }, 2025-08-14T21:21:16.8391048Z { 2025-08-14T21:21:16.8391615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8392350Z "size": 313280717, 2025-08-14T21:21:16.8393110Z "digest": "sha256:c5628c7e7b33c6491651d9cce9ee596279d693cc7c3dcfa912aca94f75341cf7" 2025-08-14T21:21:16.8394625Z }, 2025-08-14T21:21:16.8394995Z { 2025-08-14T21:21:16.8395575Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8396337Z "size": 704, 2025-08-14T21:21:16.8397153Z "digest": "sha256:836ab08052e8eb2bae68e69ae086fd23a5f04a8491c320718ab47f84f03aebb1" 2025-08-14T21:21:16.8398133Z }, 2025-08-14T21:21:16.8398504Z { 2025-08-14T21:21:16.8399077Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8399837Z "size": 1218, 2025-08-14T21:21:16.8400782Z "digest": "sha256:0802c59adcde1d758d228271cc5f1bf6acbc597053ff8e8f3c1073d59a85bf9d" 2025-08-14T21:21:16.8401714Z }, 2025-08-14T21:21:16.8402117Z { 2025-08-14T21:21:16.8402806Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8403661Z "size": 485, 2025-08-14T21:21:16.8404408Z "digest": "sha256:e97311a6a967664cbe10c5027a1ec60c514caa9a1160167d8363088fd1f9fe09" 2025-08-14T21:21:16.8405229Z }, 2025-08-14T21:21:16.8405575Z { 2025-08-14T21:21:16.8406148Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8406878Z "size": 110343847, 2025-08-14T21:21:16.8407649Z "digest": "sha256:fe9f48a18b7ef6bdbd006ea4cb784b7e73e0daa2409a1a053c02bafed7993529" 2025-08-14T21:21:16.8408923Z }, 2025-08-14T21:21:16.8409257Z { 2025-08-14T21:21:16.8409826Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8410553Z "size": 4242, 2025-08-14T21:21:16.8411292Z "digest": "sha256:4d3edeebe7da3872c026bd4e61d5f63aee7f760e1c8cf0eb226e1d95e528735b" 2025-08-14T21:21:16.8412143Z }, 2025-08-14T21:21:16.8412472Z { 2025-08-14T21:21:16.8413027Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8413740Z "size": 1709, 2025-08-14T21:21:16.8414464Z "digest": "sha256:5a5cc76ada432cccf7d18e0eb79379afb95deaaa7afec482406267924d291ae4" 2025-08-14T21:21:16.8415309Z }, 2025-08-14T21:21:16.8415631Z { 2025-08-14T21:21:16.8416183Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8416900Z "size": 724, 2025-08-14T21:21:16.8417603Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:21:16.8418425Z }, 2025-08-14T21:21:16.8418751Z { 2025-08-14T21:21:16.8419382Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8420226Z "size": 3244135942, 2025-08-14T21:21:16.8421026Z "digest": "sha256:ee623dde94b52da337d3cc0e356115ccbc33b9b5ec03f9e06fce9f05ba3572ad" 2025-08-14T21:21:16.8421863Z }, 2025-08-14T21:21:16.8422189Z { 2025-08-14T21:21:16.8422742Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8423459Z "size": 380, 2025-08-14T21:21:16.8424148Z "digest": "sha256:d6226eb61f823984003d5ac28f4d66fec9b27baf5d54a9513286483f5912cd88" 2025-08-14T21:21:16.8424968Z }, 2025-08-14T21:21:16.8425279Z { 2025-08-14T21:21:16.8425833Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8426542Z "size": 234112, 2025-08-14T21:21:16.8427258Z "digest": "sha256:6de917b18d3a540adb3b7e446e9802f89a69d39f804f40d25751639b6cea7c33" 2025-08-14T21:21:16.8428111Z }, 2025-08-14T21:21:16.8428432Z { 2025-08-14T21:21:16.8428982Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8429767Z "size": 793, 2025-08-14T21:21:16.8430590Z "digest": "sha256:76a69b57b6837bef07dbc1b481cf28a62dfd7c7063219d9f6e0d0d63067653c7" 2025-08-14T21:21:16.8431565Z }, 2025-08-14T21:21:16.8431889Z { 2025-08-14T21:21:16.8432436Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8433147Z "size": 106, 2025-08-14T21:21:16.8433855Z "digest": "sha256:451d7a75db7137d6ca0ccb4e3635d4a16f3287db4cc2a3a4d3dbc2d3b0266f06" 2025-08-14T21:21:16.8434688Z }, 2025-08-14T21:21:16.8435014Z { 2025-08-14T21:21:16.8435572Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8436788Z "size": 1496, 2025-08-14T21:21:16.8437579Z "digest": "sha256:acf5babd87f23aa905883eb434073e9a00ff41679134f2f4827dd86949f5a9d9" 2025-08-14T21:21:16.8438433Z }, 2025-08-14T21:21:16.8438786Z { 2025-08-14T21:21:16.8439372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8440111Z "size": 453553322, 2025-08-14T21:21:16.8441010Z "digest": "sha256:06e2fbb6cb2eeccd028bfaabb2e674bf25c9d84d3fa91c9e2f03bcf765f92599" 2025-08-14T21:21:16.8441881Z }, 2025-08-14T21:21:16.8442212Z { 2025-08-14T21:21:16.8442782Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8443523Z "size": 163, 2025-08-14T21:21:16.8444248Z "digest": "sha256:87ddced081ff535d58b842d961844bf1b5979ddc8ad7f2d31f606cd6c73ec758" 2025-08-14T21:21:16.8445075Z }, 2025-08-14T21:21:16.8445410Z { 2025-08-14T21:21:16.8445967Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8446687Z "size": 2484, 2025-08-14T21:21:16.8447425Z "digest": "sha256:1eb879bde5e5b9ce59b91eacb794a63e1180a55615094477886cba5016e993d7" 2025-08-14T21:21:16.8448269Z }, 2025-08-14T21:21:16.8448614Z { 2025-08-14T21:21:16.8449164Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8450281Z "size": 8102124021, 2025-08-14T21:21:16.8451039Z "digest": "sha256:c3717fc4dc94ce7fdd93c87843ab97f72e05c0e87203ae15e48b87935b189a27" 2025-08-14T21:21:16.8451907Z }, 2025-08-14T21:21:16.8452245Z { 2025-08-14T21:21:16.8452818Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8453556Z "size": 105, 2025-08-14T21:21:16.8454259Z "digest": "sha256:0c9ed178fd2ebc5a6630c12887a96b3f9884b85144d726435e77d33ab1ee11ab" 2025-08-14T21:21:16.8455077Z }, 2025-08-14T21:21:16.8455698Z { 2025-08-14T21:21:16.8456372Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8457168Z "size": 612, 2025-08-14T21:21:16.8458055Z "digest": "sha256:b4626f6a341ca316ed5525f9c12e667b82f444b8cf673a0ad0bf8b78ccfd5bc9" 2025-08-14T21:21:16.8458982Z }, 2025-08-14T21:21:16.8459310Z { 2025-08-14T21:21:16.8459878Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8460623Z "size": 677677354, 2025-08-14T21:21:16.8461371Z "digest": "sha256:4bf07717a6a2b5fd5a05d41b16dedebb413da5d50ae09457e112d5b42ecacbe1" 2025-08-14T21:21:16.8462218Z }, 2025-08-14T21:21:16.8462539Z { 2025-08-14T21:21:16.8463097Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8463823Z "size": 111, 2025-08-14T21:21:16.8464536Z "digest": "sha256:f258db683e793ef4f283958f771de6376461ec4c2d2ac2947f2ad72a998f45ad" 2025-08-14T21:21:16.8465354Z }, 2025-08-14T21:21:16.8465685Z { 2025-08-14T21:21:16.8466234Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8466954Z "size": 1555, 2025-08-14T21:21:16.8467690Z "digest": "sha256:2d70f6b6782f0d7c3342f35a89c20ca75e63acc6cffff00c2f5b0c4ae16aa20d" 2025-08-14T21:21:16.8468510Z }, 2025-08-14T21:21:16.8468841Z { 2025-08-14T21:21:16.8469375Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8470099Z "size": 107, 2025-08-14T21:21:16.8470817Z "digest": "sha256:cfbe9750bae7f94f985be33742a90ac52adfdbe5870aa09b09ad6ce80db6541b" 2025-08-14T21:21:16.8471659Z }, 2025-08-14T21:21:16.8471982Z { 2025-08-14T21:21:16.8472524Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8473234Z "size": 166, 2025-08-14T21:21:16.8473945Z "digest": "sha256:8d2ff9ef7f4c24e23debf7d02ee44f5acc6e3ed95168f970dfa061e270556036" 2025-08-14T21:21:16.8474777Z }, 2025-08-14T21:21:16.8475098Z { 2025-08-14T21:21:16.8475629Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8476342Z "size": 2838566, 2025-08-14T21:21:16.8477053Z "digest": "sha256:59a73147541eccf1318db0147d0e40952ff983b3515bbc1b2586caa5d370c2e5" 2025-08-14T21:21:16.8478211Z }, 2025-08-14T21:21:16.8478563Z { 2025-08-14T21:21:16.8479124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8479857Z "size": 107, 2025-08-14T21:21:16.8480908Z "digest": "sha256:80e89acaf7bc105eb24090a7638ae51ec2ec270cf4e181e3002f3ef6cea7d2dc" 2025-08-14T21:21:16.8481792Z }, 2025-08-14T21:21:16.8482195Z { 2025-08-14T21:21:16.8482845Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8483672Z "size": 802, 2025-08-14T21:21:16.8484414Z "digest": "sha256:c9c2b424b8e08d943dc259a3796d66eede3a1e93a6460df5db132c0036d3d6af" 2025-08-14T21:21:16.8485241Z }, 2025-08-14T21:21:16.8485568Z { 2025-08-14T21:21:16.8486112Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8486825Z "size": 26113684, 2025-08-14T21:21:16.8487549Z "digest": "sha256:26060b8c841ee0f2c22485c0d51390219230b08ce81ce14c500e40be58488ac5" 2025-08-14T21:21:16.8488356Z }, 2025-08-14T21:21:16.8488692Z { 2025-08-14T21:21:16.8489273Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8489991Z "size": 104, 2025-08-14T21:21:16.8490680Z "digest": "sha256:58b4bcac07b333b658450cb576e2bb832b11cd699d76f1f3846e739deb4a5738" 2025-08-14T21:21:16.8491867Z }, 2025-08-14T21:21:16.8492195Z { 2025-08-14T21:21:16.8492739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8493455Z "size": 425, 2025-08-14T21:21:16.8494160Z "digest": "sha256:18c4ed1ec491095788e352ae018afd84de0f251fbcfb8f74d5d893e1e9ab196d" 2025-08-14T21:21:16.8494987Z }, 2025-08-14T21:21:16.8495314Z { 2025-08-14T21:21:16.8495866Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8496585Z "size": 19308715, 2025-08-14T21:21:16.8497379Z "digest": "sha256:4d7414e539c3ad1ab34e46032ca390dae2d25805222308bb02ecc1b35713b1a2" 2025-08-14T21:21:16.8498330Z }, 2025-08-14T21:21:16.8498700Z { 2025-08-14T21:21:16.8499264Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8499973Z "size": 691, 2025-08-14T21:21:16.8500675Z "digest": "sha256:6738ba83282e002d92bff3d2b4951e3c1a67f5ec2c1bad2fd780c2f5d444748f" 2025-08-14T21:21:16.8501512Z }, 2025-08-14T21:21:16.8501849Z { 2025-08-14T21:21:16.8502393Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8503102Z "size": 724, 2025-08-14T21:21:16.8503787Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:21:16.8504593Z }, 2025-08-14T21:21:16.8504908Z { 2025-08-14T21:21:16.8505462Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8506168Z "size": 116, 2025-08-14T21:21:16.8506859Z "digest": "sha256:dfb0f24886393e1d394f1f433dc9346026679dafd7a60c3a93de17d94078c1ca" 2025-08-14T21:21:16.8507676Z }, 2025-08-14T21:21:16.8507998Z { 2025-08-14T21:21:16.8508553Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8509270Z "size": 136, 2025-08-14T21:21:16.8509978Z "digest": "sha256:dc833b0762f2e144670a660f6b7ce62cec71a5fdd24df4e67b5c6173d5834451" 2025-08-14T21:21:16.8510813Z }, 2025-08-14T21:21:16.8511130Z { 2025-08-14T21:21:16.8511678Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8512387Z "size": 139, 2025-08-14T21:21:16.8513080Z "digest": "sha256:8827df8ca2da347e0032d1bff3b0312437f711c5d0b5f2164f8a60c3368a9827" 2025-08-14T21:21:16.8513897Z }, 2025-08-14T21:21:16.8514222Z { 2025-08-14T21:21:16.8514765Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8515477Z "size": 32, 2025-08-14T21:21:16.8516187Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:21:16.8517011Z }, 2025-08-14T21:21:16.8517343Z { 2025-08-14T21:21:16.8517891Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8518920Z "size": 214, 2025-08-14T21:21:16.8519657Z "digest": "sha256:cd36d0dbfc741c82fee3a71ab009d02c22329a76463482a521b51f4e183248b3" 2025-08-14T21:21:16.8520592Z }, 2025-08-14T21:21:16.8520922Z { 2025-08-14T21:21:16.8521487Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8522316Z "size": 347, 2025-08-14T21:21:16.8523149Z "digest": "sha256:4ba8e7a736c8199931fd7ff9931a5f17b7b931d0383a3e158f1b12b191a1d250" 2025-08-14T21:21:16.8524076Z }, 2025-08-14T21:21:16.8524412Z { 2025-08-14T21:21:16.8524975Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8525683Z "size": 88301, 2025-08-14T21:21:16.8526403Z "digest": "sha256:d3f2353a1f26b127b1869d0eb92a134748c89b45f5defec4fcb2c356bc73c540" 2025-08-14T21:21:16.8527225Z }, 2025-08-14T21:21:16.8527546Z { 2025-08-14T21:21:16.8528093Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8528803Z "size": 106, 2025-08-14T21:21:16.8529515Z "digest": "sha256:22c681dc84cd6e1f55b9874ead0fb9d31d471d5cb641391a3b3dd979c1502cc6" 2025-08-14T21:21:16.8530343Z }, 2025-08-14T21:21:16.8530671Z { 2025-08-14T21:21:16.8531235Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8532292Z "size": 1666, 2025-08-14T21:21:16.8532976Z "digest": "sha256:519f3067ec6a99456e6fb280bd4b79e2478677684831299d601c9493523b67cc" 2025-08-14T21:21:16.8533788Z }, 2025-08-14T21:21:16.8534113Z { 2025-08-14T21:21:16.8534662Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8535371Z "size": 724, 2025-08-14T21:21:16.8536069Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:21:16.8536891Z }, 2025-08-14T21:21:16.8537263Z { 2025-08-14T21:21:16.8537915Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8538759Z "size": 139, 2025-08-14T21:21:16.8539475Z "digest": "sha256:6a8272f38d227d78a8f9b1ddf10bb73941fbd77e71ae3c582c27e08c0f6261c2" 2025-08-14T21:21:16.8540292Z }, 2025-08-14T21:21:16.8540618Z { 2025-08-14T21:21:16.8541171Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8541904Z "size": 120, 2025-08-14T21:21:16.8542587Z "digest": "sha256:973e2484504fec2c1292e24122716507e8ca4717a35c142358174b60c0e401cf" 2025-08-14T21:21:16.8543377Z }, 2025-08-14T21:21:16.8543702Z { 2025-08-14T21:21:16.8544241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8544973Z "size": 5373253071, 2025-08-14T21:21:16.8545727Z "digest": "sha256:ac32a18ead25409bbcd67a8636c32d8af5852a347ea5503c7c9c6234cacf45a1" 2025-08-14T21:21:16.8546554Z }, 2025-08-14T21:21:16.8546870Z { 2025-08-14T21:21:16.8547418Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8548131Z "size": 175, 2025-08-14T21:21:16.8548834Z "digest": "sha256:76846613d7067ca9de6a73db198902ca46b403bc78e2627f3f6fb805e1bbaa0a" 2025-08-14T21:21:16.8549653Z }, 2025-08-14T21:21:16.8549982Z { 2025-08-14T21:21:16.8550527Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8551237Z "size": 1896, 2025-08-14T21:21:16.8551962Z "digest": "sha256:dcf1e01c98d6a6f72674d79a4e8e4047b54796576cd06ad682c225a92820a8f5" 2025-08-14T21:21:16.8552786Z }, 2025-08-14T21:21:16.8553108Z { 2025-08-14T21:21:16.8553648Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8554373Z "size": 197509462, 2025-08-14T21:21:16.8555094Z "digest": "sha256:a88175fa027cfade948a7004100834f68828b13c0f21b314491c37084fbbe7e5" 2025-08-14T21:21:16.8555904Z }, 2025-08-14T21:21:16.8556231Z { 2025-08-14T21:21:16.8556777Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8557494Z "size": 303, 2025-08-14T21:21:16.8558184Z "digest": "sha256:7f5277f691672469f431fd90a8c2bb702c6c68333f6be2cff868f00e416c5a1a" 2025-08-14T21:21:16.8559310Z }, 2025-08-14T21:21:16.8559646Z { 2025-08-14T21:21:16.8560268Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8560999Z "size": 32, 2025-08-14T21:21:16.8561732Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:21:16.8562735Z }, 2025-08-14T21:21:16.8563132Z { 2025-08-14T21:21:16.8563780Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8564512Z "size": 108, 2025-08-14T21:21:16.8565217Z "digest": "sha256:d0b078cebcc48c1f46a5d1286ae68a90c295cb6b81f033ba9a2f9c2498d6d7e5" 2025-08-14T21:21:16.8566038Z }, 2025-08-14T21:21:16.8566368Z { 2025-08-14T21:21:16.8566918Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:21:16.8567620Z "size": 54145699, 2025-08-14T21:21:16.8568349Z "digest": "sha256:b01371e6ad0895119adb1d1dbfa20844d061be6aecadf9165128cebb28b9fd8e" 2025-08-14T21:21:16.8569175Z } 2025-08-14T21:21:16.8569518Z ] 2025-08-14T21:21:16.8569857Z } 2025-08-14T21:21:16.8570228Z + exit 0 2025-08-14T21:21:16.8621257Z ##[group]Run set -eux 2025-08-14T21:21:16.8621954Z set -eux 2025-08-14T21:21:16.8623744Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-08-14T21:21:16.8672241Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:16.8673144Z env: 2025-08-14T21:21:16.8673697Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:16.8674757Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:16.8675996Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:16.8677104Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:16.8704311Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:16.8705846Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:16.8706320Z AWS_REGION: us-east-1 2025-08-14T21:21:16.8706923Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:16.8707538Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:16.8717159Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:16.8717589Z ##[endgroup] 2025-08-14T21:21:16.8809220Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-08-14T21:21:16.8811957Z + jq --raw-output .SecretString 2025-08-14T21:21:16.8816220Z + jq -r .docker_hub_readonly_token 2025-08-14T21:21:16.8819909Z + docker login --username pytorchbot --password-stdin 2025-08-14T21:21:17.6545962Z 2025-08-14T21:21:17.6549114Z An error occurred (AccessDeniedException) when calling the GetSecretValue operation: User: arn:aws:sts::308535385114:assumed-role/gha_workflow_s3_and_ecr_read_only/GitHubActions is not authorized to perform: secretsmanager:GetSecretValue on resource: docker_hub_readonly_token because no identity-based policy allows the secretsmanager:GetSecretValue action 2025-08-14T21:21:17.7451307Z Error: Cannot perform an interactive login from a non TTY device 2025-08-14T21:21:17.7525114Z ##[error]Process completed with exit code 1. 2025-08-14T21:21:17.7722966Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-08-14T21:21:17.7723697Z with: 2025-08-14T21:21:17.7724996Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:17.7726667Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:17.7727386Z env: 2025-08-14T21:21:17.7727793Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:17.7728602Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:17.7729857Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:17.7730973Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:17.7732664Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:17.7734150Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:17.7734597Z AWS_REGION: us-east-1 2025-08-14T21:21:17.7735135Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:17.7735734Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:17.7745856Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:17.7746277Z ##[endgroup] 2025-08-14T21:21:17.7773616Z ##[group]Run set -x 2025-08-14T21:21:17.7774061Z set -x 2025-08-14T21:21:17.7774432Z set +e 2025-08-14T21:21:17.7774760Z  2025-08-14T21:21:17.7775098Z login() { 2025-08-14T21:21:17.7775858Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:21:17.7776758Z } 2025-08-14T21:21:17.7777153Z  2025-08-14T21:21:17.7777534Z retry () { 2025-08-14T21:21:17.7778033Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:21:17.7779130Z } 2025-08-14T21:21:17.7779522Z  2025-08-14T21:21:17.7779920Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:21:17.7780406Z  2025-08-14T21:21:17.7781175Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-08-14T21:21:17.7782222Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-08-14T21:21:17.7782800Z  2025-08-14T21:21:17.7783119Z set -e 2025-08-14T21:21:17.7783650Z # ignore output since only exit code is used for conditional 2025-08-14T21:21:17.7784406Z # only pull docker image if it's not available locally 2025-08-14T21:21:17.7785249Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-08-14T21:21:17.7786038Z  retry docker pull "${DOCKER_IMAGE}" 2025-08-14T21:21:17.7786530Z fi 2025-08-14T21:21:17.7834611Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:17.7835231Z env: 2025-08-14T21:21:17.7835591Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:17.7836307Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:17.7837343Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:17.7838316Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:17.7839961Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:17.7841496Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:17.7841952Z AWS_REGION: us-east-1 2025-08-14T21:21:17.7842473Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:17.7843081Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:17.7853857Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:17.7855455Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:17.7856993Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:17.7857737Z ##[endgroup] 2025-08-14T21:21:17.7939863Z + set +e 2025-08-14T21:21:17.7941061Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:17.7941811Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:17.7949910Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:21:17.7953009Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:21:19.0228464Z WARNING! Your password will be stored unencrypted in /var/home/pytorchci/.docker/config.json. 2025-08-14T21:21:19.0229653Z Configure a credential helper to remove this warning. See 2025-08-14T21:21:19.0230696Z https://docs.docker.com/engine/reference/commandline/login/#credential-stores 2025-08-14T21:21:19.0231400Z 2025-08-14T21:21:19.0232363Z Login Succeeded 2025-08-14T21:21:19.0304715Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:19.0306433Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-08-14T21:21:19.9417594Z + IMAGE_SIZE=17743.216945648193 2025-08-14T21:21:19.9418339Z + echo 'Compressed size of image in MB: 17743.216945648193' 2025-08-14T21:21:19.9419116Z Compressed size of image in MB: 17743.216945648193 2025-08-14T21:21:19.9419752Z + set -e 2025-08-14T21:21:19.9421183Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:21:19.9891524Z Prepare all required actions 2025-08-14T21:21:19.9951839Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-08-14T21:21:19.9952462Z with: 2025-08-14T21:21:19.9953301Z github-token: *** 2025-08-14T21:21:19.9953738Z env: 2025-08-14T21:21:19.9954587Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:19.9955375Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:19.9956511Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:19.9957573Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:19.9959368Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:19.9961213Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:19.9961748Z AWS_REGION: us-east-1 2025-08-14T21:21:19.9962586Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:19.9963412Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:19.9973969Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:19.9974427Z ##[endgroup] 2025-08-14T21:21:20.0004117Z ##[group]Run set -eux 2025-08-14T21:21:20.0004596Z set -eux 2025-08-14T21:21:20.0005398Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-14T21:21:20.0054126Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:20.0054742Z env: 2025-08-14T21:21:20.0055100Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:20.0055789Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:20.0056872Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:20.0057981Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:20.0059604Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:20.0061002Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:20.0061460Z AWS_REGION: us-east-1 2025-08-14T21:21:20.0061970Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:20.0062763Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:20.0072448Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:20.0073049Z GITHUB_TOKEN: *** 2025-08-14T21:21:20.0073436Z ##[endgroup] 2025-08-14T21:21:20.0153692Z + python3 .github/scripts/get_workflow_job_id.py 16976370303 gpu9498 2025-08-14T21:21:20.9166541Z Setting output job-id=48128182520 2025-08-14T21:21:20.9167900Z Setting output job-name=linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:21:20.9431434Z Prepare all required actions 2025-08-14T21:21:20.9432180Z Getting action download info 2025-08-14T21:21:21.1664598Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-08-14T21:21:22.0586990Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-08-14T21:21:22.9602213Z ##[group]Run ./.github/actions/download-build-artifacts 2025-08-14T21:21:22.9602562Z with: 2025-08-14T21:21:22.9602819Z name: linux-jammy-rocm-py3.10 2025-08-14T21:21:22.9603154Z s3-bucket: gha-artifacts 2025-08-14T21:21:22.9603447Z env: 2025-08-14T21:21:22.9603687Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:22.9604118Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:22.9604672Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:22.9605223Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:22.9606098Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:22.9606894Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:22.9607155Z AWS_REGION: us-east-1 2025-08-14T21:21:22.9607455Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:22.9607795Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:22.9619571Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:22.9619820Z ##[endgroup] 2025-08-14T21:21:22.9666366Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:21:22.9666629Z with: 2025-08-14T21:21:22.9666813Z name: linux-jammy-rocm-py3.10 2025-08-14T21:21:22.9667035Z s3-bucket: gha-artifacts 2025-08-14T21:21:22.9667249Z region: us-east-1 2025-08-14T21:21:22.9667423Z env: 2025-08-14T21:21:22.9667580Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:22.9667904Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:22.9668385Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:22.9668825Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:22.9669566Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:22.9670231Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:22.9670442Z AWS_REGION: us-east-1 2025-08-14T21:21:22.9670676Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:22.9670955Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:22.9675281Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:22.9675473Z ##[endgroup] 2025-08-14T21:21:23.4154130Z (node:2818609) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:21:23.4155076Z 2025-08-14T21:21:23.4155522Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:21:23.4156496Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:21:23.4157505Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:21:23.6016039Z Found 1 objects with prefix pytorch/pytorch/16976370303/linux-jammy-rocm-py3.10/ 2025-08-14T21:21:23.6017435Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:21:44.8214320Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:21:44.8229840Z Artifact download has finished successfully 2025-08-14T21:21:44.8722446Z ##[group]Run unzip -o artifacts.zip 2025-08-14T21:21:44.8723086Z unzip -o artifacts.zip 2025-08-14T21:21:44.8777511Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:44.8778204Z env: 2025-08-14T21:21:44.8778646Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:44.8780199Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:44.8781636Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:44.8782781Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:44.8784592Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:44.8786231Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:44.8786741Z AWS_REGION: us-east-1 2025-08-14T21:21:44.8787315Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:44.8787997Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:44.8798767Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:44.8799224Z ##[endgroup] 2025-08-14T21:21:44.8926009Z Archive: artifacts.zip 2025-08-14T21:21:44.8927644Z creating: dist/ 2025-08-14T21:21:50.2825449Z inflating: dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl 2025-08-14T21:21:50.2953453Z inflating: dist/.ninja_log 2025-08-14T21:21:50.2959135Z creating: build/custom_test_artifacts/ 2025-08-14T21:21:50.2959996Z creating: build/custom_test_artifacts/custom-op-build/ 2025-08-14T21:21:50.2961182Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-08-14T21:21:50.2962286Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:21:50.2963626Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:21:50.2965853Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/ 2025-08-14T21:21:50.2967028Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:21:50.2968315Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:21:50.2969570Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:21:50.2971019Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:21:50.2972529Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:21:50.2973884Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:21:50.2975201Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:21:50.2976463Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:21:50.2977930Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:21:50.2979427Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:21:50.2980799Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:21:50.2982328Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:21:50.2983922Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:21:50.2985322Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:21:50.2986404Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/ 2025-08-14T21:21:50.2987556Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-08-14T21:21:50.2988756Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-08-14T21:21:50.2990074Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-08-14T21:21:50.2991588Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-08-14T21:21:50.2993483Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-08-14T21:21:50.2994871Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-08-14T21:21:50.2996260Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-08-14T21:21:50.2997668Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-08-14T21:21:50.2999075Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-08-14T21:21:50.3000601Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-08-14T21:21:50.3001976Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-08-14T21:21:50.3003348Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-08-14T21:21:50.3180494Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-08-14T21:21:50.3181934Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-08-14T21:21:50.3183417Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-08-14T21:21:50.3185089Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-08-14T21:21:50.3187359Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-08-14T21:21:50.3188804Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-08-14T21:21:50.3190305Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-08-14T21:21:50.3191810Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-08-14T21:21:50.3193299Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-08-14T21:21:50.3194770Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-08-14T21:21:50.3196234Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-08-14T21:21:50.3205522Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-08-14T21:21:50.3280591Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-08-14T21:21:50.3282390Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:21:50.3284192Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:21:50.3285588Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-08-14T21:21:50.3286778Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-08-14T21:21:50.3287934Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-08-14T21:21:50.3289142Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json 2025-08-14T21:21:50.3290361Z inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_outer_vec.cc 2025-08-14T21:21:50.3291510Z inflating: build/custom_test_artifacts/custom-op-build/hipblaslt_test_vec_ext.cc 2025-08-14T21:21:50.3292569Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-08-14T21:21:50.3293539Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-08-14T21:21:50.3294624Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-08-14T21:21:50.3438465Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-08-14T21:21:50.3490312Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-08-14T21:21:50.3491551Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-08-14T21:21:50.3492438Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-08-14T21:21:50.3493518Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:21:50.3494762Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:21:50.3495946Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/ 2025-08-14T21:21:50.3497092Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:21:50.3498337Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:21:50.3499541Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:21:50.3500932Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:21:50.3502356Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:21:50.3503656Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:21:50.3504909Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:21:50.3506142Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:21:50.3507959Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:21:50.3509522Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:21:50.3510880Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:21:50.3512395Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:21:50.3513956Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:21:50.3515304Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:21:50.3516366Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/ 2025-08-14T21:21:50.3517475Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-08-14T21:21:50.3518663Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-08-14T21:21:50.3520002Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-08-14T21:21:50.3521930Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-08-14T21:21:50.3523423Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-08-14T21:21:50.3524808Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-08-14T21:21:50.3526214Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-08-14T21:21:50.3527643Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-08-14T21:21:50.3529059Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-08-14T21:21:50.3530476Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-08-14T21:21:50.3531859Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-08-14T21:21:50.3533669Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-08-14T21:21:50.3594419Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-08-14T21:21:50.3596131Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:21:50.3597524Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:21:50.3598767Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-08-14T21:21:50.3599885Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-08-14T21:21:50.3601320Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-08-14T21:21:50.3602502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json 2025-08-14T21:21:50.3603715Z inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_outer_vec.cc 2025-08-14T21:21:50.3604883Z inflating: build/custom_test_artifacts/jit-hook-build/hipblaslt_test_vec_ext.cc 2025-08-14T21:21:50.3605961Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-08-14T21:21:50.3606928Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-08-14T21:21:50.3607922Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-08-14T21:21:50.3636611Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-08-14T21:21:50.3637651Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-08-14T21:21:50.3638996Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-08-14T21:21:50.3640105Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:21:50.3642270Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:21:50.3643011Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/ 2025-08-14T21:21:50.3643719Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:21:50.3644432Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:21:50.3645117Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:21:50.3645891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:21:50.3646674Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:21:50.3647431Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:21:50.3648140Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:21:50.3648822Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:21:50.3649929Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:21:50.3651150Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:21:50.3651965Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:21:50.3653325Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:21:50.3654680Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:21:50.3655465Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:21:50.3656060Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/ 2025-08-14T21:21:50.3656680Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-08-14T21:21:50.3657665Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-08-14T21:21:50.3658455Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-08-14T21:21:50.3659320Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-08-14T21:21:50.3660138Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-08-14T21:21:50.3660899Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-08-14T21:21:50.3661692Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-08-14T21:21:50.3662493Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-08-14T21:21:50.3663290Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-08-14T21:21:50.3664083Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-08-14T21:21:50.3664874Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-08-14T21:21:50.3665723Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-08-14T21:21:50.3777020Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-08-14T21:21:50.3778127Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-08-14T21:21:50.3778923Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-08-14T21:21:50.3779822Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-08-14T21:21:50.3780712Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-08-14T21:21:50.3781520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-08-14T21:21:50.3782344Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-08-14T21:21:50.3783172Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-08-14T21:21:50.3784012Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-08-14T21:21:50.3784836Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-08-14T21:21:50.3785652Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-08-14T21:21:50.3802283Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-08-14T21:21:50.3853261Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-08-14T21:21:50.3854205Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:21:50.3855001Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:21:50.3855729Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-08-14T21:21:50.3856383Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-08-14T21:21:50.3857009Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-08-14T21:21:50.3857676Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json 2025-08-14T21:21:50.3858584Z inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_outer_vec.cc 2025-08-14T21:21:50.3859273Z inflating: build/custom_test_artifacts/custom-backend-build/hipblaslt_test_vec_ext.cc 2025-08-14T21:21:50.3859891Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-08-14T21:21:50.3860446Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-08-14T21:21:50.3861009Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-08-14T21:21:50.3954079Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-08-14T21:21:50.3991564Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-08-14T21:21:50.3992572Z creating: build/lib/ 2025-08-14T21:21:50.4066699Z inflating: build/lib/libprotobuf-lite.a 2025-08-14T21:21:50.4472303Z inflating: build/lib/libprotobuf.a 2025-08-14T21:21:50.4924737Z inflating: build/lib/libprotoc.a 2025-08-14T21:21:50.4933294Z inflating: build/lib/libpthreadpool.a 2025-08-14T21:21:50.4940690Z inflating: build/lib/libcpuinfo.a 2025-08-14T21:21:50.4947862Z inflating: build/lib/libcpuinfo_internals.a 2025-08-14T21:21:50.4948621Z inflating: build/lib/libclog.a 2025-08-14T21:21:50.4965805Z inflating: build/lib/libpytorch_qnnpack.a 2025-08-14T21:21:50.4966919Z inflating: build/lib/libnnpack_reference_layers.a 2025-08-14T21:21:50.4984082Z inflating: build/lib/libnnpack.a 2025-08-14T21:21:50.5153312Z inflating: build/lib/libmicrokernels-prod.a 2025-08-14T21:21:50.5948314Z inflating: build/lib/libmicrokernels-all.a 2025-08-14T21:21:50.6012161Z inflating: build/lib/libgtest.a 2025-08-14T21:21:50.6027733Z inflating: build/lib/libgmock.a 2025-08-14T21:21:50.6028446Z inflating: build/lib/libgmock_main.a 2025-08-14T21:21:50.6029077Z inflating: build/lib/libgtest_main.a 2025-08-14T21:21:50.6111383Z inflating: build/lib/libXNNPACK.a 2025-08-14T21:21:50.6180029Z inflating: build/lib/libbenchmark.a 2025-08-14T21:21:50.6180797Z inflating: build/lib/libbenchmark_main.a 2025-08-14T21:21:50.6187796Z inflating: build/lib/libittnotify.a 2025-08-14T21:21:50.6188530Z inflating: build/lib/libjitprofiling.a 2025-08-14T21:21:50.6247594Z inflating: build/lib/libasmjit.a 2025-08-14T21:21:50.7351368Z inflating: build/lib/libfbgemm.a 2025-08-14T21:21:50.7378696Z inflating: build/lib/libtensorpipe_uv.a 2025-08-14T21:21:50.7889211Z inflating: build/lib/libtensorpipe.a 2025-08-14T21:21:50.7999033Z inflating: build/lib/libgloo.a 2025-08-14T21:21:50.8042487Z inflating: build/lib/libonnx_proto.a 2025-08-14T21:21:50.8444872Z inflating: build/lib/libgloo_hip.a 2025-08-14T21:21:50.9103831Z inflating: build/lib/libonnx.a 2025-08-14T21:21:51.8592247Z inflating: build/lib/libdnnl.a 2025-08-14T21:21:51.8608856Z inflating: build/lib/libfmt.a 2025-08-14T21:21:51.8872490Z inflating: build/lib/libkineto.a 2025-08-14T21:21:51.8971499Z inflating: build/lib/libc10.so 2025-08-14T21:21:51.8972258Z inflating: build/lib/libtorch_global_deps.so 2025-08-14T21:21:51.8973181Z inflating: build/lib/libcaffe2_nvrtc.so 2025-08-14T21:21:51.9008685Z inflating: build/lib/libc10_hip.so 2025-08-14T21:21:54.6334192Z inflating: build/lib/libtorch_cpu.so 2025-08-14T21:21:54.6336679Z inflating: build/lib/libshm.so 2025-08-14T21:21:55.4366908Z inflating: build/lib/libtorch_hip.so 2025-08-14T21:21:55.4367319Z inflating: build/lib/libtorch.so 2025-08-14T21:21:55.4384281Z inflating: build/lib/libjitbackend_test.so 2025-08-14T21:21:55.4449731Z inflating: build/lib/libtorchbind_test.so 2025-08-14T21:21:55.4471117Z inflating: build/lib/libbackend_with_compiler.so 2025-08-14T21:21:55.4495487Z inflating: build/lib/libaoti_custom_ops.so 2025-08-14T21:21:55.6356145Z inflating: build/lib/libtorch_python.so 2025-08-14T21:21:55.6386317Z inflating: build/lib/libnnapi_backend.so 2025-08-14T21:21:55.6387022Z creating: build/bin/ 2025-08-14T21:21:55.6388207Z creating: build/bin/CMakeFiles/ 2025-08-14T21:21:55.6388852Z inflating: build/bin/cmake_install.cmake 2025-08-14T21:21:55.6389508Z inflating: build/bin/CTestTestfile.cmake 2025-08-14T21:21:55.6796323Z inflating: build/bin/protoc-3.13.0.0 2025-08-14T21:21:55.7205374Z inflating: build/bin/protoc 2025-08-14T21:21:55.7258027Z inflating: build/bin/c10_AllocatorConfig_test 2025-08-14T21:21:55.7307829Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-08-14T21:21:55.7358677Z inflating: build/bin/c10_DeviceGuard_test 2025-08-14T21:21:55.7410149Z inflating: build/bin/c10_Device_test 2025-08-14T21:21:55.7468533Z inflating: build/bin/c10_DispatchKeySet_test 2025-08-14T21:21:55.7522098Z inflating: build/bin/c10_Scalar_test 2025-08-14T21:21:55.7570709Z inflating: build/bin/c10_StreamGuard_test 2025-08-14T21:21:55.7620995Z inflating: build/bin/c10_SymInt_test 2025-08-14T21:21:55.7675361Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-08-14T21:21:55.7730483Z inflating: build/bin/c10_InlineStreamGuard_test 2025-08-14T21:21:55.7785774Z inflating: build/bin/c10_SizesAndStrides_test 2025-08-14T21:21:55.7854490Z inflating: build/bin/c10_cow_test 2025-08-14T21:21:55.7903047Z inflating: build/bin/c10_ConstexprCrc_test 2025-08-14T21:21:55.7952404Z inflating: build/bin/c10_ArrayRef_test 2025-08-14T21:21:55.8004817Z inflating: build/bin/c10_Bitset_test 2025-08-14T21:21:55.8054414Z inflating: build/bin/c10_DeadlockDetection_test 2025-08-14T21:21:55.8111567Z inflating: build/bin/c10_Enumerate_test 2025-08-14T21:21:55.8162127Z inflating: build/bin/c10_Half_test 2025-08-14T21:21:55.8214415Z inflating: build/bin/c10_IntrusiveList_test 2025-08-14T21:21:55.8269857Z inflating: build/bin/c10_LeftRight_test 2025-08-14T21:21:55.8324865Z inflating: build/bin/c10_Metaprogramming_test 2025-08-14T21:21:55.8377858Z inflating: build/bin/c10_NetworkFlow_test 2025-08-14T21:21:55.8427100Z inflating: build/bin/c10_Semaphore_test 2025-08-14T21:21:55.8482282Z inflating: build/bin/c10_ThreadLocal_test 2025-08-14T21:21:55.8531843Z inflating: build/bin/c10_Synchronized_test 2025-08-14T21:21:55.8583559Z inflating: build/bin/c10_TypeIndex_test 2025-08-14T21:21:55.8634023Z inflating: build/bin/c10_TypeList_test 2025-08-14T21:21:55.8682980Z inflating: build/bin/c10_TypeTraits_test 2025-08-14T21:21:55.8734278Z inflating: build/bin/c10_accumulate_test 2025-08-14T21:21:55.8789462Z inflating: build/bin/c10_bfloat16_test 2025-08-14T21:21:55.8845605Z inflating: build/bin/c10_complex_math_test 2025-08-14T21:21:55.8895665Z inflating: build/bin/c10_bit_cast_test 2025-08-14T21:21:55.8950332Z inflating: build/bin/c10_complex_test 2025-08-14T21:21:55.8999363Z inflating: build/bin/c10_error_test 2025-08-14T21:21:55.9049576Z inflating: build/bin/c10_flags_test 2025-08-14T21:21:55.9101442Z inflating: build/bin/c10_exception_test 2025-08-14T21:21:55.9152110Z inflating: build/bin/c10_irange_test 2025-08-14T21:21:55.9202130Z inflating: build/bin/c10_generic_math_test 2025-08-14T21:21:55.9361341Z inflating: build/bin/c10_intrusive_ptr_test 2025-08-14T21:21:55.9414046Z inflating: build/bin/c10_lazy_test 2025-08-14T21:21:55.9470627Z inflating: build/bin/c10_logging_test 2025-08-14T21:21:55.9543874Z inflating: build/bin/c10_optional_test 2025-08-14T21:21:55.9596819Z inflating: build/bin/c10_registry_test 2025-08-14T21:21:55.9657529Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-08-14T21:21:55.9708793Z inflating: build/bin/c10_ssize_test 2025-08-14T21:21:55.9858634Z inflating: build/bin/c10_small_vector_test 2025-08-14T21:21:55.9914345Z inflating: build/bin/c10_string_util_test 2025-08-14T21:21:55.9964337Z inflating: build/bin/c10_tempfile_test 2025-08-14T21:21:56.0013088Z inflating: build/bin/c10_string_view_test 2025-08-14T21:21:56.0069084Z inflating: build/bin/c10_typeid_test 2025-08-14T21:21:56.0112624Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-08-14T21:21:56.0161358Z inflating: build/bin/c10_hip_HIPAssertionsTest_1_var_test 2025-08-14T21:21:56.0209893Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_stream 2025-08-14T21:21:56.0258936Z inflating: build/bin/c10_hip_HIPAssertionsTest_catches_thread_and_block_and_device 2025-08-14T21:21:56.0307412Z inflating: build/bin/c10_hip_HIPAssertionsTest_from_2_processes 2025-08-14T21:21:56.0356522Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_blocks_and_threads 2025-08-14T21:21:56.0405186Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_multiple_blocks 2025-08-14T21:21:56.0454231Z inflating: build/bin/c10_hip_HIPAssertionsTest_multiple_writes_from_same_block 2025-08-14T21:21:56.0503244Z inflating: build/bin/c10_hip_HIPTest 2025-08-14T21:21:56.1059397Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-08-14T21:21:56.1631628Z inflating: build/bin/vec_test_all_types_AVX512 2025-08-14T21:21:56.2209064Z inflating: build/bin/vec_test_all_types_AVX2 2025-08-14T21:21:56.2261020Z inflating: build/bin/BackoffTest 2025-08-14T21:21:56.2313866Z inflating: build/bin/FileStoreTest 2025-08-14T21:21:56.2369982Z inflating: build/bin/TCPStoreTest 2025-08-14T21:21:56.2423456Z inflating: build/bin/HashStoreTest 2025-08-14T21:21:56.2488567Z inflating: build/bin/ProcessGroupGlooTest 2025-08-14T21:21:56.2490269Z inflating: build/bin/example_allreduce 2025-08-14T21:21:56.2494741Z inflating: build/bin/torch_shm_manager 2025-08-14T21:21:56.2547618Z inflating: build/bin/static_runtime_bench 2025-08-14T21:21:56.2786056Z inflating: build/bin/static_runtime_test 2025-08-14T21:21:56.2858621Z inflating: build/bin/Dict_test 2025-08-14T21:21:56.2910707Z inflating: build/bin/Dimname_test 2025-08-14T21:21:56.2975207Z inflating: build/bin/MaybeOwned_test 2025-08-14T21:21:56.3031312Z inflating: build/bin/NamedTensor_test 2025-08-14T21:21:56.3089218Z inflating: build/bin/apply_utils_test 2025-08-14T21:21:56.3147168Z inflating: build/bin/atest 2025-08-14T21:21:56.3210133Z inflating: build/bin/basic 2025-08-14T21:21:56.3264642Z inflating: build/bin/broadcast_test 2025-08-14T21:21:56.3315257Z inflating: build/bin/cpu_allocator_test 2025-08-14T21:21:56.3372506Z inflating: build/bin/cpu_generator_test 2025-08-14T21:21:56.3425059Z inflating: build/bin/cpu_profiling_allocator_test 2025-08-14T21:21:56.3514515Z inflating: build/bin/cpu_rng_test 2025-08-14T21:21:56.3565045Z inflating: build/bin/dlconvertor_test 2025-08-14T21:21:56.3621858Z inflating: build/bin/extension_backend_test 2025-08-14T21:21:56.3677128Z inflating: build/bin/half_test 2025-08-14T21:21:56.3769952Z inflating: build/bin/ivalue_test 2025-08-14T21:21:56.3819411Z inflating: build/bin/lazy_tensor_test 2025-08-14T21:21:56.3872455Z inflating: build/bin/math_kernel_test 2025-08-14T21:21:56.3925400Z inflating: build/bin/memory_format_test 2025-08-14T21:21:56.3978446Z inflating: build/bin/memory_overlapping_test 2025-08-14T21:21:56.4031122Z inflating: build/bin/mobile_memory_cleanup 2025-08-14T21:21:56.4086419Z inflating: build/bin/native_test 2025-08-14T21:21:56.4137435Z inflating: build/bin/operator_name_test 2025-08-14T21:21:56.4187871Z inflating: build/bin/operators_test 2025-08-14T21:21:56.4240011Z inflating: build/bin/packedtensoraccessor_test 2025-08-14T21:21:56.4296676Z inflating: build/bin/quantized_test 2025-08-14T21:21:56.4362805Z inflating: build/bin/pow_test 2025-08-14T21:21:56.4412590Z inflating: build/bin/reduce_ops_test 2025-08-14T21:21:56.4463366Z inflating: build/bin/reportMemoryUsage_test 2025-08-14T21:21:56.4519159Z inflating: build/bin/scalar_tensor_test 2025-08-14T21:21:56.4570226Z inflating: build/bin/StorageUtils_test 2025-08-14T21:21:56.4628101Z inflating: build/bin/scalar_test 2025-08-14T21:21:56.4679983Z inflating: build/bin/stride_properties_test 2025-08-14T21:21:56.4757871Z inflating: build/bin/tensor_iterator_test 2025-08-14T21:21:56.4812169Z inflating: build/bin/type_ptr_test 2025-08-14T21:21:56.4865856Z inflating: build/bin/test_parallel 2025-08-14T21:21:56.4916910Z inflating: build/bin/thread_init_test 2025-08-14T21:21:56.4975563Z inflating: build/bin/type_test 2025-08-14T21:21:56.5027919Z inflating: build/bin/undefined_tensor_test 2025-08-14T21:21:56.5076969Z inflating: build/bin/verify_api_visibility 2025-08-14T21:21:56.5128086Z inflating: build/bin/weakref_test 2025-08-14T21:21:56.5196289Z inflating: build/bin/legacy_vmap_test 2025-08-14T21:21:56.5247667Z inflating: build/bin/wrapdim_test 2025-08-14T21:21:56.5298789Z inflating: build/bin/xla_tensor_test 2025-08-14T21:21:56.5357537Z inflating: build/bin/IListRef_test 2025-08-14T21:21:56.5460757Z inflating: build/bin/List_test 2025-08-14T21:21:56.5576137Z inflating: build/bin/kernel_function_legacy_test 2025-08-14T21:21:56.5668645Z inflating: build/bin/kernel_function_test 2025-08-14T21:21:56.5789870Z inflating: build/bin/kernel_lambda_legacy_test 2025-08-14T21:21:56.5854764Z inflating: build/bin/KernelFunction_test 2025-08-14T21:21:56.5953748Z inflating: build/bin/kernel_lambda_test 2025-08-14T21:21:56.6013059Z inflating: build/bin/kernel_stackbased_test 2025-08-14T21:21:56.6105915Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-08-14T21:21:56.6156395Z inflating: build/bin/CppSignature_test 2025-08-14T21:21:56.6205488Z inflating: build/bin/op_allowlist_test 2025-08-14T21:21:56.6270829Z inflating: build/bin/inline_container_test 2025-08-14T21:21:56.6568177Z inflating: build/bin/op_registration_test 2025-08-14T21:21:56.6622596Z inflating: build/bin/backend_fallback_test 2025-08-14T21:21:56.6671438Z inflating: build/bin/hip_complex_math_test 2025-08-14T21:21:56.6723926Z inflating: build/bin/hip_apply_test 2025-08-14T21:21:56.6772839Z inflating: build/bin/hip_complex_test 2025-08-14T21:21:56.6821526Z inflating: build/bin/hip_distributions_test 2025-08-14T21:21:56.6870324Z inflating: build/bin/hip_generator_test 2025-08-14T21:21:56.6919335Z inflating: build/bin/hip_half_test 2025-08-14T21:21:56.6968401Z inflating: build/bin/hip_integer_divider_test 2025-08-14T21:21:56.7016898Z inflating: build/bin/hip_optional_test 2025-08-14T21:21:56.7065841Z inflating: build/bin/hip_packedtensoraccessor_test 2025-08-14T21:21:56.7114279Z inflating: build/bin/hip_vectorized_test 2025-08-14T21:21:56.7165899Z inflating: build/bin/hip_dlconvertor_test 2025-08-14T21:21:56.8184750Z inflating: build/bin/test_jit 2025-08-14T21:21:56.8479061Z inflating: build/bin/test_nativert 2025-08-14T21:21:56.8533546Z inflating: build/bin/test_dist_autograd 2025-08-14T21:21:56.8599804Z inflating: build/bin/test_cpp_rpc 2025-08-14T21:21:56.9672713Z inflating: build/bin/test_api 2025-08-14T21:21:56.9673931Z inflating: build/bin/parallel_benchmark 2025-08-14T21:21:57.0002964Z inflating: build/bin/test_lazy 2025-08-14T21:21:57.0003653Z creating: .additional_ci_files/ 2025-08-14T21:21:57.0075413Z inflating: .additional_ci_files/test-times.json 2025-08-14T21:21:57.0349022Z inflating: .additional_ci_files/test-class-times.json 2025-08-14T21:21:57.0405137Z ##[group]Run rm artifacts.zip 2025-08-14T21:21:57.0405744Z rm artifacts.zip 2025-08-14T21:21:57.0462005Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:57.0462781Z env: 2025-08-14T21:21:57.0463215Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:57.0464051Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:57.0465256Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:57.0466403Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:57.0470017Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:57.0471839Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:57.0472386Z AWS_REGION: us-east-1 2025-08-14T21:21:57.0472993Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:57.0473709Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:57.0485473Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:57.0485963Z ##[endgroup] 2025-08-14T21:21:57.4151086Z ##[group]Run df -H 2025-08-14T21:21:57.4151558Z df -H 2025-08-14T21:21:57.4201960Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:57.4202703Z env: 2025-08-14T21:21:57.4203122Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:57.4203968Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:57.4204977Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:57.4205968Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:57.4207553Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:57.4208950Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:57.4209408Z AWS_REGION: us-east-1 2025-08-14T21:21:57.4209910Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:57.4210511Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:57.4221534Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:57.4221956Z ##[endgroup] 2025-08-14T21:21:57.4376772Z Filesystem Size Used Avail Use% Mounted on 2025-08-14T21:21:57.4377688Z tmpfs 109G 33M 109G 1% /run 2025-08-14T21:21:57.4378518Z /dev/nvme0n1p2 944G 373G 524G 42% / 2025-08-14T21:21:57.4379322Z tmpfs 542G 33k 542G 1% /dev/shm 2025-08-14T21:21:57.4380109Z tmpfs 5.3M 0 5.3M 0% /run/lock 2025-08-14T21:21:57.4380864Z /dev/nvme0n1p1 536M 6.4M 530M 2% /boot/efi 2025-08-14T21:21:57.4381674Z /dev/nvme1n1p1 3.8T 1.3T 2.4T 35% /media/4TB 2025-08-14T21:21:57.4382466Z tmpfs 109G 33k 109G 1% /run/user/1307800118 2025-08-14T21:21:57.4383293Z 172.18.148.8:/export/amd2 5.5T 278G 5.3T 6% /mnt 2025-08-14T21:21:57.4384370Z pure1.jax.cs.cpe.ice.amd.com:/homes/amd-pytorch 108G 348M 108G 1% /home/amd-pytorch 2025-08-14T21:21:57.4385813Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage 165T 137T 29T 83% /groups 2025-08-14T21:21:57.4386873Z pure2.jax.cs.cpe.ice.amd.com:/GroupStorage/Scratch 5.5T 1.8T 3.8T 33% /scratch 2025-08-14T21:21:57.4387926Z pure1.jax.cs.cpe.ice.amd.com:/homes/jlay 108G 19G 89G 18% /home/jlay 2025-08-14T21:21:57.4453203Z Prepare all required actions 2025-08-14T21:21:57.4453988Z Getting action download info 2025-08-14T21:21:57.6593079Z ##[group]Run ./.github/actions/download-td-artifacts 2025-08-14T21:21:57.6593734Z with: 2025-08-14T21:21:57.6594093Z env: 2025-08-14T21:21:57.6594470Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:57.6595235Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:57.6596400Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:57.6597460Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:57.6599297Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:57.6601056Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:57.6601571Z AWS_REGION: us-east-1 2025-08-14T21:21:57.6602214Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:57.6602894Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:57.6613469Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:57.6613920Z ##[endgroup] 2025-08-14T21:21:57.6669727Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:21:57.6670334Z with: 2025-08-14T21:21:57.6670702Z name: td_results 2025-08-14T21:21:57.6671154Z s3-bucket: gha-artifacts 2025-08-14T21:21:57.6671642Z region: us-east-1 2025-08-14T21:21:57.6672048Z env: 2025-08-14T21:21:57.6672468Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:57.6673271Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:57.6674423Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:57.6675493Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:57.6677300Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:57.6678929Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:57.6679446Z AWS_REGION: us-east-1 2025-08-14T21:21:57.6680007Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:57.6681141Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:57.6690829Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:57.6691260Z ##[endgroup] 2025-08-14T21:21:58.1289479Z (node:2818689) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:21:58.1291069Z 2025-08-14T21:21:58.1291434Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:21:58.1292405Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:21:58.1293418Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:21:58.3110648Z Found 1 objects with prefix pytorch/pytorch/16976370303/td_results/ 2025-08-14T21:21:58.3111965Z Starting download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2025-08-14T21:21:58.5840406Z Finished download (1/1): /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/td_results.json 2025-08-14T21:21:58.5851842Z Artifact download has finished successfully 2025-08-14T21:21:58.6448789Z ##[group]Run mkdir -p .additional_ci_files 2025-08-14T21:21:58.6449488Z mkdir -p .additional_ci_files 2025-08-14T21:21:58.6450288Z mv td_results.json .additional_ci_files/td_results.json || true 2025-08-14T21:21:58.6500561Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:21:58.6501007Z env: 2025-08-14T21:21:58.6501266Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:58.6501796Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:58.6502900Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:58.6504206Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:58.6506671Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:58.6508361Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:58.6508891Z AWS_REGION: us-east-1 2025-08-14T21:21:58.6509484Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:58.6510172Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:58.6521674Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:58.6522175Z ##[endgroup] 2025-08-14T21:21:58.6738240Z ##[group]Run .github/scripts/parse_ref.py 2025-08-14T21:21:58.6739009Z .github/scripts/parse_ref.py 2025-08-14T21:21:58.6782746Z shell: /usr/bin/bash -e {0} 2025-08-14T21:21:58.6783024Z env: 2025-08-14T21:21:58.6783243Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:58.6783666Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:58.6784352Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:58.6785566Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:58.6787614Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:58.6789223Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:58.6789739Z AWS_REGION: us-east-1 2025-08-14T21:21:58.6790314Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:58.6791063Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:58.6802758Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:58.6803233Z ##[endgroup] 2025-08-14T21:21:58.7251245Z Setting output branch=main 2025-08-14T21:21:58.7482046Z Prepare all required actions 2025-08-14T21:21:58.7482850Z Getting action download info 2025-08-14T21:21:58.9312436Z ##[group]Run ./.github/actions/filter-test-configs 2025-08-14T21:21:58.9312977Z with: 2025-08-14T21:21:58.9313649Z github-token: *** 2025-08-14T21:21:58.9315951Z test-matrix: {"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]} 2025-08-14T21:21:58.9318758Z job-name: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:21:58.9320074Z env: 2025-08-14T21:21:58.9320768Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:58.9321506Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:58.9322518Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:58.9323444Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:58.9325030Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:58.9326436Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:58.9326889Z AWS_REGION: us-east-1 2025-08-14T21:21:58.9327365Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:58.9327966Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:58.9337254Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:58.9337680Z ##[endgroup] 2025-08-14T21:21:58.9424754Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:21:58.9425244Z with: 2025-08-14T21:21:58.9425572Z shell: bash 2025-08-14T21:21:58.9425953Z timeout_minutes: 10 2025-08-14T21:21:58.9426349Z max_attempts: 5 2025-08-14T21:21:58.9426732Z retry_wait_seconds: 30 2025-08-14T21:21:58.9427993Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:21:58.9429272Z polling_interval_seconds: 1 2025-08-14T21:21:58.9429721Z warning_on_retry: true 2025-08-14T21:21:58.9430170Z continue_on_error: false 2025-08-14T21:21:58.9430571Z env: 2025-08-14T21:21:58.9430890Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:21:58.9431563Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:21:58.9432579Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:21:58.9433531Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:21:58.9435097Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:21:58.9436494Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:21:58.9436956Z AWS_REGION: us-east-1 2025-08-14T21:21:58.9437469Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:21:58.9438079Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:21:58.9447772Z AWS_SESSION_TOKEN: *** 2025-08-14T21:21:58.9448421Z GITHUB_TOKEN: *** 2025-08-14T21:21:58.9448804Z ##[endgroup] 2025-08-14T21:21:59.0471773Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:21:59.3311842Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:21:59.5054167Z Requirement already satisfied: requests==2.27.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (2.27.1) 2025-08-14T21:21:59.5057594Z Requirement already satisfied: pyyaml==6.0.2 in /var/home/pytorchci/.local/lib/python3.10/site-packages (6.0.2) 2025-08-14T21:21:59.5145694Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3/dist-packages (from requests==2.27.1) (1.26.5) 2025-08-14T21:21:59.5153244Z Requirement already satisfied: charset-normalizer~=2.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from requests==2.27.1) (2.0.12) 2025-08-14T21:21:59.5161530Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3/dist-packages (from requests==2.27.1) (3.3) 2025-08-14T21:21:59.5165487Z Requirement already satisfied: certifi>=2017.4.17 in /usr/lib/python3/dist-packages (from requests==2.27.1) (2020.6.20) 2025-08-14T21:22:00.0466063Z Command completed after 1 attempt(s). 2025-08-14T21:22:00.0607234Z ##[group]Run set -x 2025-08-14T21:22:00.0607725Z set -x 2025-08-14T21:22:00.0608134Z  2025-08-14T21:22:00.0609269Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:22:00.0610166Z # in runner workspace 2025-08-14T21:22:00.0610896Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-08-14T21:22:00.0662679Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:22:00.0663310Z env: 2025-08-14T21:22:00.0663667Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:22:00.0664347Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:22:00.0665362Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:22:00.0666343Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:22:00.0667951Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:22:00.0669406Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:22:00.0669861Z AWS_REGION: us-east-1 2025-08-14T21:22:00.0670375Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:22:00.0670980Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:22:00.0680937Z AWS_SESSION_TOKEN: *** 2025-08-14T21:22:00.0681339Z ##[endgroup] 2025-08-14T21:22:00.0767720Z + python3 /var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-08-14T21:22:00.1102700Z Setting output branch=main 2025-08-14T21:22:00.1209886Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:22:00.1210624Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:22:00.1211242Z echo "Job name: ${JOB_NAME}" 2025-08-14T21:22:00.1211763Z  2025-08-14T21:22:00.1212444Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:22:00.1213313Z # in runner workspace 2025-08-14T21:22:00.1214080Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-08-14T21:22:00.1215013Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-08-14T21:22:00.1215632Z  --job-name "${JOB_NAME}" \ 2025-08-14T21:22:00.1218427Z  --test-matrix "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}" \ 2025-08-14T21:22:00.1221516Z  --selected-test-configs "" \ 2025-08-14T21:22:00.1222108Z  --pr-number "${PR_NUMBER}" \ 2025-08-14T21:22:00.1222654Z  --tag "${TAG}" \ 2025-08-14T21:22:00.1223164Z  --event-name "${EVENT_NAME}" \ 2025-08-14T21:22:00.1223732Z  --schedule "${SCHEDULE}" \ 2025-08-14T21:22:00.1224281Z  --branch "${HEAD_BRANCH}" 2025-08-14T21:22:00.1275754Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:22:00.1276463Z env: 2025-08-14T21:22:00.1276855Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:22:00.1278017Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:22:00.1279243Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:22:00.1280512Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:22:00.1282454Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:22:00.1284332Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:22:00.1284846Z AWS_REGION: us-east-1 2025-08-14T21:22:00.1285425Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:22:00.1286107Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:22:00.1297280Z AWS_SESSION_TOKEN: *** 2025-08-14T21:22:00.1298016Z GITHUB_TOKEN: *** 2025-08-14T21:22:00.1298963Z JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:22:00.1300143Z PR_NUMBER: 2025-08-14T21:22:00.1300606Z TAG: 2025-08-14T21:22:00.1301003Z EVENT_NAME: schedule 2025-08-14T21:22:00.1301461Z SCHEDULE: 45 4,12,20 * * 1-5 2025-08-14T21:22:00.1301947Z HEAD_BRANCH: main 2025-08-14T21:22:00.1302344Z ##[endgroup] 2025-08-14T21:22:00.1391474Z Workflow: periodic 2025-08-14T21:22:00.1392595Z Job name: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:22:00.5692765Z Setting output keep-going=True 2025-08-14T21:22:00.5693534Z Setting output ci-verbose-test-logs=False 2025-08-14T21:22:00.5694224Z Setting output ci-test-showlocals=False 2025-08-14T21:22:00.5694852Z Setting output ci-no-test-timeout=False 2025-08-14T21:22:00.5695441Z Setting output ci-no-td=False 2025-08-14T21:22:00.5696058Z Setting output ci-td-distributed=False 2025-08-14T21:22:00.5696662Z Setting output is-unstable=False 2025-08-14T21:22:00.5697220Z Setting output reenabled-issues= 2025-08-14T21:22:00.5700188Z Setting output test-matrix={"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]} 2025-08-14T21:22:00.5703221Z Setting output is-test-matrix-empty=False 2025-08-14T21:22:00.5974693Z ##[group]Run echo "Filtered matrix:" 2025-08-14T21:22:00.5975305Z echo "Filtered matrix:" 2025-08-14T21:22:00.5977991Z echo "{"include": [{"config": "distributed", "shard": 1, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 2, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}, {"config": "distributed", "shard": 3, "num_shards": 3, "runner": "linux.rocm.gpu.4", "owners": ["module:rocm", "oncall:distributed"]}]}" 2025-08-14T21:22:00.5980704Z  2025-08-14T21:22:00.5981071Z echo 2025-08-14T21:22:00.5981557Z echo "Is the current job unstable? False" 2025-08-14T21:22:00.5982142Z  2025-08-14T21:22:00.5982490Z echo 2025-08-14T21:22:00.5982924Z echo "Is keep-going label set? True" 2025-08-14T21:22:00.5983472Z  2025-08-14T21:22:00.5983816Z echo 2025-08-14T21:22:00.5984216Z echo "Reenabled issues? " 2025-08-14T21:22:00.6035846Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:22:00.6036527Z env: 2025-08-14T21:22:00.6036917Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:22:00.6037683Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:22:00.6038823Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:22:00.6040681Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:22:00.6042636Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:22:00.6044263Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:22:00.6044775Z AWS_REGION: us-east-1 2025-08-14T21:22:00.6045339Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:22:00.6046016Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:22:00.6057004Z AWS_SESSION_TOKEN: *** 2025-08-14T21:22:00.6057472Z ##[endgroup] 2025-08-14T21:22:00.6145205Z Filtered matrix: 2025-08-14T21:22:00.6147992Z {include: [{config: distributed, shard: 1, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 2, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}, {config: distributed, shard: 3, num_shards: 3, runner: linux.rocm.gpu.4, owners: [module:rocm, oncall:distributed]}]} 2025-08-14T21:22:00.6151240Z 2025-08-14T21:22:00.6151512Z Is the current job unstable? False 2025-08-14T21:22:00.6151911Z 2025-08-14T21:22:00.6152127Z Is keep-going label set? True 2025-08-14T21:22:00.6152478Z 2025-08-14T21:22:00.6152662Z Reenabled issues? 2025-08-14T21:22:00.6227666Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:22:00.6228714Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:22:00.6284233Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:22:00.6284997Z env: 2025-08-14T21:22:00.6285411Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:22:00.6286242Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:22:00.6287468Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:22:00.6288604Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:22:00.6290494Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:22:00.6292182Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:22:00.6292803Z AWS_REGION: us-east-1 2025-08-14T21:22:00.6293395Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:22:00.6294129Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:22:00.6306017Z AWS_SESSION_TOKEN: *** 2025-08-14T21:22:00.6306510Z JOB_TIMEOUT: 300 2025-08-14T21:22:00.6306940Z ##[endgroup] 2025-08-14T21:22:00.6486776Z ##[group]Run set -x 2025-08-14T21:22:00.6487414Z set -x 2025-08-14T21:22:00.6487841Z  2025-08-14T21:22:00.6488320Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-08-14T21:22:00.6489100Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-08-14T21:22:00.6489821Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-08-14T21:22:00.6490474Z  TEST_COMMAND=.ci/caffe2/test.sh 2025-08-14T21:22:00.6491016Z else 2025-08-14T21:22:00.6491467Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:22:00.6492016Z fi 2025-08-14T21:22:00.6492384Z  2025-08-14T21:22:00.6492986Z # detached container should get cleaned up by teardown_ec2_linux 2025-08-14T21:22:00.6493963Z # TODO: Stop building test binaries as part of the build phase 2025-08-14T21:22:00.6494873Z # Used for GPU_FLAG since that doesn't play nice 2025-08-14T21:22:00.6495734Z # shellcheck disable=SC2086,SC2090 2025-08-14T21:22:00.6496382Z container_name=$(docker run \ 2025-08-14T21:22:00.6496970Z  ${GPU_FLAG:-} \ 2025-08-14T21:22:00.6497485Z  -e BUILD_ENVIRONMENT \ 2025-08-14T21:22:00.6498056Z  -e PR_NUMBER \ 2025-08-14T21:22:00.6498557Z  -e GITHUB_ACTIONS \ 2025-08-14T21:22:00.6499079Z  -e GITHUB_REPOSITORY \ 2025-08-14T21:22:00.6499611Z  -e GITHUB_WORKFLOW \ 2025-08-14T21:22:00.6500126Z  -e GITHUB_JOB \ 2025-08-14T21:22:00.6500615Z  -e GITHUB_RUN_ID \ 2025-08-14T21:22:00.6501115Z  -e GITHUB_RUN_NUMBER \ 2025-08-14T21:22:00.6501643Z  -e GITHUB_RUN_ATTEMPT \ 2025-08-14T21:22:00.6502174Z  -e JOB_ID \ 2025-08-14T21:22:00.6502629Z  -e JOB_NAME \ 2025-08-14T21:22:00.6503095Z  -e BRANCH \ 2025-08-14T21:22:00.6503527Z  -e SHA1 \ 2025-08-14T21:22:00.6503999Z  -e AWS_DEFAULT_REGION \ 2025-08-14T21:22:00.6504535Z  -e IN_WHEEL_TEST \ 2025-08-14T21:22:00.6505046Z  -e SHARD_NUMBER \ 2025-08-14T21:22:00.6505543Z  -e TEST_CONFIG \ 2025-08-14T21:22:00.6506029Z  -e NUM_TEST_SHARDS \ 2025-08-14T21:22:00.6507022Z  -e REENABLED_ISSUES \ 2025-08-14T21:22:00.6507573Z  -e CONTINUE_THROUGH_ERROR \ 2025-08-14T21:22:00.6508131Z  -e VERBOSE_TEST_LOGS \ 2025-08-14T21:22:00.6508648Z  -e TEST_SHOWLOCALS \ 2025-08-14T21:22:00.6509149Z  -e NO_TEST_TIMEOUT \ 2025-08-14T21:22:00.6509625Z  -e NO_TD \ 2025-08-14T21:22:00.6510131Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-08-14T21:22:00.6510799Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-08-14T21:22:00.6511455Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-08-14T21:22:00.6512069Z  -e TESTS_TO_INCLUDE \ 2025-08-14T21:22:00.6512594Z  -e DASHBOARD_TAG \ 2025-08-14T21:22:00.6513251Z  --env-file="${RUNNER_TEMP}/github_env_${GITHUB_RUN_ID}" \ 2025-08-14T21:22:00.6514000Z  --ulimit stack=10485760:83886080 \ 2025-08-14T21:22:00.6514578Z  --ulimit core=0 \ 2025-08-14T21:22:00.6515137Z  --security-opt seccomp=unconfined \ 2025-08-14T21:22:00.6515749Z  --cap-add=SYS_PTRACE \ 2025-08-14T21:22:00.6516266Z  --shm-size="8g" \ 2025-08-14T21:22:00.6516730Z  --tty \ 2025-08-14T21:22:00.6517137Z  --detach \ 2025-08-14T21:22:00.6517606Z  --name="${container_name}" \ 2025-08-14T21:22:00.6518170Z  --user jenkins \ 2025-08-14T21:22:00.6518786Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-08-14T21:22:00.6519509Z  -w /var/lib/jenkins/workspace \ 2025-08-14T21:22:00.6520089Z  "${DOCKER_IMAGE}" 2025-08-14T21:22:00.6520692Z ) 2025-08-14T21:22:00.6521136Z # save container name for later step 2025-08-14T21:22:00.6522285Z echo "CONTAINER_NAME=${container_name}" >> "$GITHUB_ENV" 2025-08-14T21:22:00.6523884Z # jenkins user does not have write permission to mounted workspace; work-around by copying within container to jenkins home 2025-08-14T21:22:00.6525734Z docker exec -t "${container_name}" sh -c "cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && ${TEST_COMMAND}" 2025-08-14T21:22:00.6577107Z shell: /usr/bin/bash -e {0} 2025-08-14T21:22:00.6577624Z env: 2025-08-14T21:22:00.6578029Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:22:00.6578831Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-14T21:22:00.6580025Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-14T21:22:00.6581230Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-14T21:22:00.6583264Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-14T21:22:00.6584891Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:22:00.6585399Z AWS_REGION: us-east-1 2025-08-14T21:22:00.6585960Z AWS_ACCESS_KEY_ID: *** 2025-08-14T21:22:00.6586649Z AWS_SECRET_ACCESS_KEY: *** 2025-08-14T21:22:00.6597585Z AWS_SESSION_TOKEN: *** 2025-08-14T21:22:00.6598132Z BUILD_ENVIRONMENT: linux-jammy-rocm-py3.10 2025-08-14T21:22:00.6598722Z PR_NUMBER: 2025-08-14T21:22:00.6599159Z GITHUB_REPOSITORY: pytorch/pytorch 2025-08-14T21:22:00.6599725Z GITHUB_WORKFLOW: periodic 2025-08-14T21:22:00.6600518Z GITHUB_JOB: test 2025-08-14T21:22:00.6600990Z GITHUB_RUN_ID: 16976370303 2025-08-14T21:22:00.6601484Z GITHUB_RUN_NUMBER: 17921 2025-08-14T21:22:00.6601941Z GITHUB_RUN_ATTEMPT: 1 2025-08-14T21:22:00.6602377Z JOB_ID: 48128182520 2025-08-14T21:22:00.6603314Z JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:22:00.6604498Z BRANCH: main 2025-08-14T21:22:00.6605088Z SHA1: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:00.6605786Z CONTINUE_THROUGH_ERROR: True 2025-08-14T21:22:00.6606287Z VERBOSE_TEST_LOGS: False 2025-08-14T21:22:00.6606757Z TEST_SHOWLOCALS: False 2025-08-14T21:22:00.6607637Z NO_TEST_TIMEOUT: False 2025-08-14T21:22:00.6608069Z NO_TD: False 2025-08-14T21:22:00.6608464Z TEST_CONFIG: distributed 2025-08-14T21:22:00.6608897Z SHARD_NUMBER: 2 2025-08-14T21:22:00.6609298Z NUM_TEST_SHARDS: 3 2025-08-14T21:22:00.6609714Z REENABLED_ISSUES: 2025-08-14T21:22:00.6611002Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:22:00.6612417Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-08-14T21:22:00.6612988Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-08-14T21:22:00.6613516Z TESTS_TO_INCLUDE: 2025-08-14T21:22:00.6613923Z DASHBOARD_TAG: 2025-08-14T21:22:00.6614327Z ##[endgroup] 2025-08-14T21:22:00.6701257Z + [[ distributed == \m\u\l\t\i\g\p\u ]] 2025-08-14T21:22:00.6702021Z + [[ linux-jammy-rocm-py3.10 == *onnx* ]] 2025-08-14T21:22:00.6702652Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:22:00.6721938Z +++ nproc --ignore=2 2025-08-14T21:22:00.6759472Z ++ docker run --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e MAX_JOBS=126 -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e TESTS_TO_INCLUDE -e DASHBOARD_TAG --env-file=/var/home/pytorchci/actions-runner/_work/_temp/github_env_16976370303 --ulimit stack=10485760:83886080 --ulimit core=0 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --shm-size=8g --tty --detach --name= --user jenkins -v /var/home/pytorchci/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-rocm-n-py3-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:22:00.8860406Z + container_name=aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-14T21:22:00.8861994Z + echo CONTAINER_NAME=aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-14T21:22:00.8864411Z + docker exec -t aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 sh -c 'cd .. && cp -R workspace pytorch && cd pytorch && pip install dist/*.whl && .ci/pytorch/test.sh' 2025-08-14T21:22:17.0095411Z Processing ./dist/torch-2.9.0a0+git1fc683c-cp310-cp310-linux_x86_64.whl 2025-08-14T21:22:17.7795493Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.18.0) 2025-08-14T21:22:17.7797701Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (4.14.1) 2025-08-14T21:22:17.7800402Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (1.13.3) 2025-08-14T21:22:17.7804200Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2.8.8) 2025-08-14T21:22:17.7806352Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (3.1.6) 2025-08-14T21:22:17.7809998Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from torch==2.9.0a0+git1fc683c) (2025.5.1) 2025-08-14T21:22:17.8111817Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git1fc683c) (1.3.0) 2025-08-14T21:22:17.8155739Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.10/lib/python3.10/site-packages (from jinja2->torch==2.9.0a0+git1fc683c) (3.0.2) 2025-08-14T21:22:18.1718727Z Installing collected packages: torch 2025-08-14T21:22:29.0311124Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-08-14T21:22:29.0313038Z helion 0.1.0 requires filecheck, which is not installed. 2025-08-14T21:22:29.0313954Z Successfully installed torch-2.9.0a0+git1fc683c 2025-08-14T21:22:29.0799665Z + export TERM=vt100 2025-08-14T21:22:29.0801466Z + TERM=vt100 2025-08-14T21:22:29.0809762Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:22:29.0840989Z + source .ci/pytorch/common.sh 2025-08-14T21:22:29.0852780Z +++ dirname .ci/pytorch/common.sh 2025-08-14T21:22:29.0879158Z ++ source .ci/pytorch/common_utils.sh 2025-08-14T21:22:29.0880963Z +++ declare -f -t trap_add 2025-08-14T21:22:29.0891918Z ++ set -ex -o pipefail 2025-08-14T21:22:29.0892595Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-14T21:22:29.0893656Z ++ unset HIP_PLATFORM 2025-08-14T21:22:29.0895077Z ++ export PYTORCH_TEST_WITH_ROCM=1 2025-08-14T21:22:29.0896495Z ++ PYTORCH_TEST_WITH_ROCM=1 2025-08-14T21:22:29.0897785Z ++ BUILD_TEST_LIBTORCH=0 2025-08-14T21:22:29.0905669Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:22:29.0932888Z + source .ci/pytorch/common-build.sh 2025-08-14T21:22:29.0957921Z ++ [[ linux-jammy-rocm-py3.10 != *win-* ]] 2025-08-14T21:22:29.0958668Z ++++ dirname .ci/pytorch/common-build.sh 2025-08-14T21:22:29.0987436Z +++ cd .ci/pytorch 2025-08-14T21:22:29.0987976Z +++ pwd -P 2025-08-14T21:22:29.0994534Z ++ script_dir=/var/lib/jenkins/pytorch/.ci/pytorch 2025-08-14T21:22:29.1000943Z ++ [[ linux-jammy-rocm-py3.10 == *-pch* ]] 2025-08-14T21:22:29.1001613Z ++ which sccache 2025-08-14T21:22:29.1030259Z ++ [[ -z '' ]] 2025-08-14T21:22:29.1030776Z ++ unset SCCACHE_BUCKET 2025-08-14T21:22:29.1031270Z ++ unset SCCACHE_REGION 2025-08-14T21:22:29.1031786Z ++ sccache --stop-server 2025-08-14T21:22:29.1102973Z ++ true 2025-08-14T21:22:29.1103610Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-08-14T21:22:29.1135670Z ++ trap_add sccache_epilogue EXIT 2025-08-14T21:22:29.1136465Z ++ trap_add_cmd=sccache_epilogue 2025-08-14T21:22:29.1136999Z ++ shift 2025-08-14T21:22:29.1137418Z ++ for trap_add_name in "$@" 2025-08-14T21:22:29.1156704Z ++++ trap -p EXIT 2025-08-14T21:22:29.1168073Z +++ eval 'extract_trap_cmd ' 2025-08-14T21:22:29.1168684Z ++++ extract_trap_cmd 2025-08-14T21:22:29.1169135Z ++++ printf '%s\n' '' 2025-08-14T21:22:29.1169622Z +++ printf '%s\n' sccache_epilogue 2025-08-14T21:22:29.1172814Z ++ trap -- ' 2025-08-14T21:22:29.1173358Z sccache_epilogue' EXIT 2025-08-14T21:22:29.1173872Z ++ [[ -n '' ]] 2025-08-14T21:22:29.1174443Z ++ [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-14T21:22:29.1175276Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:22:29.1176053Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:22:29.1176628Z ++ sccache --start-server 2025-08-14T21:22:29.1232354Z sccache: Starting the server... 2025-08-14T21:22:29.1942399Z sccache: Listening on address 127.0.0.1:4226 2025-08-14T21:22:29.1971566Z ++ sccache --zero-stats 2025-08-14T21:22:29.2023601Z Statistics zeroed. 2025-08-14T21:22:29.2032807Z ++ which ccache 2025-08-14T21:22:29.2066313Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]] 2025-08-14T21:22:29.2066974Z + echo 'Environment variables:' 2025-08-14T21:22:29.2067547Z Environment variables: 2025-08-14T21:22:29.2067993Z + env 2025-08-14T21:22:29.2096802Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2025-08-14T21:22:29.2097762Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:22:29.2098317Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10 2025-08-14T21:22:29.2098965Z HOSTNAME=gpu9498.jax.cs.cpe.ice.amd.com 2025-08-14T21:22:29.2100242Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2101450Z GITHUB_ACTION=__self 2025-08-14T21:22:29.2102624Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:22:29.2103174Z GITHUB_RUN_NUMBER=17921 2025-08-14T21:22:29.2103641Z TEST_CONFIG=distributed 2025-08-14T21:22:29.2104107Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:22:29.2104660Z AWS_DEFAULT_REGION=us-east-1 2025-08-14T21:22:29.2105169Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-08-14T21:22:29.2105663Z GITHUB_REF_TYPE=branch 2025-08-14T21:22:29.2109142Z *** 2025-08-14T21:22:29.2109568Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:22:29.2110104Z GITHUB_ACTIONS=true 2025-08-14T21:22:29.2110603Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2111255Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2112229Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main 2025-08-14T21:22:29.2113103Z UCC_HOME=/usr 2025-08-14T21:22:29.2113507Z VERBOSE_TEST_LOGS=False 2025-08-14T21:22:29.2113965Z GITHUB_REF=refs/heads/main 2025-08-14T21:22:29.2114419Z SHARD_NUMBER=2 2025-08-14T21:22:29.2114842Z GITHUB_REF_PROTECTED=true 2025-08-14T21:22:29.2115296Z HOME=/var/lib/jenkins 2025-08-14T21:22:29.2115789Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:22:29.2116387Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:22:29.2116902Z LANG=C.UTF-8 2025-08-14T21:22:29.2117384Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6 2025-08-14T21:22:29.2118014Z PYTORCH_TEST_WITH_ROCM=1 2025-08-14T21:22:29.2118458Z NUM_TEST_SHARDS=3 2025-08-14T21:22:29.2118861Z UCX_HOME=/usr 2025-08-14T21:22:29.2119952Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2121810Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:22:29.2123233Z MAGMA_HOME=/opt/rocm/magma 2025-08-14T21:22:29.2124369Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2125956Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:22:29.2126903Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:22:29.2127358Z DASHBOARD_TAG= 2025-08-14T21:22:29.2127770Z GITHUB_RUN_ID=16976370303 2025-08-14T21:22:29.2128951Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2130199Z GITHUB_ACTOR=huydhn 2025-08-14T21:22:29.2130603Z PR_NUMBER= 2025-08-14T21:22:29.2130990Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:22:29.2131430Z ANACONDA_PYTHON_VERSION=3.10 2025-08-14T21:22:29.2132024Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:22:29.2132628Z TERM=vt100 2025-08-14T21:22:29.2132997Z INSTALLED_VISION=yes 2025-08-14T21:22:29.2133415Z BRANCH=main 2025-08-14T21:22:29.2133814Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:22:29.2134292Z TESTS_TO_INCLUDE= 2025-08-14T21:22:29.2135210Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2025-08-14T21:22:29.2136292Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:22:29.2136863Z PYTORCH_ROCM_ARCH=gfx90a;gfx942 2025-08-14T21:22:29.2137420Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d 2025-08-14T21:22:29.2138005Z REENABLED_ISSUES= 2025-08-14T21:22:29.2138386Z SHLVL=1 2025-08-14T21:22:29.2138729Z MAX_JOBS=126 2025-08-14T21:22:29.2139111Z GITHUB_ACTOR_ID=475357 2025-08-14T21:22:29.2139696Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2140358Z GITHUB_REF_NAME=main 2025-08-14T21:22:29.2140775Z ROCM_PATH=/opt/rocm 2025-08-14T21:22:29.2141175Z GITHUB_JOB=test 2025-08-14T21:22:29.2141570Z NO_TEST_TIMEOUT=False 2025-08-14T21:22:29.2142018Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:22:29.2142532Z LC_ALL=C.UTF-8 2025-08-14T21:22:29.2142937Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:22:29.2143387Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:22:29.2143844Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:22:29.2145613Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:22:29.2147740Z GITHUB_BASE_REF= 2025-08-14T21:22:29.2148127Z CI=true 2025-08-14T21:22:29.2148507Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:22:29.2148989Z JOB_ID=48128182520 2025-08-14T21:22:29.2149399Z GITHUB_HEAD_REF= 2025-08-14T21:22:29.2149798Z GITHUB_ACTION_REF= 2025-08-14T21:22:29.2150207Z TEST_SHOWLOCALS=False 2025-08-14T21:22:29.2150658Z GITHUB_WORKFLOW=periodic 2025-08-14T21:22:29.2151144Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:22:29.2152335Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2153537Z NO_TD=False 2025-08-14T21:22:29.2153925Z OLDPWD=/var/lib/jenkins 2025-08-14T21:22:29.2154352Z _=/usr/bin/env 2025-08-14T21:22:29.2154926Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-08-14T21:22:29.2385793Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch 2025-08-14T21:22:29.2386998Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/bin 2025-08-14T21:22:29.2388016Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/lib 2025-08-14T21:22:29.2389226Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/test 2025-08-14T21:22:29.2390133Z + BUILD_DIR=build 2025-08-14T21:22:29.2390647Z + BUILD_RENAMED_DIR=build_renamed 2025-08-14T21:22:29.2391177Z + BUILD_BIN_DIR=build/bin 2025-08-14T21:22:29.2391630Z + SHARD_NUMBER=2 2025-08-14T21:22:29.2392028Z + NUM_TEST_SHARDS=3 2025-08-14T21:22:29.2392475Z + export TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:22:29.2393415Z + TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:22:29.2393944Z + export VALGRIND=ON 2025-08-14T21:22:29.2394377Z + VALGRIND=ON 2025-08-14T21:22:29.2394825Z + [[ linux-jammy-rocm-py3.10 == *clang9* ]] 2025-08-14T21:22:29.2395441Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]] 2025-08-14T21:22:29.2396016Z + [[ linux-jammy-rocm-py3.10 == *s390x* ]] 2025-08-14T21:22:29.2396546Z + [[ 0 == \1 ]] 2025-08-14T21:22:29.2396935Z + [[ True == \1 ]] 2025-08-14T21:22:29.2397372Z + [[ linux-jammy-rocm-py3.10 != *bazel* ]] 2025-08-14T21:22:29.2407630Z ++ realpath build/custom_test_artifacts 2025-08-14T21:22:29.2437537Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/pytorch/build/custom_test_artifacts 2025-08-14T21:22:29.2438428Z + [[ -n '' ]] 2025-08-14T21:22:29.2438856Z + echo 'Environment variables' 2025-08-14T21:22:29.2439361Z Environment variables 2025-08-14T21:22:29.2439834Z + env 2025-08-14T21:22:29.2473126Z GITHUB_WORKSPACE=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2025-08-14T21:22:29.2474129Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:22:29.2474741Z BUILD_ENVIRONMENT=linux-jammy-rocm-py3.10 2025-08-14T21:22:29.2475359Z HOSTNAME=gpu9498.jax.cs.cpe.ice.amd.com 2025-08-14T21:22:29.2476551Z GITHUB_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/add_path_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2477848Z GITHUB_ACTION=__self 2025-08-14T21:22:29.2478400Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:22:29.2479023Z GITHUB_RUN_NUMBER=17921 2025-08-14T21:22:29.2479519Z TEST_CONFIG=distributed 2025-08-14T21:22:29.2479983Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:22:29.2480632Z AWS_DEFAULT_REGION=us-east-1 2025-08-14T21:22:29.2481136Z GITHUB_TRIGGERING_ACTOR=huydhn 2025-08-14T21:22:29.2481637Z GITHUB_REF_TYPE=branch 2025-08-14T21:22:29.2482195Z *** 2025-08-14T21:22:29.2482649Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:22:29.2483144Z GITHUB_ACTIONS=true 2025-08-14T21:22:29.2483638Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2484308Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2485234Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/periodic.yml@refs/heads/main 2025-08-14T21:22:29.2486081Z UCC_HOME=/usr 2025-08-14T21:22:29.2487008Z TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:22:29.2487563Z VERBOSE_TEST_LOGS=False 2025-08-14T21:22:29.2488015Z GITHUB_REF=refs/heads/main 2025-08-14T21:22:29.2488468Z SHARD_NUMBER=2 2025-08-14T21:22:29.2488877Z GITHUB_REF_PROTECTED=true 2025-08-14T21:22:29.2489329Z HOME=/var/lib/jenkins 2025-08-14T21:22:29.2489815Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:22:29.2490393Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:22:29.2490904Z LANG=C.UTF-8 2025-08-14T21:22:29.2491370Z UCX_COMMIT=cc312eaa4655c0cc5c2bcd796db938f90563bcf6 2025-08-14T21:22:29.2492002Z PYTORCH_TEST_WITH_ROCM=1 2025-08-14T21:22:29.2492472Z NUM_TEST_SHARDS=3 2025-08-14T21:22:29.2492867Z UCX_HOME=/usr 2025-08-14T21:22:29.2493976Z GITHUB_STATE=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/save_state_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2495682Z JOB_NAME=linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-14T21:22:29.2496735Z MAGMA_HOME=/opt/rocm/magma 2025-08-14T21:22:29.2498048Z GITHUB_ENV=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_env_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2499618Z GITHUB_EVENT_PATH=/var/home/pytorchci/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:22:29.2500566Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:22:29.2501012Z DASHBOARD_TAG= 2025-08-14T21:22:29.2501410Z GITHUB_RUN_ID=16976370303 2025-08-14T21:22:29.2502573Z GITHUB_STEP_SUMMARY=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/step_summary_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2503817Z GITHUB_ACTOR=huydhn 2025-08-14T21:22:29.2504219Z PR_NUMBER= 2025-08-14T21:22:29.2504599Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:22:29.2505326Z VALGRIND=ON 2025-08-14T21:22:29.2505744Z ANACONDA_PYTHON_VERSION=3.10 2025-08-14T21:22:29.2506342Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:22:29.2506973Z TERM=vt100 2025-08-14T21:22:29.2507345Z INSTALLED_VISION=yes 2025-08-14T21:22:29.2507775Z BRANCH=main 2025-08-14T21:22:29.2508169Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:22:29.2508639Z TESTS_TO_INCLUDE= 2025-08-14T21:22:29.2509580Z GITHUB_ACTION_PATH=/var/home/pytorchci/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-rocm 2025-08-14T21:22:29.2510691Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:22:29.2511262Z PYTORCH_ROCM_ARCH=gfx90a;gfx942 2025-08-14T21:22:29.2511838Z UCC_COMMIT=0c0fc21559835044ab107199e334f7157d6a0d3d 2025-08-14T21:22:29.2512426Z REENABLED_ISSUES= 2025-08-14T21:22:29.2512819Z SHLVL=1 2025-08-14T21:22:29.2513165Z MAX_JOBS=126 2025-08-14T21:22:29.2513547Z GITHUB_ACTOR_ID=475357 2025-08-14T21:22:29.2514120Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:22:29.2514794Z GITHUB_REF_NAME=main 2025-08-14T21:22:29.2515202Z ROCM_PATH=/opt/rocm 2025-08-14T21:22:29.2515601Z GITHUB_JOB=test 2025-08-14T21:22:29.2516000Z NO_TEST_TIMEOUT=False 2025-08-14T21:22:29.2516471Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:22:29.2516998Z LC_ALL=C.UTF-8 2025-08-14T21:22:29.2517411Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:22:29.2517896Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:22:29.2518360Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:22:29.2520113Z PATH=/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:22:29.2522308Z GITHUB_BASE_REF= 2025-08-14T21:22:29.2522768Z CI=true 2025-08-14T21:22:29.2523212Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:22:29.2523783Z JOB_ID=48128182520 2025-08-14T21:22:29.2524197Z GITHUB_HEAD_REF= 2025-08-14T21:22:29.2524587Z GITHUB_ACTION_REF= 2025-08-14T21:22:29.2524982Z TEST_SHOWLOCALS=False 2025-08-14T21:22:29.2525427Z GITHUB_WORKFLOW=periodic 2025-08-14T21:22:29.2525901Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:22:29.2527088Z GITHUB_OUTPUT=/var/home/pytorchci/actions-runner/_work/_temp/_runner_file_commands/set_output_71435060-d88c-4df0-8ca2-21c726a512ba 2025-08-14T21:22:29.2528648Z NO_TD=False 2025-08-14T21:22:29.2529042Z OLDPWD=/var/lib/jenkins 2025-08-14T21:22:29.2529463Z _=/usr/bin/env 2025-08-14T21:22:29.2529869Z + echo 'Testing pytorch' 2025-08-14T21:22:29.2530315Z Testing pytorch 2025-08-14T21:22:29.2530726Z + export LANG=C.UTF-8 2025-08-14T21:22:29.2531132Z + LANG=C.UTF-8 2025-08-14T21:22:29.2531506Z + PR_NUMBER= 2025-08-14T21:22:29.2531923Z + [[ distributed == \d\e\f\a\u\l\t ]] 2025-08-14T21:22:29.2532488Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-08-14T21:22:29.2533091Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-14T21:22:29.2533677Z + export HIP_VISIBLE_DEVICES=0,1,2,3 2025-08-14T21:22:29.2534202Z + HIP_VISIBLE_DEVICES=0,1,2,3 2025-08-14T21:22:29.2534700Z + [[ distributed == \s\l\o\w ]] 2025-08-14T21:22:29.2535364Z + [[ linux-jammy-rocm-py3.10 == *slow-gradcheck* ]] 2025-08-14T21:22:29.2536126Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]] 2025-08-14T21:22:29.2536749Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-14T21:22:29.2537356Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-08-14T21:22:29.2537956Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda 2025-08-14T21:22:29.2538511Z + [[ distributed == *crossref* ]] 2025-08-14T21:22:29.2539034Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-14T21:22:29.2539567Z + export VALGRIND=OFF 2025-08-14T21:22:29.2539978Z + VALGRIND=OFF 2025-08-14T21:22:29.2540344Z + rocminfo 2025-08-14T21:22:29.2649953Z ROCk module version 6.10.5 is loaded 2025-08-14T21:22:29.4652360Z ===================== 2025-08-14T21:22:29.4653033Z HSA System Attributes 2025-08-14T21:22:29.4653518Z ===================== 2025-08-14T21:22:29.4654098Z Runtime Version: 1.15 2025-08-14T21:22:29.4654691Z Runtime Ext Version: 1.7 2025-08-14T21:22:29.4655719Z System Timestamp Freq.: 1000.000000MHz 2025-08-14T21:22:29.4656615Z Sig. Max Wait Duration: 18446744073709551615 (0xFFFFFFFFFFFFFFFF) (timestamp count) 2025-08-14T21:22:29.4657573Z Machine Model: LARGE 2025-08-14T21:22:29.4658364Z System Endianness: LITTLE 2025-08-14T21:22:29.4659032Z Mwaitx: DISABLED 2025-08-14T21:22:29.4659586Z XNACK enabled: NO 2025-08-14T21:22:29.4660079Z DMAbuf Support: YES 2025-08-14T21:22:29.4660563Z VMM Support: YES 2025-08-14T21:22:29.4660896Z 2025-08-14T21:22:29.4661064Z ========== 2025-08-14T21:22:29.4661506Z HSA Agents 2025-08-14T21:22:29.4661965Z ========== 2025-08-14T21:22:29.4662375Z ******* 2025-08-14T21:22:29.4662775Z Agent 1 2025-08-14T21:22:29.4663210Z ******* 2025-08-14T21:22:29.4663753Z Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.4664450Z Uuid: CPU-XX 2025-08-14T21:22:29.4665255Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.4666027Z Vendor Name: CPU 2025-08-14T21:22:29.4666742Z Feature: None specified 2025-08-14T21:22:29.4667426Z Profile: FULL_PROFILE 2025-08-14T21:22:29.4668167Z Float Round Mode: NEAR 2025-08-14T21:22:29.4668864Z Max Queue Number: 0(0x0) 2025-08-14T21:22:29.4669546Z Queue Min Size: 0(0x0) 2025-08-14T21:22:29.4670307Z Queue Max Size: 0(0x0) 2025-08-14T21:22:29.4670992Z Queue Type: MULTI 2025-08-14T21:22:29.4671730Z Node: 0 2025-08-14T21:22:29.4672486Z Device Type: CPU 2025-08-14T21:22:29.4673208Z Cache Info: 2025-08-14T21:22:29.4673742Z L1: 32768(0x8000) KB 2025-08-14T21:22:29.4674779Z Chip ID: 0(0x0) 2025-08-14T21:22:29.4675475Z ASIC Revision: 0(0x0) 2025-08-14T21:22:29.4676193Z Cacheline Size: 64(0x40) 2025-08-14T21:22:29.4676909Z Max Clock Freq. (MHz): 2000 2025-08-14T21:22:29.4677562Z BDFID: 0 2025-08-14T21:22:29.4678228Z Internal Node ID: 0 2025-08-14T21:22:29.4678958Z Compute Unit: 64 2025-08-14T21:22:29.4679625Z SIMDs per CU: 0 2025-08-14T21:22:29.4680509Z Shader Engines: 0 2025-08-14T21:22:29.4681263Z Shader Arrs. per Eng.: 0 2025-08-14T21:22:29.4682121Z WatchPts on Addr. Ranges:1 2025-08-14T21:22:29.4682920Z Memory Properties: 2025-08-14T21:22:29.4683470Z Features: None 2025-08-14T21:22:29.4684051Z Pool Info: 2025-08-14T21:22:29.4684586Z Pool 1 2025-08-14T21:22:29.4685263Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.4685945Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:22:29.4686419Z Allocatable: TRUE 2025-08-14T21:22:29.4686852Z Alloc Granule: 4KB 2025-08-14T21:22:29.4687303Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4687954Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4688413Z Accessible by all: TRUE 2025-08-14T21:22:29.4688789Z Pool 2 2025-08-14T21:22:29.4689153Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.4689596Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:22:29.4690028Z Allocatable: TRUE 2025-08-14T21:22:29.4690471Z Alloc Granule: 4KB 2025-08-14T21:22:29.4690934Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4691395Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4691838Z Accessible by all: TRUE 2025-08-14T21:22:29.4692219Z Pool 3 2025-08-14T21:22:29.4692572Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2025-08-14T21:22:29.4692990Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:22:29.4693394Z Allocatable: TRUE 2025-08-14T21:22:29.4693826Z Alloc Granule: 4KB 2025-08-14T21:22:29.4694282Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4694735Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4695171Z Accessible by all: TRUE 2025-08-14T21:22:29.4695544Z Pool 4 2025-08-14T21:22:29.4695883Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.4696289Z Size: 528196636(0x1f7ba41c) KB 2025-08-14T21:22:29.4696869Z Allocatable: TRUE 2025-08-14T21:22:29.4697304Z Alloc Granule: 4KB 2025-08-14T21:22:29.4697763Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4698216Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4698657Z Accessible by all: TRUE 2025-08-14T21:22:29.4699310Z ISA Info: 2025-08-14T21:22:29.4699688Z ******* 2025-08-14T21:22:29.4700214Z Agent 2 2025-08-14T21:22:29.4700734Z ******* 2025-08-14T21:22:29.4701354Z Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.4702153Z Uuid: CPU-XX 2025-08-14T21:22:29.4702867Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.4703591Z Vendor Name: CPU 2025-08-14T21:22:29.4704284Z Feature: None specified 2025-08-14T21:22:29.4704988Z Profile: FULL_PROFILE 2025-08-14T21:22:29.4705691Z Float Round Mode: NEAR 2025-08-14T21:22:29.4706401Z Max Queue Number: 0(0x0) 2025-08-14T21:22:29.4707101Z Queue Min Size: 0(0x0) 2025-08-14T21:22:29.4707792Z Queue Max Size: 0(0x0) 2025-08-14T21:22:29.4708468Z Queue Type: MULTI 2025-08-14T21:22:29.4709107Z Node: 1 2025-08-14T21:22:29.4709766Z Device Type: CPU 2025-08-14T21:22:29.4710377Z Cache Info: 2025-08-14T21:22:29.4710880Z L1: 32768(0x8000) KB 2025-08-14T21:22:29.4711499Z Chip ID: 0(0x0) 2025-08-14T21:22:29.4712163Z ASIC Revision: 0(0x0) 2025-08-14T21:22:29.4713133Z Cacheline Size: 64(0x40) 2025-08-14T21:22:29.4713871Z Max Clock Freq. (MHz): 2000 2025-08-14T21:22:29.4714527Z BDFID: 0 2025-08-14T21:22:29.4715209Z Internal Node ID: 1 2025-08-14T21:22:29.4715892Z Compute Unit: 64 2025-08-14T21:22:29.4716557Z SIMDs per CU: 0 2025-08-14T21:22:29.4717234Z Shader Engines: 0 2025-08-14T21:22:29.4717942Z Shader Arrs. per Eng.: 0 2025-08-14T21:22:29.4718673Z WatchPts on Addr. Ranges:1 2025-08-14T21:22:29.4719320Z Memory Properties: 2025-08-14T21:22:29.4719799Z Features: None 2025-08-14T21:22:29.4720573Z Pool Info: 2025-08-14T21:22:29.4721075Z Pool 1 2025-08-14T21:22:29.4721686Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.4722498Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:22:29.4723300Z Allocatable: TRUE 2025-08-14T21:22:29.4724146Z Alloc Granule: 4KB 2025-08-14T21:22:29.4724915Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4725690Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4726422Z Accessible by all: TRUE 2025-08-14T21:22:29.4727052Z Pool 2 2025-08-14T21:22:29.4727623Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.4728314Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:22:29.4728983Z Allocatable: TRUE 2025-08-14T21:22:29.4729727Z Alloc Granule: 4KB 2025-08-14T21:22:29.4730522Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4731313Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4732460Z Accessible by all: TRUE 2025-08-14T21:22:29.4733139Z Pool 3 2025-08-14T21:22:29.4733736Z Segment: GLOBAL; FLAGS: KERNARG, FINE GRAINED 2025-08-14T21:22:29.4734451Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:22:29.4735154Z Allocatable: TRUE 2025-08-14T21:22:29.4735904Z Alloc Granule: 4KB 2025-08-14T21:22:29.4736686Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4737480Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4738256Z Accessible by all: TRUE 2025-08-14T21:22:29.4738918Z Pool 4 2025-08-14T21:22:29.4739606Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.4740455Z Size: 528455492(0x1f7f9744) KB 2025-08-14T21:22:29.4741182Z Allocatable: TRUE 2025-08-14T21:22:29.4741937Z Alloc Granule: 4KB 2025-08-14T21:22:29.4742717Z Alloc Recommended Granule:4KB 2025-08-14T21:22:29.4743503Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4744274Z Accessible by all: TRUE 2025-08-14T21:22:29.4744931Z ISA Info: 2025-08-14T21:22:29.4745391Z ******* 2025-08-14T21:22:29.4745845Z Agent 3 2025-08-14T21:22:29.4746615Z ******* 2025-08-14T21:22:29.4747141Z Name: gfx90a 2025-08-14T21:22:29.4747823Z Uuid: GPU-bb978094b96db57c 2025-08-14T21:22:29.4748578Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.4749347Z Vendor Name: AMD 2025-08-14T21:22:29.4750081Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.4750810Z Profile: BASE_PROFILE 2025-08-14T21:22:29.4751569Z Float Round Mode: NEAR 2025-08-14T21:22:29.4752322Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.4753052Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.4753771Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.4754502Z Queue Type: MULTI 2025-08-14T21:22:29.4755181Z Node: 2 2025-08-14T21:22:29.4755870Z Device Type: GPU 2025-08-14T21:22:29.4756516Z Cache Info: 2025-08-14T21:22:29.4757049Z L1: 16(0x10) KB 2025-08-14T21:22:29.4757671Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.4758305Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.4759001Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.4759750Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.4760614Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.4761329Z BDFID: 12544 2025-08-14T21:22:29.4762095Z Internal Node ID: 2 2025-08-14T21:22:29.4762982Z Compute Unit: 104 2025-08-14T21:22:29.4763837Z SIMDs per CU: 4 2025-08-14T21:22:29.4764571Z Shader Engines: 8 2025-08-14T21:22:29.4765756Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.4766550Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.4767341Z Coherent Host Access: FALSE 2025-08-14T21:22:29.4768039Z Memory Properties: 2025-08-14T21:22:29.4768564Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.4769261Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.4770022Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.4770784Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4771489Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4772085Z x 1024(0x400) 2025-08-14T21:22:29.4772696Z y 1024(0x400) 2025-08-14T21:22:29.4773288Z z 1024(0x400) 2025-08-14T21:22:29.4773964Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.4774701Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.4775440Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4776100Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4776637Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4777240Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4777846Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4778549Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.4779877Z Packet Processor uCode:: 92 2025-08-14T21:22:29.4780802Z SDMA engine uCode:: 9 2025-08-14T21:22:29.4781558Z IOMMU Support:: None 2025-08-14T21:22:29.4782214Z Pool Info: 2025-08-14T21:22:29.4782687Z Pool 1 2025-08-14T21:22:29.4783287Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.4784020Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4784742Z Allocatable: TRUE 2025-08-14T21:22:29.4785498Z Alloc Granule: 4KB 2025-08-14T21:22:29.4786285Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4787079Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4787853Z Accessible by all: FALSE 2025-08-14T21:22:29.4788518Z Pool 2 2025-08-14T21:22:29.4789122Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.4789861Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4790583Z Allocatable: TRUE 2025-08-14T21:22:29.4791340Z Alloc Granule: 4KB 2025-08-14T21:22:29.4792122Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4792911Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4793683Z Accessible by all: FALSE 2025-08-14T21:22:29.4794345Z Pool 3 2025-08-14T21:22:29.4794925Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.4795628Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4796335Z Allocatable: TRUE 2025-08-14T21:22:29.4797091Z Alloc Granule: 4KB 2025-08-14T21:22:29.4797866Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4798962Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4799739Z Accessible by all: FALSE 2025-08-14T21:22:29.4800532Z Pool 4 2025-08-14T21:22:29.4801117Z Segment: GROUP 2025-08-14T21:22:29.4801839Z Size: 64(0x40) KB 2025-08-14T21:22:29.4802672Z Allocatable: FALSE 2025-08-14T21:22:29.4803562Z Alloc Granule: 0KB 2025-08-14T21:22:29.4804403Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.4805196Z Alloc Alignment: 0KB 2025-08-14T21:22:29.4805971Z Accessible by all: FALSE 2025-08-14T21:22:29.4806631Z ISA Info: 2025-08-14T21:22:29.4807095Z ISA 1 2025-08-14T21:22:29.4807717Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.4808538Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.4809318Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.4810095Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4810888Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4811631Z Fast f16: TRUE 2025-08-14T21:22:29.4812373Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4813084Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4814031Z x 1024(0x400) 2025-08-14T21:22:29.4814674Z y 1024(0x400) 2025-08-14T21:22:29.4815277Z z 1024(0x400) 2025-08-14T21:22:29.4815973Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4816647Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4817202Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4817803Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4818412Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4819116Z FBarrier Max Size: 32 2025-08-14T21:22:29.4819891Z ******* 2025-08-14T21:22:29.4820426Z Agent 4 2025-08-14T21:22:29.4820925Z ******* 2025-08-14T21:22:29.4821445Z Name: gfx90a 2025-08-14T21:22:29.4822132Z Uuid: GPU-f4a8e4f76a6ff8b4 2025-08-14T21:22:29.4822883Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.4823654Z Vendor Name: AMD 2025-08-14T21:22:29.4824378Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.4825108Z Profile: BASE_PROFILE 2025-08-14T21:22:29.4825848Z Float Round Mode: NEAR 2025-08-14T21:22:29.4826592Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.4827319Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.4828043Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.4828763Z Queue Type: MULTI 2025-08-14T21:22:29.4829450Z Node: 3 2025-08-14T21:22:29.4830137Z Device Type: GPU 2025-08-14T21:22:29.4830775Z Cache Info: 2025-08-14T21:22:29.4831296Z L1: 16(0x10) KB 2025-08-14T21:22:29.4832273Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.4832920Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.4833616Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.4834355Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.4835107Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.4835804Z BDFID: 13312 2025-08-14T21:22:29.4836500Z Internal Node ID: 3 2025-08-14T21:22:29.4837238Z Compute Unit: 104 2025-08-14T21:22:29.4837958Z SIMDs per CU: 4 2025-08-14T21:22:29.4838691Z Shader Engines: 8 2025-08-14T21:22:29.4839437Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.4840522Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.4841348Z Coherent Host Access: FALSE 2025-08-14T21:22:29.4842103Z Memory Properties: 2025-08-14T21:22:29.4842745Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.4843559Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.4844376Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.4845128Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4845816Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4846394Z x 1024(0x400) 2025-08-14T21:22:29.4847357Z y 1024(0x400) 2025-08-14T21:22:29.4847971Z z 1024(0x400) 2025-08-14T21:22:29.4848630Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.4849388Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.4850132Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4850802Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4851344Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4851965Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4852567Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4853271Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.4854086Z Packet Processor uCode:: 92 2025-08-14T21:22:29.4854872Z SDMA engine uCode:: 9 2025-08-14T21:22:29.4855627Z IOMMU Support:: None 2025-08-14T21:22:29.4856273Z Pool Info: 2025-08-14T21:22:29.4856752Z Pool 1 2025-08-14T21:22:29.4857375Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.4858124Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4858845Z Allocatable: TRUE 2025-08-14T21:22:29.4859663Z Alloc Granule: 4KB 2025-08-14T21:22:29.4860618Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4861462Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4862238Z Accessible by all: FALSE 2025-08-14T21:22:29.4862906Z Pool 2 2025-08-14T21:22:29.4863526Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.4864245Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4864965Z Allocatable: TRUE 2025-08-14T21:22:29.4866076Z Alloc Granule: 4KB 2025-08-14T21:22:29.4866877Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4867670Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4868450Z Accessible by all: FALSE 2025-08-14T21:22:29.4869099Z Pool 3 2025-08-14T21:22:29.4869681Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.4870391Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4871095Z Allocatable: TRUE 2025-08-14T21:22:29.4871852Z Alloc Granule: 4KB 2025-08-14T21:22:29.4872637Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4873425Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4874212Z Accessible by all: FALSE 2025-08-14T21:22:29.4874869Z Pool 4 2025-08-14T21:22:29.4875437Z Segment: GROUP 2025-08-14T21:22:29.4876102Z Size: 64(0x40) KB 2025-08-14T21:22:29.4876808Z Allocatable: FALSE 2025-08-14T21:22:29.4877564Z Alloc Granule: 0KB 2025-08-14T21:22:29.4878349Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.4879165Z Alloc Alignment: 0KB 2025-08-14T21:22:29.4880379Z Accessible by all: FALSE 2025-08-14T21:22:29.4881098Z ISA Info: 2025-08-14T21:22:29.4881593Z ISA 1 2025-08-14T21:22:29.4882360Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.4883362Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.4884264Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.4885051Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4885865Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4886596Z Fast f16: TRUE 2025-08-14T21:22:29.4887338Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4888057Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4888677Z x 1024(0x400) 2025-08-14T21:22:29.4889319Z y 1024(0x400) 2025-08-14T21:22:29.4889928Z z 1024(0x400) 2025-08-14T21:22:29.4890599Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4891290Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4891862Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4892485Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4893098Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4893795Z FBarrier Max Size: 32 2025-08-14T21:22:29.4894459Z ******* 2025-08-14T21:22:29.4894934Z Agent 5 2025-08-14T21:22:29.4895362Z ******* 2025-08-14T21:22:29.4895877Z Name: gfx90a 2025-08-14T21:22:29.4896576Z Uuid: GPU-5dd3a7090d90a096 2025-08-14T21:22:29.4897330Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.4898101Z Vendor Name: AMD 2025-08-14T21:22:29.4899195Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.4900065Z Profile: BASE_PROFILE 2025-08-14T21:22:29.4900935Z Float Round Mode: NEAR 2025-08-14T21:22:29.4901689Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.4902415Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.4903122Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.4903831Z Queue Type: MULTI 2025-08-14T21:22:29.4904505Z Node: 4 2025-08-14T21:22:29.4905191Z Device Type: GPU 2025-08-14T21:22:29.4905845Z Cache Info: 2025-08-14T21:22:29.4906364Z L1: 16(0x10) KB 2025-08-14T21:22:29.4906998Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.4907633Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.4908321Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.4909052Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.4909798Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.4910484Z BDFID: 4352 2025-08-14T21:22:29.4911179Z Internal Node ID: 4 2025-08-14T21:22:29.4911900Z Compute Unit: 104 2025-08-14T21:22:29.4912930Z SIMDs per CU: 4 2025-08-14T21:22:29.4913662Z Shader Engines: 8 2025-08-14T21:22:29.4914394Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.4915156Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.4915945Z Coherent Host Access: FALSE 2025-08-14T21:22:29.4916626Z Memory Properties: 2025-08-14T21:22:29.4917158Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.4917842Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.4918589Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.4919346Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4920032Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4920718Z x 1024(0x400) 2025-08-14T21:22:29.4921321Z y 1024(0x400) 2025-08-14T21:22:29.4921908Z z 1024(0x400) 2025-08-14T21:22:29.4922567Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.4923320Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.4924083Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4924756Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4925293Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4925910Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4926511Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4927216Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.4928025Z Packet Processor uCode:: 92 2025-08-14T21:22:29.4928803Z SDMA engine uCode:: 9 2025-08-14T21:22:29.4929582Z IOMMU Support:: None 2025-08-14T21:22:29.4930243Z Pool Info: 2025-08-14T21:22:29.4930728Z Pool 1 2025-08-14T21:22:29.4931356Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.4932471Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4933198Z Allocatable: TRUE 2025-08-14T21:22:29.4933955Z Alloc Granule: 4KB 2025-08-14T21:22:29.4934751Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4935564Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4936349Z Accessible by all: FALSE 2025-08-14T21:22:29.4937017Z Pool 2 2025-08-14T21:22:29.4937622Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.4938379Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4939128Z Allocatable: TRUE 2025-08-14T21:22:29.4939892Z Alloc Granule: 4KB 2025-08-14T21:22:29.4940707Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4941524Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4942302Z Accessible by all: FALSE 2025-08-14T21:22:29.4942968Z Pool 3 2025-08-14T21:22:29.4943558Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.4944273Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.4970331Z Allocatable: TRUE 2025-08-14T21:22:29.4971074Z Alloc Granule: 4KB 2025-08-14T21:22:29.4972193Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.4972923Z Alloc Alignment: 4KB 2025-08-14T21:22:29.4973599Z Accessible by all: FALSE 2025-08-14T21:22:29.4974203Z Pool 4 2025-08-14T21:22:29.4974718Z Segment: GROUP 2025-08-14T21:22:29.4975295Z Size: 64(0x40) KB 2025-08-14T21:22:29.4975897Z Allocatable: FALSE 2025-08-14T21:22:29.4976530Z Alloc Granule: 0KB 2025-08-14T21:22:29.4977250Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.4977982Z Alloc Alignment: 0KB 2025-08-14T21:22:29.4978654Z Accessible by all: FALSE 2025-08-14T21:22:29.4979239Z ISA Info: 2025-08-14T21:22:29.4979655Z ISA 1 2025-08-14T21:22:29.4980190Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.4980889Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.4981561Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.4982226Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4982895Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.4983514Z Fast f16: TRUE 2025-08-14T21:22:29.4984129Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.4984720Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.4985247Z x 1024(0x400) 2025-08-14T21:22:29.4985793Z y 1024(0x400) 2025-08-14T21:22:29.4986299Z z 1024(0x400) 2025-08-14T21:22:29.4986863Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.4987757Z Grid Max Size per Dimension: 2025-08-14T21:22:29.4988238Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.4988776Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.4989310Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.4989895Z FBarrier Max Size: 32 2025-08-14T21:22:29.4990446Z ******* 2025-08-14T21:22:29.4990845Z Agent 6 2025-08-14T21:22:29.4991228Z ******* 2025-08-14T21:22:29.4991674Z Name: gfx90a 2025-08-14T21:22:29.4992263Z Uuid: GPU-2b7505fff509489f 2025-08-14T21:22:29.4992919Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.4993583Z Vendor Name: AMD 2025-08-14T21:22:29.4994214Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.4994858Z Profile: BASE_PROFILE 2025-08-14T21:22:29.4995471Z Float Round Mode: NEAR 2025-08-14T21:22:29.4996086Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.4996696Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.4997306Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.4997903Z Queue Type: MULTI 2025-08-14T21:22:29.4998474Z Node: 5 2025-08-14T21:22:29.4999047Z Device Type: GPU 2025-08-14T21:22:29.4999833Z Cache Info: 2025-08-14T21:22:29.5000415Z L1: 16(0x10) KB 2025-08-14T21:22:29.5000959Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.5001529Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.5002140Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.5002762Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.5003387Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.5003959Z BDFID: 5120 2025-08-14T21:22:29.5004548Z Internal Node ID: 5 2025-08-14T21:22:29.5005167Z Compute Unit: 104 2025-08-14T21:22:29.5005768Z SIMDs per CU: 4 2025-08-14T21:22:29.5006387Z Shader Engines: 8 2025-08-14T21:22:29.5007013Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.5007654Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.5008322Z Coherent Host Access: FALSE 2025-08-14T21:22:29.5008901Z Memory Properties: 2025-08-14T21:22:29.5009356Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.5009937Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.5010594Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.5011230Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5011815Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5012302Z x 1024(0x400) 2025-08-14T21:22:29.5012808Z y 1024(0x400) 2025-08-14T21:22:29.5013314Z z 1024(0x400) 2025-08-14T21:22:29.5013883Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.5014508Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.5015484Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5016037Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5016489Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5017003Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5017512Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5018104Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.5018798Z Packet Processor uCode:: 92 2025-08-14T21:22:29.5019458Z SDMA engine uCode:: 9 2025-08-14T21:22:29.5020087Z IOMMU Support:: None 2025-08-14T21:22:29.5020640Z Pool Info: 2025-08-14T21:22:29.5021055Z Pool 1 2025-08-14T21:22:29.5021576Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.5022212Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5022815Z Allocatable: TRUE 2025-08-14T21:22:29.5023451Z Alloc Granule: 4KB 2025-08-14T21:22:29.5024125Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5024807Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5025459Z Accessible by all: FALSE 2025-08-14T21:22:29.5026017Z Pool 2 2025-08-14T21:22:29.5026524Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.5027426Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5028065Z Allocatable: TRUE 2025-08-14T21:22:29.5028697Z Alloc Granule: 4KB 2025-08-14T21:22:29.5029377Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5030059Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5030712Z Accessible by all: FALSE 2025-08-14T21:22:29.5031263Z Pool 3 2025-08-14T21:22:29.5031764Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.5032354Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5032945Z Allocatable: TRUE 2025-08-14T21:22:29.5033580Z Alloc Granule: 4KB 2025-08-14T21:22:29.5034259Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5034923Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5035575Z Accessible by all: FALSE 2025-08-14T21:22:29.5036126Z Pool 4 2025-08-14T21:22:29.5036610Z Segment: GROUP 2025-08-14T21:22:29.5037170Z Size: 64(0x40) KB 2025-08-14T21:22:29.5037757Z Allocatable: FALSE 2025-08-14T21:22:29.5038392Z Alloc Granule: 0KB 2025-08-14T21:22:29.5039057Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.5039720Z Alloc Alignment: 0KB 2025-08-14T21:22:29.5040478Z Accessible by all: FALSE 2025-08-14T21:22:29.5041041Z ISA Info: 2025-08-14T21:22:29.5041453Z ISA 1 2025-08-14T21:22:29.5041969Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.5042657Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.5043627Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.5044281Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5044945Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5045580Z Fast f16: TRUE 2025-08-14T21:22:29.5046244Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5046850Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5047368Z x 1024(0x400) 2025-08-14T21:22:29.5047898Z y 1024(0x400) 2025-08-14T21:22:29.5048417Z z 1024(0x400) 2025-08-14T21:22:29.5049005Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5049579Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5050064Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5050615Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5051142Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5051730Z FBarrier Max Size: 32 2025-08-14T21:22:29.5052272Z ******* 2025-08-14T21:22:29.5052663Z Agent 7 2025-08-14T21:22:29.5053035Z ******* 2025-08-14T21:22:29.5053478Z Name: gfx90a 2025-08-14T21:22:29.5054061Z Uuid: GPU-2d3a0c893d332691 2025-08-14T21:22:29.5055152Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.5055825Z Vendor Name: AMD 2025-08-14T21:22:29.5056445Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.5057049Z Profile: BASE_PROFILE 2025-08-14T21:22:29.5057687Z Float Round Mode: NEAR 2025-08-14T21:22:29.5058327Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.5058947Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.5059552Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.5060162Z Queue Type: MULTI 2025-08-14T21:22:29.5060736Z Node: 6 2025-08-14T21:22:29.5061307Z Device Type: GPU 2025-08-14T21:22:29.5061844Z Cache Info: 2025-08-14T21:22:29.5062300Z L1: 16(0x10) KB 2025-08-14T21:22:29.5062847Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.5063385Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.5063991Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.5064623Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.5065243Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.5065822Z BDFID: 44544 2025-08-14T21:22:29.5066405Z Internal Node ID: 6 2025-08-14T21:22:29.5067025Z Compute Unit: 104 2025-08-14T21:22:29.5067617Z SIMDs per CU: 4 2025-08-14T21:22:29.5068223Z Shader Engines: 8 2025-08-14T21:22:29.5068859Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.5069509Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.5070166Z Coherent Host Access: FALSE 2025-08-14T21:22:29.5071022Z Memory Properties: 2025-08-14T21:22:29.5071475Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.5072055Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.5072700Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.5073343Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5073922Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5074414Z x 1024(0x400) 2025-08-14T21:22:29.5074924Z y 1024(0x400) 2025-08-14T21:22:29.5075411Z z 1024(0x400) 2025-08-14T21:22:29.5075972Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.5076605Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.5077229Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5077798Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5078246Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5078764Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5079272Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5079866Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.5080818Z Packet Processor uCode:: 92 2025-08-14T21:22:29.5081477Z SDMA engine uCode:: 9 2025-08-14T21:22:29.5082106Z IOMMU Support:: None 2025-08-14T21:22:29.5082655Z Pool Info: 2025-08-14T21:22:29.5083360Z Pool 1 2025-08-14T21:22:29.5083883Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.5084505Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5085124Z Allocatable: TRUE 2025-08-14T21:22:29.5085762Z Alloc Granule: 4KB 2025-08-14T21:22:29.5086428Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5087106Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5087756Z Accessible by all: FALSE 2025-08-14T21:22:29.5088314Z Pool 2 2025-08-14T21:22:29.5088827Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.5089442Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5090044Z Allocatable: TRUE 2025-08-14T21:22:29.5090676Z Alloc Granule: 4KB 2025-08-14T21:22:29.5091332Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5092005Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5092645Z Accessible by all: FALSE 2025-08-14T21:22:29.5093196Z Pool 3 2025-08-14T21:22:29.5093689Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.5094274Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5094872Z Allocatable: TRUE 2025-08-14T21:22:29.5095507Z Alloc Granule: 4KB 2025-08-14T21:22:29.5096163Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5096832Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5097484Z Accessible by all: FALSE 2025-08-14T21:22:29.5098040Z Pool 4 2025-08-14T21:22:29.5098505Z Segment: GROUP 2025-08-14T21:22:29.5099392Z Size: 64(0x40) KB 2025-08-14T21:22:29.5099988Z Allocatable: FALSE 2025-08-14T21:22:29.5100610Z Alloc Granule: 0KB 2025-08-14T21:22:29.5101267Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.5101932Z Alloc Alignment: 0KB 2025-08-14T21:22:29.5102583Z Accessible by all: FALSE 2025-08-14T21:22:29.5103146Z ISA Info: 2025-08-14T21:22:29.5103543Z ISA 1 2025-08-14T21:22:29.5104072Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.5104747Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.5105405Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.5106080Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5106749Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5107371Z Fast f16: TRUE 2025-08-14T21:22:29.5107992Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5108580Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5109084Z x 1024(0x400) 2025-08-14T21:22:29.5109610Z y 1024(0x400) 2025-08-14T21:22:29.5110141Z z 1024(0x400) 2025-08-14T21:22:29.5110965Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5111558Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5112033Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5112564Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5113085Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5113676Z FBarrier Max Size: 32 2025-08-14T21:22:29.5114221Z ******* 2025-08-14T21:22:29.5114603Z Agent 8 2025-08-14T21:22:29.5114976Z ******* 2025-08-14T21:22:29.5115398Z Name: gfx90a 2025-08-14T21:22:29.5115984Z Uuid: GPU-8f10eb83a29cc55d 2025-08-14T21:22:29.5116617Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.5117272Z Vendor Name: AMD 2025-08-14T21:22:29.5117883Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.5118494Z Profile: BASE_PROFILE 2025-08-14T21:22:29.5119126Z Float Round Mode: NEAR 2025-08-14T21:22:29.5119753Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.5120482Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.5121082Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.5121678Z Queue Type: MULTI 2025-08-14T21:22:29.5122244Z Node: 7 2025-08-14T21:22:29.5122812Z Device Type: GPU 2025-08-14T21:22:29.5123349Z Cache Info: 2025-08-14T21:22:29.5123779Z L1: 16(0x10) KB 2025-08-14T21:22:29.5124323Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.5124867Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.5125465Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.5126459Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.5127088Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.5127669Z BDFID: 45824 2025-08-14T21:22:29.5128252Z Internal Node ID: 7 2025-08-14T21:22:29.5128864Z Compute Unit: 104 2025-08-14T21:22:29.5129463Z SIMDs per CU: 4 2025-08-14T21:22:29.5130078Z Shader Engines: 8 2025-08-14T21:22:29.5130704Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.5131366Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.5132031Z Coherent Host Access: FALSE 2025-08-14T21:22:29.5132609Z Memory Properties: 2025-08-14T21:22:29.5133075Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.5133660Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.5134298Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.5134946Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5135538Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5136016Z x 1024(0x400) 2025-08-14T21:22:29.5136525Z y 1024(0x400) 2025-08-14T21:22:29.5137034Z z 1024(0x400) 2025-08-14T21:22:29.5137594Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.5138519Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.5139164Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5139729Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5140192Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5140721Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5141233Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5141831Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.5142528Z Packet Processor uCode:: 92 2025-08-14T21:22:29.5143188Z SDMA engine uCode:: 9 2025-08-14T21:22:29.5143819Z IOMMU Support:: None 2025-08-14T21:22:29.5144383Z Pool Info: 2025-08-14T21:22:29.5144787Z Pool 1 2025-08-14T21:22:29.5145312Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.5145946Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5146568Z Allocatable: TRUE 2025-08-14T21:22:29.5147270Z Alloc Granule: 4KB 2025-08-14T21:22:29.5147966Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5148660Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5149323Z Accessible by all: FALSE 2025-08-14T21:22:29.5149899Z Pool 2 2025-08-14T21:22:29.5150437Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.5151054Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5151667Z Allocatable: TRUE 2025-08-14T21:22:29.5152321Z Alloc Granule: 4KB 2025-08-14T21:22:29.5152998Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5153681Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5154607Z Accessible by all: FALSE 2025-08-14T21:22:29.5155173Z Pool 3 2025-08-14T21:22:29.5155675Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.5156265Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5156870Z Allocatable: TRUE 2025-08-14T21:22:29.5157504Z Alloc Granule: 4KB 2025-08-14T21:22:29.5158167Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5158838Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5159490Z Accessible by all: FALSE 2025-08-14T21:22:29.5160046Z Pool 4 2025-08-14T21:22:29.5160623Z Segment: GROUP 2025-08-14T21:22:29.5161207Z Size: 64(0x40) KB 2025-08-14T21:22:29.5161797Z Allocatable: FALSE 2025-08-14T21:22:29.5162429Z Alloc Granule: 0KB 2025-08-14T21:22:29.5163089Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.5163757Z Alloc Alignment: 0KB 2025-08-14T21:22:29.5164402Z Accessible by all: FALSE 2025-08-14T21:22:29.5164960Z ISA Info: 2025-08-14T21:22:29.5165353Z ISA 1 2025-08-14T21:22:29.5165882Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.5166857Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.5167537Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.5168192Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5168875Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5169494Z Fast f16: TRUE 2025-08-14T21:22:29.5170117Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5170717Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5171231Z x 1024(0x400) 2025-08-14T21:22:29.5171751Z y 1024(0x400) 2025-08-14T21:22:29.5172257Z z 1024(0x400) 2025-08-14T21:22:29.5172832Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5173400Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5173873Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5174398Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5174923Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5175510Z FBarrier Max Size: 32 2025-08-14T21:22:29.5176052Z ******* 2025-08-14T21:22:29.5176435Z Agent 9 2025-08-14T21:22:29.5176812Z ******* 2025-08-14T21:22:29.5177246Z Name: gfx90a 2025-08-14T21:22:29.5177823Z Uuid: GPU-2987fc4d26ca2e7a 2025-08-14T21:22:29.5178453Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.5179094Z Vendor Name: AMD 2025-08-14T21:22:29.5179713Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.5180319Z Profile: BASE_PROFILE 2025-08-14T21:22:29.5180946Z Float Round Mode: NEAR 2025-08-14T21:22:29.5181881Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.5182494Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.5183097Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.5183694Z Queue Type: MULTI 2025-08-14T21:22:29.5184260Z Node: 8 2025-08-14T21:22:29.5184830Z Device Type: GPU 2025-08-14T21:22:29.5185368Z Cache Info: 2025-08-14T21:22:29.5185800Z L1: 16(0x10) KB 2025-08-14T21:22:29.5186327Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.5186868Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.5187458Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.5188078Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.5188715Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.5189294Z BDFID: 36352 2025-08-14T21:22:29.5189883Z Internal Node ID: 8 2025-08-14T21:22:29.5190502Z Compute Unit: 104 2025-08-14T21:22:29.5191094Z SIMDs per CU: 4 2025-08-14T21:22:29.5191700Z Shader Engines: 8 2025-08-14T21:22:29.5192340Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.5192984Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.5193894Z Coherent Host Access: FALSE 2025-08-14T21:22:29.5194476Z Memory Properties: 2025-08-14T21:22:29.5194926Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.5195520Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.5196150Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.5196784Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5197367Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5197853Z x 1024(0x400) 2025-08-14T21:22:29.5198354Z y 1024(0x400) 2025-08-14T21:22:29.5198847Z z 1024(0x400) 2025-08-14T21:22:29.5199404Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.5200033Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.5200890Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5201450Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5201933Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5202486Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5203008Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5203610Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.5204309Z Packet Processor uCode:: 92 2025-08-14T21:22:29.5204979Z SDMA engine uCode:: 9 2025-08-14T21:22:29.5205620Z IOMMU Support:: None 2025-08-14T21:22:29.5206182Z Pool Info: 2025-08-14T21:22:29.5206605Z Pool 1 2025-08-14T21:22:29.5207125Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.5207754Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5208373Z Allocatable: TRUE 2025-08-14T21:22:29.5209013Z Alloc Granule: 4KB 2025-08-14T21:22:29.5210016Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5210701Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5211362Z Accessible by all: FALSE 2025-08-14T21:22:29.5211925Z Pool 2 2025-08-14T21:22:29.5212433Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.5213037Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5213637Z Allocatable: TRUE 2025-08-14T21:22:29.5214281Z Alloc Granule: 4KB 2025-08-14T21:22:29.5214968Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5215658Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5216311Z Accessible by all: FALSE 2025-08-14T21:22:29.5216887Z Pool 3 2025-08-14T21:22:29.5217387Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.5217980Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5218575Z Allocatable: TRUE 2025-08-14T21:22:29.5219209Z Alloc Granule: 4KB 2025-08-14T21:22:29.5219876Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5220548Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5221204Z Accessible by all: FALSE 2025-08-14T21:22:29.5222084Z Pool 4 2025-08-14T21:22:29.5222594Z Segment: GROUP 2025-08-14T21:22:29.5223190Z Size: 64(0x40) KB 2025-08-14T21:22:29.5223797Z Allocatable: FALSE 2025-08-14T21:22:29.5224430Z Alloc Granule: 0KB 2025-08-14T21:22:29.5225095Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.5225762Z Alloc Alignment: 0KB 2025-08-14T21:22:29.5226407Z Accessible by all: FALSE 2025-08-14T21:22:29.5226967Z ISA Info: 2025-08-14T21:22:29.5227378Z ISA 1 2025-08-14T21:22:29.5227895Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.5228575Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.5229235Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.5229887Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5230556Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5231196Z Fast f16: TRUE 2025-08-14T21:22:29.5231814Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5232408Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5232921Z x 1024(0x400) 2025-08-14T21:22:29.5233441Z y 1024(0x400) 2025-08-14T21:22:29.5233949Z z 1024(0x400) 2025-08-14T21:22:29.5234513Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5235078Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5235560Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5236077Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5236590Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5237436Z FBarrier Max Size: 32 2025-08-14T21:22:29.5237971Z ******* 2025-08-14T21:22:29.5238354Z Agent 10 2025-08-14T21:22:29.5238720Z ******* 2025-08-14T21:22:29.5239151Z Name: gfx90a 2025-08-14T21:22:29.5239723Z Uuid: GPU-a88b5a95f52381e5 2025-08-14T21:22:29.5240533Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.5241183Z Vendor Name: AMD 2025-08-14T21:22:29.5241781Z Feature: KERNEL_DISPATCH 2025-08-14T21:22:29.5242392Z Profile: BASE_PROFILE 2025-08-14T21:22:29.5243006Z Float Round Mode: NEAR 2025-08-14T21:22:29.5243629Z Max Queue Number: 128(0x80) 2025-08-14T21:22:29.5244245Z Queue Min Size: 64(0x40) 2025-08-14T21:22:29.5244837Z Queue Max Size: 131072(0x20000) 2025-08-14T21:22:29.5245429Z Queue Type: MULTI 2025-08-14T21:22:29.5246008Z Node: 9 2025-08-14T21:22:29.5246613Z Device Type: GPU 2025-08-14T21:22:29.5247159Z Cache Info: 2025-08-14T21:22:29.5247605Z L1: 16(0x10) KB 2025-08-14T21:22:29.5248131Z L2: 8192(0x2000) KB 2025-08-14T21:22:29.5248966Z Chip ID: 29708(0x740c) 2025-08-14T21:22:29.5249577Z ASIC Revision: 1(0x1) 2025-08-14T21:22:29.5250202Z Cacheline Size: 128(0x80) 2025-08-14T21:22:29.5250832Z Max Clock Freq. (MHz): 1700 2025-08-14T21:22:29.5251408Z BDFID: 37632 2025-08-14T21:22:29.5252000Z Internal Node ID: 9 2025-08-14T21:22:29.5252609Z Compute Unit: 104 2025-08-14T21:22:29.5253196Z SIMDs per CU: 4 2025-08-14T21:22:29.5253795Z Shader Engines: 8 2025-08-14T21:22:29.5254425Z Shader Arrs. per Eng.: 1 2025-08-14T21:22:29.5255066Z WatchPts on Addr. Ranges:4 2025-08-14T21:22:29.5255718Z Coherent Host Access: FALSE 2025-08-14T21:22:29.5256295Z Memory Properties: 2025-08-14T21:22:29.5256747Z Features: KERNEL_DISPATCH 2025-08-14T21:22:29.5257322Z Fast F16 Operation: TRUE 2025-08-14T21:22:29.5257968Z Wavefront Size: 64(0x40) 2025-08-14T21:22:29.5258612Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5259188Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5259672Z x 1024(0x400) 2025-08-14T21:22:29.5260182Z y 1024(0x400) 2025-08-14T21:22:29.5260678Z z 1024(0x400) 2025-08-14T21:22:29.5261242Z Max Waves Per CU: 32(0x20) 2025-08-14T21:22:29.5261872Z Max Work-item Per CU: 2048(0x800) 2025-08-14T21:22:29.5262503Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5263052Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5263490Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5264007Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5264845Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5265453Z Max fbarriers/Workgrp: 32 2025-08-14T21:22:29.5266155Z Packet Processor uCode:: 92 2025-08-14T21:22:29.5266820Z SDMA engine uCode:: 9 2025-08-14T21:22:29.5267458Z IOMMU Support:: None 2025-08-14T21:22:29.5268009Z Pool Info: 2025-08-14T21:22:29.5268416Z Pool 1 2025-08-14T21:22:29.5268933Z Segment: GLOBAL; FLAGS: COARSE GRAINED 2025-08-14T21:22:29.5269567Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5270197Z Allocatable: TRUE 2025-08-14T21:22:29.5270839Z Alloc Granule: 4KB 2025-08-14T21:22:29.5271503Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5272189Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5272844Z Accessible by all: FALSE 2025-08-14T21:22:29.5273401Z Pool 2 2025-08-14T21:22:29.5273909Z Segment: GLOBAL; FLAGS: EXTENDED FINE GRAINED 2025-08-14T21:22:29.5274534Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5275145Z Allocatable: TRUE 2025-08-14T21:22:29.5275789Z Alloc Granule: 4KB 2025-08-14T21:22:29.5276460Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5277375Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5278038Z Accessible by all: FALSE 2025-08-14T21:22:29.5278602Z Pool 3 2025-08-14T21:22:29.5279116Z Segment: GLOBAL; FLAGS: FINE GRAINED 2025-08-14T21:22:29.5279711Z Size: 67092480(0x3ffc000) KB 2025-08-14T21:22:29.5280407Z Allocatable: TRUE 2025-08-14T21:22:29.5281044Z Alloc Granule: 4KB 2025-08-14T21:22:29.5281709Z Alloc Recommended Granule:2048KB 2025-08-14T21:22:29.5282382Z Alloc Alignment: 4KB 2025-08-14T21:22:29.5283039Z Accessible by all: FALSE 2025-08-14T21:22:29.5283601Z Pool 4 2025-08-14T21:22:29.5284092Z Segment: GROUP 2025-08-14T21:22:29.5284669Z Size: 64(0x40) KB 2025-08-14T21:22:29.5285268Z Allocatable: FALSE 2025-08-14T21:22:29.5285902Z Alloc Granule: 0KB 2025-08-14T21:22:29.5286564Z Alloc Recommended Granule:0KB 2025-08-14T21:22:29.5287233Z Alloc Alignment: 0KB 2025-08-14T21:22:29.5287888Z Accessible by all: FALSE 2025-08-14T21:22:29.5288453Z ISA Info: 2025-08-14T21:22:29.5288853Z ISA 1 2025-08-14T21:22:29.5289378Z Name: amdgcn-amd-amdhsa--gfx90a:sramecc+:xnack- 2025-08-14T21:22:29.5290069Z Machine Models: HSA_MACHINE_MODEL_LARGE 2025-08-14T21:22:29.5290742Z Profiles: HSA_PROFILE_BASE 2025-08-14T21:22:29.5291405Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5292081Z Default Rounding Mode: NEAR 2025-08-14T21:22:29.5293023Z Fast f16: TRUE 2025-08-14T21:22:29.5293645Z Workgroup Max Size: 1024(0x400) 2025-08-14T21:22:29.5294242Z Workgroup Max Size per Dimension: 2025-08-14T21:22:29.5294756Z x 1024(0x400) 2025-08-14T21:22:29.5295289Z y 1024(0x400) 2025-08-14T21:22:29.5295808Z z 1024(0x400) 2025-08-14T21:22:29.5296379Z Grid Max Size: 4294967295(0xffffffff) 2025-08-14T21:22:29.5296949Z Grid Max Size per Dimension: 2025-08-14T21:22:29.5297426Z x 4294967295(0xffffffff) 2025-08-14T21:22:29.5297956Z y 4294967295(0xffffffff) 2025-08-14T21:22:29.5298473Z z 4294967295(0xffffffff) 2025-08-14T21:22:29.5299055Z FBarrier Max Size: 32 2025-08-14T21:22:29.5299619Z *** Done *** 2025-08-14T21:22:29.5300024Z + rocminfo 2025-08-14T21:22:29.5300390Z + grep -E 'Name:.*\sgfx|Marketing' 2025-08-14T21:22:29.7147141Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.7147956Z Marketing Name: AMD EPYC 7713 64-Core Processor 2025-08-14T21:22:29.7148578Z Name: gfx90a 2025-08-14T21:22:29.7149147Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7149709Z Name: gfx90a 2025-08-14T21:22:29.7150264Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7151224Z Name: gfx90a 2025-08-14T21:22:29.7151849Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7152417Z Name: gfx90a 2025-08-14T21:22:29.7152988Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7153553Z Name: gfx90a 2025-08-14T21:22:29.7154138Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7154838Z Name: gfx90a 2025-08-14T21:22:29.7155536Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7156248Z Name: gfx90a 2025-08-14T21:22:29.7156954Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7157658Z Name: gfx90a 2025-08-14T21:22:29.7158363Z Marketing Name: AMD Instinct MI250X/MI250 2025-08-14T21:22:29.7346857Z + MAYBE_ROCM=rocm/ 2025-08-14T21:22:29.7347517Z + [[ linux-jammy-rocm-py3.10 == *xpu* ]] 2025-08-14T21:22:29.7348243Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]] 2025-08-14T21:22:29.7348922Z + pip_install ninja==1.10.2 2025-08-14T21:22:29.7349620Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-08-14T21:22:29.7350559Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-08-14T21:22:30.2057501Z Collecting ninja==1.10.2 2025-08-14T21:22:30.2236609Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-08-14T21:22:30.2321147Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-08-14T21:22:30.5401380Z Installing collected packages: ninja 2025-08-14T21:22:30.5402084Z Attempting uninstall: ninja 2025-08-14T21:22:30.5408072Z Found existing installation: ninja 1.11.1.3 2025-08-14T21:22:30.5427071Z Uninstalling ninja-1.11.1.3: 2025-08-14T21:22:30.5510416Z Successfully uninstalled ninja-1.11.1.3 2025-08-14T21:22:30.5799907Z Successfully installed ninja-1.10.2 2025-08-14T21:22:30.6378313Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:22:30.6382883Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/rocm/llvm/bin:/opt/rocm/opencl/bin:/opt/rocm/hip/bin:/opt/rocm/hcc/bin:/opt/rocm/bin:/opt/conda/envs/py_3.10/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:22:30.6385059Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]] 2025-08-14T21:22:30.6385709Z + [[ linux-jammy-rocm-py3.10 == *asan* ]] 2025-08-14T21:22:30.6386325Z + [[ linux-jammy-rocm-py3.10 == *-debug* ]] 2025-08-14T21:22:30.6386939Z + [[ linux-jammy-rocm-py3.10 != *-bazel-* ]] 2025-08-14T21:22:30.6387867Z + echo 'We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass' 2025-08-14T21:22:30.6388989Z We are not in debug mode: linux-jammy-rocm-py3.10. Expect the assertion to pass 2025-08-14T21:22:30.6389801Z + cd test 2025-08-14T21:22:30.6390419Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-08-14T21:22:32.1162220Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-08-14T21:22:32.1163114Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-08-14T21:22:32.1163935Z + [[ distributed == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-08-14T21:22:32.1172091Z + DYNAMO_BENCHMARK_FLAGS=() 2025-08-14T21:22:32.1172902Z + [[ distributed == *pr_time_benchmarks* ]] 2025-08-14T21:22:32.1173657Z + [[ distributed == *dynamo_eager* ]] 2025-08-14T21:22:32.1174304Z + [[ distributed == *aot_eager* ]] 2025-08-14T21:22:32.1174848Z + [[ distributed == *aot_inductor* ]] 2025-08-14T21:22:32.1175426Z + [[ distributed == *max_autotune_inductor* ]] 2025-08-14T21:22:32.1176591Z + [[ distributed == *inductor* ]] 2025-08-14T21:22:32.1177276Z + [[ distributed == *dynamic* ]] 2025-08-14T21:22:32.1177899Z + [[ distributed == *cpu* ]] 2025-08-14T21:22:32.1178534Z + DYNAMO_BENCHMARK_FLAGS+=(--device cuda) 2025-08-14T21:22:32.1220157Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]] 2025-08-14T21:22:32.1220907Z + [[ linux-jammy-rocm-py3.10 == *-bazel-* ]] 2025-08-14T21:22:32.1229192Z + cd test 2025-08-14T21:22:32.1229867Z + python -c 'import torch; print(torch.__config__.show())' 2025-08-14T21:22:33.5344888Z PyTorch built with: 2025-08-14T21:22:33.5345405Z - GCC 11.4 2025-08-14T21:22:33.5345856Z - C++ Version: 201703 2025-08-14T21:22:33.5346931Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:22:33.5348279Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:22:33.5349110Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:22:33.5349786Z - LAPACK is enabled (usually provided by MKL) 2025-08-14T21:22:33.5350394Z - NNPACK is enabled 2025-08-14T21:22:33.5350897Z - CPU capability usage: AVX2 2025-08-14T21:22:33.5351414Z - HIP Runtime 6.4.43484 2025-08-14T21:22:33.5351885Z - MIOpen 3.4.0 2025-08-14T21:22:33.5352290Z - Magma 2.7.2 2025-08-14T21:22:33.5360699Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=1fc683cf17c8c673044538d10266c00f92987be2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=ON, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-08-14T21:22:33.5369793Z 2025-08-14T21:22:33.8081627Z + cd test 2025-08-14T21:22:33.8082366Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-08-14T21:22:35.0110995Z ATen/Parallel: 2025-08-14T21:22:35.0111635Z at::get_num_threads() : 128 2025-08-14T21:22:35.0112221Z at::get_num_interop_threads() : 128 2025-08-14T21:22:35.0112795Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:22:35.0113340Z omp_get_max_threads() : 128 2025-08-14T21:22:35.0114385Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:22:35.0115506Z mkl_get_max_threads() : 128 2025-08-14T21:22:35.0116205Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:22:35.0117015Z std::thread::hardware_concurrency() : 128 2025-08-14T21:22:35.0117616Z Environment variables: 2025-08-14T21:22:35.0118082Z OMP_NUM_THREADS : [not set] 2025-08-14T21:22:35.0118558Z MKL_NUM_THREADS : [not set] 2025-08-14T21:22:35.0119052Z ATen parallel backend: OpenMP 2025-08-14T21:22:35.0119383Z 2025-08-14T21:22:35.2581591Z + [[ distributed == *numpy_2* ]] 2025-08-14T21:22:35.2582314Z + [[ linux-jammy-rocm-py3.10 == *aarch64* ]] 2025-08-14T21:22:35.2582984Z + [[ distributed == *backward* ]] 2025-08-14T21:22:35.2583627Z + [[ distributed == *xla* ]] 2025-08-14T21:22:35.2584224Z + [[ distributed == *executorch* ]] 2025-08-14T21:22:35.2584877Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]] 2025-08-14T21:22:35.2585597Z + [[ linux-jammy-rocm-py3.10 == *libtorch* ]] 2025-08-14T21:22:35.2586288Z + [[ distributed == distributed ]] 2025-08-14T21:22:35.2587557Z + test_distributed 2025-08-14T21:22:35.2588070Z + echo 'Testing distributed python tests' 2025-08-14T21:22:35.2588689Z Testing distributed python tests 2025-08-14T21:22:35.2589413Z + python test/run_test.py --distributed-tests --shard 2 3 --verbose 2025-08-14T21:22:38.0057773Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:22:38.0060520Z import pkg_resources 2025-08-14T21:22:38.7113871Z Excluding distributed/rpc/test_faulty_agent on ROCm 2025-08-14T21:22:38.7114775Z Excluding distributed/rpc/test_tensorpipe_agent on ROCm 2025-08-14T21:22:38.7115550Z Excluding distributed/rpc/test_share_memory on ROCm 2025-08-14T21:22:38.7116356Z Excluding distributed/rpc/cuda/test_tensorpipe_agent on ROCm 2025-08-14T21:22:40.0571298Z Downloading https://ossci-metrics.s3.amazonaws.com/disabled-tests-condensed.json to /var/lib/jenkins/pytorch/test/.pytorch-disabled-tests.json 2025-08-14T21:22:40.3302343Z Ignoring disabled issues: [''] 2025-08-14T21:22:40.3525261Z Found test times from artifacts 2025-08-14T21:22:40.4027154Z Found test times from artifacts 2025-08-14T21:22:40.4038726Z Running all tests 2025-08-14T21:22:40.4134605Z Running parallel tests on 8 processes 2025-08-14T21:22:40.4157860Z Name: tests to run (est. time: 127.74min) 2025-08-14T21:22:40.4158652Z Serial tests (73): 2025-08-14T21:22:40.4159290Z distributed/fsdp/test_distributed_checkpoint 1/1 2025-08-14T21:22:40.4160049Z distributed/test_dynamo_distributed 1/2 2025-08-14T21:22:40.4160876Z distributed/test_c10d_functional_native 1/1 2025-08-14T21:22:40.4161542Z distributed/tensor/test_attention 1/1 2025-08-14T21:22:40.4162221Z distributed/fsdp/test_fsdp_fx 1/1 2025-08-14T21:22:40.4162827Z distributed/test_nvshmem_triton 1/1 2025-08-14T21:22:40.4163624Z distributed/tensor/test_experimental_ops 1/1 2025-08-14T21:22:40.4164413Z distributed/checkpoint/test_hf_storage 1/1 2025-08-14T21:22:40.4165160Z distributed/tensor/test_common_rules 1/1 2025-08-14T21:22:40.4166424Z distributed/tensor/test_matrix_ops 1/1 2025-08-14T21:22:40.4167172Z distributed/tensor/experimental/test_register_sharding 1/1 2025-08-14T21:22:40.4167943Z distributed/optim/test_named_optimizer 1/1 2025-08-14T21:22:40.4168633Z distributed/_composable/test_replicate_with_fsdp 1/1 2025-08-14T21:22:40.4169341Z distributed/_tools/test_memory_tracker 1/1 2025-08-14T21:22:40.4169981Z distributed/_composable/test_checkpoint 1/1 2025-08-14T21:22:40.4170699Z distributed/checkpoint/_experimental/test_staging 1/1 2025-08-14T21:22:40.4171574Z distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 2025-08-14T21:22:40.4172377Z distributed/tensor/test_pointwise_ops 1/1 2025-08-14T21:22:40.4173135Z distributed/checkpoint/_experimental/test_checkpointer 1/1 2025-08-14T21:22:40.4173957Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 2025-08-14T21:22:40.4174617Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 2025-08-14T21:22:40.4175241Z distributed/test_functional_api 1/1 2025-08-14T21:22:40.4175843Z distributed/tensor/test_math_ops 1/1 2025-08-14T21:22:40.4176431Z distributed/test_symmetric_memory 1/1 2025-08-14T21:22:40.4177070Z distributed/checkpoint/e2e/test_fine_tuning 1/1 2025-08-14T21:22:40.4177735Z distributed/pipelining/test_stage 1/1 2025-08-14T21:22:40.4178361Z distributed/checkpoint/test_tp_checkpoint 1/1 2025-08-14T21:22:40.4178988Z distributed/test_nvshmem 1/1 2025-08-14T21:22:40.4179554Z distributed/tensor/test_embedding_ops 1/1 2025-08-14T21:22:40.4180169Z distributed/test_control_collectives 1/1 2025-08-14T21:22:40.4180934Z distributed/checkpoint/test_hf_safetensor_e2e 1/1 2025-08-14T21:22:40.4181764Z distributed/checkpoint/test_hsdp_checkpoint 1/1 2025-08-14T21:22:40.4183015Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 2025-08-14T21:22:40.4183822Z distributed/tensor/test_optimizers 1/1 2025-08-14T21:22:40.4184545Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 2025-08-14T21:22:40.4185365Z distributed/tensor/parallel/test_tp_random_state 1/1 2025-08-14T21:22:40.4186257Z distributed/_composable/test_composability/test_2d_composability 1/2 2025-08-14T21:22:40.4187137Z distributed/fsdp/test_fsdp_optim_state 1/1 2025-08-14T21:22:40.4187866Z distributed/_composable/fsdp/test_fully_shard_state 1/1 2025-08-14T21:22:40.4188578Z distributed/test_dist2 1/1 2025-08-14T21:22:40.4189137Z distributed/fsdp/test_fsdp_memory 1/1 2025-08-14T21:22:40.4189786Z distributed/_composable/test_replicate 1/1 2025-08-14T21:22:40.4190511Z distributed/_composable/fsdp/test_fully_shard_init 1/1 2025-08-14T21:22:40.4191226Z distributed/fsdp/test_fsdp_grad_acc 1/1 2025-08-14T21:22:40.4191839Z distributed/test_c10d_logger 1/1 2025-08-14T21:22:40.4192450Z distributed/test_c10d_object_collectives 1/1 2025-08-14T21:22:40.4193183Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 2025-08-14T21:22:40.4193958Z distributed/checkpoint/test_file_system_checkpoint 1/1 2025-08-14T21:22:40.4194689Z distributed/test_distributed_spawn 1/8 2025-08-14T21:22:40.4195321Z distributed/test_distributed_spawn 4/8 2025-08-14T21:22:40.4195934Z distributed/test_distributed_spawn 7/8 2025-08-14T21:22:40.4196587Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 2025-08-14T21:22:40.4197283Z distributed/fsdp/test_fsdp_fine_tune 1/1 2025-08-14T21:22:40.4197910Z distributed/fsdp/test_fsdp_comm_hooks 1/1 2025-08-14T21:22:40.4198524Z distributed/fsdp/test_fsdp_meta 1/1 2025-08-14T21:22:40.4199171Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 2025-08-14T21:22:40.4199818Z distributed/fsdp/test_fsdp_comm 1/1 2025-08-14T21:22:40.4200520Z distributed/test_data_parallel 1/1 2025-08-14T21:22:40.4201228Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 2025-08-14T21:22:40.4201961Z distributed/nn/jit/test_instantiator 1/1 2025-08-14T21:22:40.4202561Z distributed/test_pg_wrapper 1/1 2025-08-14T21:22:40.4203529Z distributed/fsdp/test_fsdp_core 1/3 2025-08-14T21:22:40.4204193Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 2025-08-14T21:22:40.4204943Z distributed/fsdp/test_fsdp_multiple_forward 1/1 2025-08-14T21:22:40.4205711Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 2025-08-14T21:22:40.4206534Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 2025-08-14T21:22:40.4207308Z distributed/_shard/sharded_tensor/ops/test_init 1/1 2025-08-14T21:22:40.4208156Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 2025-08-14T21:22:40.4208929Z distributed/test_c10d_gloo 1/2 2025-08-14T21:22:40.4209588Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 2025-08-14T21:22:40.4210399Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 2025-08-14T21:22:40.4211091Z distributed/test_c10d_nccl 3/3 2025-08-14T21:22:40.4211727Z distributed/tensor/test_dtensor_dispatch_overhead 1/1 2025-08-14T21:22:40.4212389Z Parallel tests (0): 2025-08-14T21:22:40.4212871Z Name: excluded (est. time: 0.0min) 2025-08-14T21:22:40.4213370Z Serial tests (0): 2025-08-14T21:22:40.4213793Z Parallel tests (0): 2025-08-14T21:22:40.4214606Z Running distributed/fsdp/test_distributed_checkpoint 1/1 ... [2025-08-14 21:22:40.413794] 2025-08-14T21:22:40.4215567Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:22:40.4217688Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_distributed_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:22:40.414136] 2025-08-14T21:22:59.3227986Z 2025-08-14T21:22:59.3231399Z distributed/fsdp/test_distributed_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_distributed_checkpoint_1.1_08c367a74bce90b7_.log 2025-08-14T21:22:59.3235901Z Running 2 items in this shard: test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type0_cuda, test/distributed/fsdp/test_distributed_checkpoint.py::TestDistributedCheckpointCUDA::test_distributed_checkpoint_state_dict_type1_cuda 2025-08-14T21:22:59.3238539Z 2025-08-14T21:22:59.3239047Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T21:22:59.3240032Z Uploading artifacts took 0.00 seconds 2025-08-14T21:22:59.3241191Z Running distributed/test_dynamo_distributed 1/2 ... [2025-08-14 21:22:59.323610] 2025-08-14T21:22:59.3242179Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:22:59.3248777Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dynamo_distributed.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:22:59.324200] 2025-08-14T21:28:20.6686993Z 2025-08-14T21:28:20.6694122Z distributed/test_dynamo_distributed 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dynamo_distributed_1.2_a4008ee8d0a5d04d_.log 2025-08-14T21:28:20.6727539Z Running 37 items in this shard: test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_call_method_forward, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_ddp_optimizer_inductor_strides_dont_specialize, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_aot_eager, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_issue90375, test/distributed/test_dynamo_distributed.py::TestFakeDistributedSingleProc::test_unbacked_symbol_splitting_no_binding, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_scalar, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_speculation_divergence, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_automatic_dynamic_tensor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_dim_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_scalar_missing_source, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_compiler_collectives_type_mismatch, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_activation_checkpointing, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_ddp_baseline_aot_eager_multiprocess, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_aot_eager, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_fsdp_unspecialized_forced_getattr_no_inline, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_get_pg_attr, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_guard_collective, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_aot_eager_static_graph, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor, test/distributed/test_dynamo_distributed.py::TestMultiProc::test_hf_bert_ddp_inductor_static_graph, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_compiled_flex_attention_full_model_ddp, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ddp_baseline_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_diff_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_dup_tensors_same_source, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_orig_params_assert, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_guards, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_fsdp_skip_register_attr_or_module, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_ctx_manager, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_layout_optimizations_inference, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_graph_split_inductor_transpose, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_higher_order_op, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_ignored_parameters, test/distributed/test_dynamo_distributed.py::TestSingleProc::test_no_split 2025-08-14T21:28:20.6756487Z 2025-08-14T21:28:20.6757002Z Running distributed/test_c10d_functional_native 1/1 ... [2025-08-14 21:28:20.669329] 2025-08-14T21:28:20.6757916Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:28:20.6759919Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_functional_native.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:28:20.669902] 2025-08-14T21:32:31.5305459Z 2025-08-14T21:32:31.5312098Z distributed/test_c10d_functional_native 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_functional_native_1.1_95eb669ad477682e_.log 2025-08-14T21:32:31.5337399Z Running 32 items in this shard: test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_coalesced_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_reduce_single_, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_all_to_all_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_broadcast, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_fixed_striding, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_functional_collectives_inference_mode, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_inductor_dtypeview_memory_leak, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_threading, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_unwaited, test/distributed/test_c10d_functional_native.py::TestWithNCCL::test_wait_tensor, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_collectives, test/distributed/test_c10d_functional_native.py::PyWorkTest::test_wait_tensor, test/distributed/test_c10d_functional_native.py::CompileTestCPU::test_inductor_all_reduce_cpu, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_gather_into_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_non_contig_input, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_reduce_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_all_to_all_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_broadcast, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_inplace_op_on_view, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reduce_scatter_tensor_single, test/distributed/test_c10d_functional_native.py::CompileTest::test_inductor_reuse_buffer_after_inplace_collective, test/distributed/test_c10d_functional_native.py::CompileTest::test_ranks_and_tag, test/distributed/test_c10d_functional_native.py::CompileTest::test_wait_tensor 2025-08-14T21:32:31.5361412Z 2025-08-14T21:32:31.5361915Z Running distributed/tensor/test_attention 1/1 ... [2025-08-14 21:32:31.531060] 2025-08-14T21:32:31.5362829Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:32:31.5364930Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_attention.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:32:31.531654] 2025-08-14T21:33:46.6802205Z 2025-08-14T21:33:46.6808240Z distributed/tensor/test_attention 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_attention_1.1_9d15368fec38a304_.log 2025-08-14T21:33:46.6813511Z Running 5 items in this shard: test/distributed/tensor/test_attention.py::RingAttentionTest::test_is_causal_behavior, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_custom_transformer, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_native_transformer, test/distributed/tensor/test_attention.py::RingAttentionTest::test_ring_attention_sdpa, test/distributed/tensor/test_attention.py::RingFlexAttentionTest::test_ring_flex_attention 2025-08-14T21:33:46.6817344Z 2025-08-14T21:33:46.6817802Z Running distributed/fsdp/test_fsdp_fx 1/1 ... [2025-08-14 21:33:46.680538] 2025-08-14T21:33:46.6818656Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:33:46.6821673Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fx.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:33:46.681138] 2025-08-14T21:33:50.8573453Z 2025-08-14T21:33:50.8575506Z distributed/fsdp/test_fsdp_fx 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fx_1.1_df96f1803e47041a_.log 2025-08-14T21:33:50.8577954Z Running 1 items in this shard: test/distributed/fsdp/test_fsdp_fx.py::TestSymbolicTracingCUDA::test_symbolic_tracing_outputs_cuda 2025-08-14T21:33:50.8579089Z 2025-08-14T21:33:50.8581643Z Running distributed/test_nvshmem_triton 1/1 ... [2025-08-14 21:33:50.857624] 2025-08-14T21:33:50.8582710Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:33:50.8588980Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nvshmem_triton.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:33:50.858240] 2025-08-14T21:33:57.1896587Z 2025-08-14T21:33:57.1897807Z distributed/test_nvshmem_triton 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nvshmem_triton_1.1_e65643e7835ba45d_.log 2025-08-14T21:33:57.1925629Z Running 38 items in this shard: test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_alltoall, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_barrier, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_broadcast, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_fence, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_get, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_get_ring, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_bfloat16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_float16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_float32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_float64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_int16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_int32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_int64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_minmax_reduce_int8, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_bfloat16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_float16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_float32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_float64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_int16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_int32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_int64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_prod_reduce_int8, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_put, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_put_signal_add, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_put_signal_set, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_quiet, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_signal_wait_until, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_bfloat16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_float16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_float32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_float64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_int16, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_int32, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_int64, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_int8, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sum_reduce_uint8, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_sync, test/distributed/test_nvshmem_triton.py::NVSHMEMTritonTest::test_triton_wait_until 2025-08-14T21:33:57.1953609Z 2025-08-14T21:33:57.1954193Z Running distributed/tensor/test_experimental_ops 1/1 ... [2025-08-14 21:33:57.189876] 2025-08-14T21:33:57.1955180Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:33:57.1957338Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_experimental_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:33:57.190473] 2025-08-14T21:34:18.9064249Z 2025-08-14T21:34:18.9066059Z distributed/tensor/test_experimental_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_experimental_ops_1.1_8ecac5fe05974536_.log 2025-08-14T21:34:18.9070390Z Running 3 items in this shard: test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_bernoulli, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_nll, test/distributed/tensor/test_experimental_ops.py::DistOtherOpsTest::test_slice 2025-08-14T21:34:18.9072990Z 2025-08-14T21:34:18.9073701Z Running distributed/checkpoint/test_hf_storage 1/1 ... [2025-08-14 21:34:18.906683] 2025-08-14T21:34:18.9074664Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:34:18.9077783Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hf_storage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:18.907218] 2025-08-14T21:34:22.8319113Z 2025-08-14T21:34:22.8321083Z distributed/checkpoint/test_hf_storage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hf_storage_1.1_49357e8dc46b7124_.log 2025-08-14T21:34:22.8325944Z Running 5 items in this shard: test/distributed/checkpoint/test_hf_storage.py::TestHfStorage::test_read_data_hf, test/distributed/checkpoint/test_hf_storage.py::TestHfStorage::test_read_metadata_hf, test/distributed/checkpoint/test_hf_storage.py::TestHfStorage::test_write_data_hf, test/distributed/checkpoint/test_hf_storage.py::TestHfStorage::test_write_data_with_sharding, test/distributed/checkpoint/test_hf_storage.py::TestHfStorage::test_write_metadata_hf 2025-08-14T21:34:22.8329448Z 2025-08-14T21:34:22.8329939Z Running distributed/tensor/test_common_rules 1/1 ... [2025-08-14 21:34:22.832203] 2025-08-14T21:34:22.8330842Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:34:22.8332943Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_common_rules.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:22.832735] 2025-08-14T21:34:30.6166068Z 2025-08-14T21:34:30.6168254Z distributed/tensor/test_common_rules 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_common_rules_1.1_82f0778ea29f1287_.log 2025-08-14T21:34:30.6177023Z Running 10 items in this shard: test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_basic_propagation, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_errors, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_linearity, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_merge_sharding, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_einop_pointwise_propagation, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_multi_sharding_on_mesh_dim, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_broadcasting, test/distributed/tensor/test_common_rules.py::CommonRulesTest::test_pointwise_rules_suggestion 2025-08-14T21:34:30.6200008Z 2025-08-14T21:34:30.6200918Z Running distributed/tensor/test_matrix_ops 1/1 ... [2025-08-14 21:34:30.617005] 2025-08-14T21:34:30.6201834Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:34:30.6203807Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_matrix_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:34:30.617626] 2025-08-14T21:36:10.0318066Z 2025-08-14T21:36:10.0319938Z distributed/tensor/test_matrix_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_matrix_ops_1.1_354ffc7aae3b0f76_.log 2025-08-14T21:36:10.0349526Z Running 15 items in this shard: test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_auto_redistribute, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_addmm_empty_operand, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_baddbmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_bmm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_dtensor_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_grouped_mm_kwargs0, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_grouped_mm_kwargs1, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_matmul, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_dot_product_attention, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_scaled_mm, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_t_partial, test/distributed/tensor/test_matrix_ops.py::DistMatrixOpsTest::test_tensordot_shampoo 2025-08-14T21:36:10.0359808Z 2025-08-14T21:36:10.0360633Z Running distributed/tensor/experimental/test_register_sharding 1/1 ... [2025-08-14 21:36:10.031652] 2025-08-14T21:36:10.0361743Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:36:10.0363950Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/experimental/test_register_sharding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:36:10.031935] 2025-08-14T21:36:26.9372395Z 2025-08-14T21:36:26.9374389Z distributed/tensor/experimental/test_register_sharding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.experimental.test_register_sharding_1.1_d0d819ec9233bc8a_.log 2025-08-14T21:36:26.9378106Z Running 2 items in this shard: test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_argmax, test/distributed/tensor/experimental/test_register_sharding.py::TestRegisterSharding::test_softmax_fwd 2025-08-14T21:36:26.9380210Z 2025-08-14T21:36:26.9380860Z Running distributed/optim/test_named_optimizer 1/1 ... [2025-08-14 21:36:26.937520] 2025-08-14T21:36:26.9382769Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:36:26.9387303Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:36:26.938148] 2025-08-14T21:36:28.6829128Z 2025-08-14T21:36:28.6831120Z distributed/optim/test_named_optimizer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.optim.test_named_optimizer_1.1_e3415aeb0aba9ccc_.log 2025-08-14T21:36:28.6832681Z 2025-08-14T21:36:28.6836611Z Running distributed/_composable/test_replicate_with_fsdp 1/1 ... [2025-08-14 21:36:28.683092] 2025-08-14T21:36:28.6837738Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:36:28.6842313Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate_with_fsdp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:36:28.683623] 2025-08-14T21:37:17.3177723Z 2025-08-14T21:37:17.3179967Z distributed/_composable/test_replicate_with_fsdp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_with_fsdp_1.1_52b3d3c346520753_.log 2025-08-14T21:37:17.3193979Z Running 5 items in this shard: test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_tp_device_mesh, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_replicate_transformer_managed_modules, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_parity_2d_mlp, test/distributed/_composable/test_replicate_with_fsdp.py::ReplicateTest::test_train_replicate_fsdp 2025-08-14T21:37:17.3198309Z 2025-08-14T21:37:17.3198834Z Running distributed/_tools/test_memory_tracker 1/1 ... [2025-08-14 21:37:17.318040] 2025-08-14T21:37:17.3199750Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:17.3202185Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:17.318586] 2025-08-14T21:37:25.5543268Z 2025-08-14T21:37:25.5545391Z distributed/_tools/test_memory_tracker 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._tools.test_memory_tracker_1.1_db9e745b5c55ceb3_.log 2025-08-14T21:37:25.5547642Z Running 1 items in this shard: test/distributed/_tools/test_memory_tracker.py::TestMemoryTracker::test_local_model 2025-08-14T21:37:25.5548535Z 2025-08-14T21:37:25.5550223Z Running distributed/_composable/test_checkpoint 1/1 ... [2025-08-14 21:37:25.554584] 2025-08-14T21:37:25.5551136Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:25.5558581Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:25.555170] 2025-08-14T21:37:31.3345639Z 2025-08-14T21:37:31.3347567Z distributed/_composable/test_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_checkpoint_1.1_90507023c7a59e5f_.log 2025-08-14T21:37:31.3353302Z Running 6 items in this shard: test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_checkpoint_kwargs, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_clears_state_on_error_in_forward, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_multi_args, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_random_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_cpu, test/distributed/_composable/test_checkpoint.py::TestCheckpoint::test_tensor_only_gpu 2025-08-14T21:37:31.3358854Z 2025-08-14T21:37:31.3359467Z Running distributed/checkpoint/_experimental/test_staging 1/1 ... [2025-08-14 21:37:31.334926] 2025-08-14T21:37:31.3360654Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:31.3363491Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_staging.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:31.335528] 2025-08-14T21:37:35.3110255Z 2025-08-14T21:37:35.3112164Z distributed/checkpoint/_experimental/test_staging 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_staging_1.1_0fc5e15691c0c3b9_.log 2025-08-14T21:37:35.3122015Z Running 7 items in this shard: test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_async_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_non_blocking_without_cuda, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_cuda_tensors_staging, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_different_option_combinations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_multiple_staging_operations, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_resource_cleanup, test/distributed/checkpoint/_experimental/test_staging.py::TestDefaultStager::test_sync_staging 2025-08-14T21:37:35.3128633Z 2025-08-14T21:37:35.3129332Z Running distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 ... [2025-08-14 21:37:35.311234] 2025-08-14T21:37:35.3130437Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:35.3132641Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpoint_writer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:35.311755] 2025-08-14T21:37:39.2378530Z 2025-08-14T21:37:39.2380766Z distributed/checkpoint/_experimental/test_checkpoint_writer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpoint_writer_1.1_0ab9ce9a074d109d_.log 2025-08-14T21:37:39.2390459Z Running 8 items in this shard: test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriterConfig::test_custom_values, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriterConfig::test_default_values, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_close, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_calls_barrier, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_calls_commit_hooks, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_creates_checkpoint_file, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_without_barrier, test/distributed/checkpoint/_experimental/test_checkpoint_writer.py::TestCheckpointWriter::test_write_without_commit_hook 2025-08-14T21:37:39.2398224Z 2025-08-14T21:37:39.2398748Z Running distributed/tensor/test_pointwise_ops 1/1 ... [2025-08-14 21:37:39.238093] 2025-08-14T21:37:39.2399657Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:39.2402117Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_pointwise_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:39.238619] 2025-08-14T21:37:44.4171849Z 2025-08-14T21:37:44.4173479Z distributed/tensor/test_pointwise_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_pointwise_ops_1.1_14189488ea9990b0_.log 2025-08-14T21:37:44.4180057Z Running 7 items in this shard: test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_activations, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_backward, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_dropout_errors, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_mul_out, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_mul_partial, test/distributed/tensor/test_pointwise_ops.py::DistElementwiseOpsTest::test_partial_add 2025-08-14T21:37:44.4185226Z 2025-08-14T21:37:44.4185867Z Running distributed/checkpoint/_experimental/test_checkpointer 1/1 ... [2025-08-14 21:37:44.417607] 2025-08-14T21:37:44.4186945Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:44.4189141Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/_experimental/test_checkpointer.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:44.418205] 2025-08-14T21:37:47.9424942Z 2025-08-14T21:37:47.9427929Z distributed/checkpoint/_experimental/test_checkpointer 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint._experimental.test_checkpointer_1.1_ef811bbc66a34cbe_.log 2025-08-14T21:37:47.9435267Z Running 4 items in this shard: test/distributed/checkpoint/_experimental/test_checkpointer.py::TestSyncCheckpointer::test_partial_load, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestSyncCheckpointer::test_partial_load_with_nested_dict, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestSyncCheckpointer::test_read_with_map_location, test/distributed/checkpoint/_experimental/test_checkpointer.py::TestSyncCheckpointer::test_sync_save_and_read 2025-08-14T21:37:47.9439180Z 2025-08-14T21:37:47.9439707Z Running distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 ... [2025-08-14 21:37:47.942728] 2025-08-14T21:37:47.9440779Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:37:47.9442872Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/fsdp/test_fsdp_dsd.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:37:47.943241] 2025-08-14T21:38:43.9881127Z 2025-08-14T21:38:43.9883985Z distributed/checkpoint/fsdp/test_fsdp_dsd 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.fsdp.test_fsdp_dsd_1.1_e30cbc707ca4c00e_.log 2025-08-14T21:38:43.9894790Z Running 6 items in this shard: test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_cpu_offload_full_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_1d_fsdp_get_model_state_dict, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp1_and_load_with_fsdp2_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_fsdp2_tp_and_load_with_tp, test/distributed/checkpoint/fsdp/test_fsdp_dsd.py::TestFullyShardWithDistributedStateDict::test_save_with_tp_and_load_with_fsdp2_tp 2025-08-14T21:38:43.9905924Z 2025-08-14T21:38:43.9906555Z Running distributed/checkpoint/e2e/test_fsdp_ep 1/1 ... [2025-08-14 21:38:43.988566] 2025-08-14T21:38:43.9907541Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:38:43.9909602Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fsdp_ep.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:38:43.989122] 2025-08-14T21:38:51.6731182Z 2025-08-14T21:38:51.6733907Z distributed/checkpoint/e2e/test_fsdp_ep 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fsdp_ep_1.1_dc2743d2cbeb017f_.log 2025-08-14T21:38:51.6736661Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fsdp_ep.py::TestFSDPWithEP::test_e2e 2025-08-14T21:38:51.6737576Z 2025-08-14T21:38:51.6738835Z Running distributed/test_functional_api 1/1 ... [2025-08-14 21:38:51.673350] 2025-08-14T21:38:51.6739700Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:38:51.6746075Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_functional_api.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:38:51.673941] 2025-08-14T21:41:01.4492607Z 2025-08-14T21:41:01.4494534Z distributed/test_functional_api 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_functional_api_1.1_2c60536c8d1cf7af_.log 2025-08-14T21:41:01.4507431Z Running 11 items in this shard: test/distributed/test_functional_api.py::TestMetaCollectives::test_all_reduce, test/distributed/test_functional_api.py::TestMakeFx::test_all_reduce_tracing, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_gather_into_tensor_coalesced_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_1d_input_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_all_to_all_single_split_sizes_none_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_with_dce_code_cuda, test/distributed/test_functional_api.py::TestCollectivesWithDistributedBackendCUDA::test_tracing_with_fakepg_cuda, test/distributed/test_functional_api.py::TestDistributedBackendCollectivesWithWorldSize4CUDA::test_permute_tensor_with_sub_group_cuda, test/distributed/test_functional_api.py::TestFunctionalAutogradWithDistributedBackendCUDA::test_all_to_all_single_cuda 2025-08-14T21:41:01.4518020Z 2025-08-14T21:41:01.4518520Z Running distributed/tensor/test_math_ops 1/1 ... [2025-08-14 21:41:01.449765] 2025-08-14T21:41:01.4519407Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:41:01.4521551Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_math_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:41:01.450392] 2025-08-14T21:43:24.1664707Z 2025-08-14T21:43:24.1667438Z distributed/tensor/test_math_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_math_ops_1.1_47eb1faa4bf5978d_.log 2025-08-14T21:43:24.1681391Z Running 20 items in this shard: test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_conj_complex_dtensor, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_cumsum, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_add_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_foreach_norm_different_mesh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_histc, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_bwd_req_grad, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_layer_norm_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_linalg_eigh, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_linear_op_reductions, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_mean, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_nll_loss_and_cross_entropy, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_rotary_embedding_complex_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_shard0_svd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_shard_math_ops, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_softmax_fwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_softmax_with_bwd, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_topk, test/distributed/tensor/test_math_ops.py::DistMathOpsTest::test_upsampling 2025-08-14T21:43:24.1695854Z 2025-08-14T21:43:24.1696338Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T21:43:24.1697210Z Uploading artifacts took 0.00 seconds 2025-08-14T21:43:24.1698045Z Running distributed/test_symmetric_memory 1/1 ... [2025-08-14 21:43:24.166989] 2025-08-14T21:43:24.1698905Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:43:24.1701351Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_symmetric_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:24.167558] 2025-08-14T21:43:52.6010114Z 2025-08-14T21:43:52.6016498Z PRINTING LOG FILE of distributed/test_symmetric_memory 1/1 (test/test-reports/distributed.test_symmetric_memory_1.1_add93377493fef3b_.log) 2025-08-14T21:43:52.6020079Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6022715Z import pkg_resources 2025-08-14T21:43:52.6024064Z Test results will be stored in test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-fca985c300d83d85.xml 2025-08-14T21:43:52.6025553Z ============================= test session starts ============================== 2025-08-14T21:43:52.6026602Z platform linux -- Python 3.10.18, pytest-7.3.2, pluggy-1.6.0 -- /opt/conda/envs/py_3.10/bin/python 2025-08-14T21:43:52.6027540Z cachedir: .pytest_cache 2025-08-14T21:43:52.6028658Z hypothesis profile 'pytorch_ci' -> database=None, max_examples=50, derandomize=True, suppress_health_check=[HealthCheck.too_slow] 2025-08-14T21:43:52.6029859Z rootdir: /var/lib/jenkins/pytorch 2025-08-14T21:43:52.6030409Z configfile: pytest.ini 2025-08-14T21:43:52.6031519Z plugins: cpp-2.3.0, rerunfailures-14.0, subtests-0.13.1, flakefinder-1.1.0, xdoctest-1.1.0, xdist-3.3.1, hypothesis-5.35.1, typeguard-4.3.0 2025-08-14T21:43:52.6032720Z collecting ... collected 88 items 2025-08-14T21:43:52.6033331Z stepcurrent: Cannot find last run test, not skipping 2025-08-14T21:43:52.6112832Z Running 88 items in this shard: test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_allow_overlapping_devices, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent_set_device_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent_set_device_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_set_device_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_set_device_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_backend, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_signal_pad, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_has_multicast_support, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_large_alloc, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_multimem_all_gather_matmul, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2, test/distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_subgroup, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_barrier_timeout, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_put_signal_timeout, test/distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_wait_signal_timeout, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_16, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_8, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter_corner_cases, test/distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_two_shot_all_reduce, test/distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_memset32, test/distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_stream_write_value32 2025-08-14T21:43:52.6192579Z 2025-08-14T21:43:52.6194163Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_allow_overlapping_devices I0814 21:43:27.345000 136401 site-packages/torch/testing/_internal/common_distributed.py:1732] Testing class SymmetricMemoryTest on 4 cuda 2025-08-14T21:43:52.6196696Z I0814 21:43:27.348000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 0 with pid 136469 2025-08-14T21:43:52.6198414Z I0814 21:43:27.350000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 1 with pid 136470 2025-08-14T21:43:52.6200120Z I0814 21:43:27.351000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 2 with pid 136471 2025-08-14T21:43:52.6202006Z I0814 21:43:27.353000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 3 with pid 136472 2025-08-14T21:43:52.6205267Z SKIPPED [0.0005s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158135 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 1%] 2025-08-14T21:43:52.6210512Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_cuda_nvlink_connectivity_detection SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158136 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 2%] 2025-08-14T21:43:52.6216703Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent_set_device_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158422 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 3%] 2025-08-14T21:43:52.6223358Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_persistent_set_device_True SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158464 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 4%] 2025-08-14T21:43:52.6228983Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_set_device_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158759 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 5%] 2025-08-14T21:43:52.6234577Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_empty_strided_p2p_set_device_True SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159066 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 6%] 2025-08-14T21:43:52.6240744Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_0 SKIPPED [0.0005s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159396 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 7%] 2025-08-14T21:43:52.6246757Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_gather_dim_1 SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159643 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 9%] 2025-08-14T21:43:52.6256636Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158916 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 10%] 2025-08-14T21:43:52.6262544Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_False_is_b_row_major_True SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159347 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 11%] 2025-08-14T21:43:52.6267907Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159499 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 12%] 2025-08-14T21:43:52.6273168Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_matmul_native_symm_mem_input_True_is_b_row_major_True SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159764 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 13%] 2025-08-14T21:43:52.6278928Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-replicated SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159921 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 14%] 2025-08-14T21:43:52.6284755Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_row-wise-sharded SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160049 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 15%] 2025-08-14T21:43:52.6290332Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_0_scale_mode_tensor-wise SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160085 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 17%] 2025-08-14T21:43:52.6296426Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-replicated SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160177 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 18%] 2025-08-14T21:43:52.6302442Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_row-wise-sharded SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160203 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 19%] 2025-08-14T21:43:52.6307775Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_all_gather_scaled_matmul_gather_dim_1_scale_mode_tensor-wise SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160347 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 20%] 2025-08-14T21:43:52.6312929Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_0 SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160445 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 21%] 2025-08-14T21:43:52.6317880Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_matmul_reduce_scatter_scatter_dim_1 SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160506 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 22%] 2025-08-14T21:43:52.6323665Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160549 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 23%] 2025-08-14T21:43:52.6328934Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_0_rowwise_True SKIPPED [0.0005s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160621 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 25%] 2025-08-14T21:43:52.6334907Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_False SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/160665 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 26%] 2025-08-14T21:43:52.6340809Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6344085Z import pkg_resources 2025-08-14T21:43:52.6346251Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6348513Z import pkg_resources 2025-08-14T21:43:52.6350650Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6352831Z import pkg_resources 2025-08-14T21:43:52.6354962Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6357166Z import pkg_resources 2025-08-14T21:43:52.6358023Z [rank0]:I0814 21:43:30.517000 136469 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6360685Z W0814 21:43:30.519000 136401 site-packages/torch/testing/_internal/common_distributed.py:1790] Detected failure from Rank 0 in: __main__.SymmetricMemoryTest.test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True, skipping rest of tests in Test class: SymmetricMemoryTest 2025-08-14T21:43:52.6362651Z ('RERUN', {'yellow': True}) [3.1308s] [ 27%] 2025-08-14T21:43:52.6364648Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True SKIPPED [0.0022s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True) [ 27%] 2025-08-14T21:43:52.6367793Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_backend SKIPPED [0.0019s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_get_backend) [ 28%] 2025-08-14T21:43:52.6370239Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_get_signal_pad SKIPPED [0.0019s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_get_signal_pad) [ 29%] 2025-08-14T21:43:52.6373039Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_has_multicast_support SKIPPED [0.0019s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_has_multicast_support) [ 30%] 2025-08-14T21:43:52.6375780Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_large_alloc SKIPPED [0.0020s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_large_alloc) [ 31%] 2025-08-14T21:43:52.6378395Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_False SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_all_gather_symm_mem_input_False) [ 32%] 2025-08-14T21:43:52.6381544Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_all_gather_symm_mem_input_True SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_all_gather_symm_mem_input_True) [ 34%] 2025-08-14T21:43:52.6385239Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False SKIPPED [0.0015s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_False) [ 35%] 2025-08-14T21:43:52.6388997Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_reduce_scatter_reduce_op_avg_symm_mem_input_True) [ 36%] 2025-08-14T21:43:52.6392474Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_False) [ 37%] 2025-08-14T21:43:52.6395938Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_low_contention_reduce_scatter_reduce_op_sum_symm_mem_input_True) [ 38%] 2025-08-14T21:43:52.6398608Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_multimem_all_gather_matmul SKIPPED [0.0003s] (multicast support is not available) [ 39%] 2025-08-14T21:43:52.6400872Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_0 SKIPPED [0.0016s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_optimal_layout_dim_0) [ 40%] 2025-08-14T21:43:52.6403284Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_1 SKIPPED [0.0007s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_optimal_layout_dim_1) [ 42%] 2025-08-14T21:43:52.6405683Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_optimal_layout_dim_2 SKIPPED [0.0007s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_optimal_layout_dim_2) [ 43%] 2025-08-14T21:43:52.6408055Z distributed/test_symmetric_memory.py::SymmetricMemoryTest::test_subgroup [rank0]:I0814 21:43:30.577000 136469 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6410070Z [rank3]:I0814 21:43:30.605000 136472 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6411672Z [rank3]:I0814 21:43:30.606000 136472 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6413231Z [rank1]:I0814 21:43:30.629000 136470 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6415216Z [rank1]:I0814 21:43:30.631000 136470 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6416636Z [rank2]:I0814 21:43:30.640000 136471 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6417959Z [rank2]:I0814 21:43:30.641000 136471 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6420256Z [rank0]:[W814 21:43:31.473632494 ProcessGroupNCCL.cpp:1519] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) 2025-08-14T21:43:52.6422903Z I0814 21:43:31.329000 136401 site-packages/torch/testing/_internal/common_distributed.py:1759] Class SymmetricMemoryTest finished 2025-08-14T21:43:52.6424551Z SKIPPED [0.0007s] (Previous test failed, skipping __main__.SymmetricMemoryTest.test_subgroup) [ 44%] 2025-08-14T21:43:52.6426453Z distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_barrier_timeout I0814 21:43:31.336000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 0 with pid 136757 2025-08-14T21:43:52.6428489Z I0814 21:43:31.338000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 1 with pid 136758 2025-08-14T21:43:52.6429989Z I0814 21:43:31.340000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 2 with pid 136759 2025-08-14T21:43:52.6431731Z I0814 21:43:31.341000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 3 with pid 136760 2025-08-14T21:43:52.6434500Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6436727Z import pkg_resources 2025-08-14T21:43:52.6438878Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6441155Z import pkg_resources 2025-08-14T21:43:52.6443547Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6445805Z import pkg_resources 2025-08-14T21:43:52.6447912Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6450102Z import pkg_resources 2025-08-14T21:43:52.6471138Z I0814 21:43:34.477000 136759 site-packages/torch/testing/_internal/common_distributed.py:818] Process 2 skipping test test_barrier_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6473977Z I0814 21:43:34.589000 136760 site-packages/torch/testing/_internal/common_distributed.py:818] Process 3 skipping test test_barrier_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6476656Z I0814 21:43:34.611000 136757 site-packages/torch/testing/_internal/common_distributed.py:818] Process 0 skipping test test_barrier_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6479726Z I0814 21:43:34.617000 136758 site-packages/torch/testing/_internal/common_distributed.py:818] Process 1 skipping test test_barrier_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6482004Z SKIPPED [4.0182s] (Test skipped at subprocess level, look at subprocess log for skip reason) [ 45%] 2025-08-14T21:43:52.6484236Z distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_put_signal_timeout I0814 21:43:35.357000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 0 with pid 137029 2025-08-14T21:43:52.6486326Z I0814 21:43:35.358000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 1 with pid 137030 2025-08-14T21:43:52.6487832Z I0814 21:43:35.360000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 2 with pid 137031 2025-08-14T21:43:52.6489318Z I0814 21:43:35.362000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 3 with pid 137032 2025-08-14T21:43:52.6492099Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6494340Z import pkg_resources 2025-08-14T21:43:52.6496777Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6499034Z import pkg_resources 2025-08-14T21:43:52.6501458Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6503674Z import pkg_resources 2025-08-14T21:43:52.6505768Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6507935Z import pkg_resources 2025-08-14T21:43:52.6509413Z I0814 21:43:38.538000 137031 site-packages/torch/testing/_internal/common_distributed.py:818] Process 2 skipping test test_put_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6512056Z I0814 21:43:38.627000 137032 site-packages/torch/testing/_internal/common_distributed.py:818] Process 3 skipping test test_put_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6514637Z I0814 21:43:38.646000 137030 site-packages/torch/testing/_internal/common_distributed.py:818] Process 1 skipping test test_put_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6517237Z I0814 21:43:38.656000 137029 site-packages/torch/testing/_internal/common_distributed.py:818] Process 0 skipping test test_put_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6519114Z SKIPPED [4.0179s] (Test skipped at subprocess level, look at subprocess log for skip reason) [ 46%] 2025-08-14T21:43:52.6521144Z distributed/test_symmetric_memory.py::SymmMemNegativeTest::test_wait_signal_timeout I0814 21:43:39.378000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 0 with pid 137301 2025-08-14T21:43:52.6523799Z I0814 21:43:39.380000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 1 with pid 137302 2025-08-14T21:43:52.6525342Z I0814 21:43:39.382000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 2 with pid 137303 2025-08-14T21:43:52.6526819Z I0814 21:43:39.383000 136401 site-packages/torch/testing/_internal/common_distributed.py:741] Started process 3 with pid 137304 2025-08-14T21:43:52.6529552Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6531744Z import pkg_resources 2025-08-14T21:43:52.6533845Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6536058Z import pkg_resources 2025-08-14T21:43:52.6538150Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6540487Z import pkg_resources 2025-08-14T21:43:52.6543010Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6545240Z import pkg_resources 2025-08-14T21:43:52.6546718Z I0814 21:43:42.668000 137302 site-packages/torch/testing/_internal/common_distributed.py:818] Process 1 skipping test test_wait_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6549337Z I0814 21:43:42.677000 137303 site-packages/torch/testing/_internal/common_distributed.py:818] Process 2 skipping test test_wait_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6551966Z I0814 21:43:42.678000 137301 site-packages/torch/testing/_internal/common_distributed.py:818] Process 0 skipping test test_wait_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6554571Z I0814 21:43:42.690000 137304 site-packages/torch/testing/_internal/common_distributed.py:818] Process 3 skipping test test_wait_signal_timeout for following reason: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6556453Z SKIPPED [4.3195s] (Test skipped at subprocess level, look at subprocess log for skip reason) [ 47%] 2025-08-14T21:43:52.6558526Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_16 I0814 21:43:43.698000 136401 site-packages/torch/testing/_internal/common_distributed.py:1732] Testing class SymmMemCollectiveTest on 4 cuda 2025-08-14T21:43:52.6560865Z I0814 21:43:43.702000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 0 with pid 137573 2025-08-14T21:43:52.6562450Z I0814 21:43:43.705000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 1 with pid 137574 2025-08-14T21:43:52.6564205Z I0814 21:43:43.708000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 2 with pid 137575 2025-08-14T21:43:52.6565994Z I0814 21:43:43.729000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 3 with pid 137576 2025-08-14T21:43:52.6567097Z SKIPPED [0.0004s] (multicast support is not available) [ 48%] 2025-08-14T21:43:52.6568451Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_4 SKIPPED [0.0004s] (multicast support is not available) [ 50%] 2025-08-14T21:43:52.6570415Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_gather_align_bytes_8 SKIPPED [0.0004s] (multicast support is not available) [ 51%] 2025-08-14T21:43:52.6572508Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_4 SKIPPED [0.0004s] (multicast support is not available) [ 52%] 2025-08-14T21:43:52.6574739Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8192 SKIPPED [0.0004s] (multicast support is not available) [ 53%] 2025-08-14T21:43:52.6576981Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_16_size_bytes_8196 SKIPPED [0.0006s] (multicast support is not available) [ 54%] 2025-08-14T21:43:52.6579193Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_4 SKIPPED [0.0003s] (multicast support is not available) [ 55%] 2025-08-14T21:43:52.6581681Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8192 SKIPPED [0.0003s] (multicast support is not available) [ 56%] 2025-08-14T21:43:52.6584161Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_4_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 57%] 2025-08-14T21:43:52.6586417Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_4 SKIPPED [0.0003s] (multicast support is not available) [ 59%] 2025-08-14T21:43:52.6588645Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8192 SKIPPED [0.0003s] (multicast support is not available) [ 60%] 2025-08-14T21:43:52.6590873Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_bfloat16_align_bytes_8_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 61%] 2025-08-14T21:43:52.6593066Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_4 SKIPPED [0.0003s] (multicast support is not available) [ 62%] 2025-08-14T21:43:52.6595259Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8192 SKIPPED [0.0005s] (multicast support is not available) [ 63%] 2025-08-14T21:43:52.6597470Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_16_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 64%] 2025-08-14T21:43:52.6599675Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_4 SKIPPED [0.0003s] (multicast support is not available) [ 65%] 2025-08-14T21:43:52.6602154Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8192 SKIPPED [0.0003s] (multicast support is not available) [ 67%] 2025-08-14T21:43:52.6604353Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_4_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 68%] 2025-08-14T21:43:52.6606538Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_4 SKIPPED [0.0003s] (multicast support is not available) [ 69%] 2025-08-14T21:43:52.6608726Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8192 SKIPPED [0.0003s] (multicast support is not available) [ 70%] 2025-08-14T21:43:52.6611563Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_all_reduce_float32_align_bytes_8_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 71%] 2025-08-14T21:43:52.6614270Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_4 SKIPPED [0.0009s] (multicast support is not available) [ 72%] 2025-08-14T21:43:52.6616240Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8192 SKIPPED [0.0001s] (multicast support is not available) [ 73%] 2025-08-14T21:43:52.6617315Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_16_size_bytes_8196 SKIPPED [0.0001s] (multicast support is not available) [ 75%] 2025-08-14T21:43:52.6618381Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_4 SKIPPED [0.0001s] (multicast support is not available) [ 76%] 2025-08-14T21:43:52.6619452Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8192 SKIPPED [0.0001s] (multicast support is not available) [ 77%] 2025-08-14T21:43:52.6620520Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_4_size_bytes_8196 SKIPPED [0.0001s] (multicast support is not available) [ 78%] 2025-08-14T21:43:52.6621799Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_4 SKIPPED [0.0001s] (multicast support is not available) [ 79%] 2025-08-14T21:43:52.6622860Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8192 SKIPPED [0.0001s] (multicast support is not available) [ 80%] 2025-08-14T21:43:52.6623936Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_bfloat16_align_bytes_8_size_bytes_8196 SKIPPED [0.0003s] (multicast support is not available) [ 81%] 2025-08-14T21:43:52.6624995Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_4 SKIPPED [0.0001s] (multicast support is not available) [ 82%] 2025-08-14T21:43:52.6626059Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8192 SKIPPED [0.0001s] (multicast support is not available) [ 84%] 2025-08-14T21:43:52.6627120Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_16_size_bytes_8196 SKIPPED [0.0001s] (multicast support is not available) [ 85%] 2025-08-14T21:43:52.6628178Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_4 SKIPPED [0.0001s] (multicast support is not available) [ 86%] 2025-08-14T21:43:52.6629237Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8192 SKIPPED [0.0001s] (multicast support is not available) [ 87%] 2025-08-14T21:43:52.6630308Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_4_size_bytes_8196 SKIPPED [0.0001s] (multicast support is not available) [ 88%] 2025-08-14T21:43:52.6631362Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_4 SKIPPED [0.0001s] (multicast support is not available) [ 89%] 2025-08-14T21:43:52.6632414Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8192 SKIPPED [0.0003s] (multicast support is not available) [ 90%] 2025-08-14T21:43:52.6633593Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_multimem_one_shot_all_reduce_float32_align_bytes_8_size_bytes_8196 SKIPPED [0.0001s] (multicast support is not available) [ 92%] 2025-08-14T21:43:52.6635232Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_one_shot_all_reduce SKIPPED [0.0005s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158138 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 93%] 2025-08-14T21:43:52.6637426Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/158760 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 94%] 2025-08-14T21:43:52.6639617Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_reduce_scatter_corner_cases SKIPPED [0.0004s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159067 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 95%] 2025-08-14T21:43:52.6642146Z distributed/test_symmetric_memory.py::SymmMemCollectiveTest::test_two_shot_all_reduce /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6643454Z import pkg_resources 2025-08-14T21:43:52.6644441Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6645469Z import pkg_resources 2025-08-14T21:43:52.6646449Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6647457Z import pkg_resources 2025-08-14T21:43:52.6648425Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6649438Z import pkg_resources 2025-08-14T21:43:52.6649827Z [rank3]:I0814 21:43:46.995000 137576 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6650466Z [rank3]:I0814 21:43:46.995000 137576 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6651081Z [rank1]:I0814 21:43:46.996000 137574 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6651695Z [rank1]:I0814 21:43:46.996000 137574 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6652310Z [rank0]:I0814 21:43:47.022000 137573 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6653086Z [rank0]:I0814 21:43:47.022000 137573 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6653695Z [rank2]:I0814 21:43:47.023000 137575 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6654302Z [rank2]:I0814 21:43:47.023000 137575 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6654980Z I0814 21:43:47.712000 136401 site-packages/torch/testing/_internal/common_distributed.py:1759] Class SymmMemCollectiveTest finished 2025-08-14T21:43:52.6656198Z SKIPPED [0.0005s] (Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/159397 for platform(s) rocm. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests.) [ 96%] 2025-08-14T21:43:52.6657653Z distributed/test_symmetric_memory.py::LoweringTest::test_lowering_one_shot_all_reduce I0814 21:43:47.716000 136401 site-packages/torch/testing/_internal/common_distributed.py:1732] Testing class LoweringTest on 4 cuda 2025-08-14T21:43:52.6658620Z I0814 21:43:47.720000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 0 with pid 137853 2025-08-14T21:43:52.6659317Z I0814 21:43:47.723000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 1 with pid 137854 2025-08-14T21:43:52.6660002Z I0814 21:43:47.726000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 2 with pid 137855 2025-08-14T21:43:52.6660682Z I0814 21:43:47.730000 136401 site-packages/torch/testing/_internal/common_distributed.py:1714] Started process 3 with pid 137856 2025-08-14T21:43:52.6662057Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6663086Z import pkg_resources 2025-08-14T21:43:52.6664066Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6665076Z import pkg_resources 2025-08-14T21:43:52.6666065Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6667091Z import pkg_resources 2025-08-14T21:43:52.6668064Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:52.6669089Z import pkg_resources 2025-08-14T21:43:52.6669475Z [rank1]:I0814 21:43:50.897000 137854 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6670101Z [rank1]:I0814 21:43:50.898000 137854 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6670717Z [rank3]:I0814 21:43:50.982000 137856 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6671324Z [rank3]:I0814 21:43:50.982000 137856 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6671936Z [rank2]:I0814 21:43:50.990000 137855 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6672668Z [rank2]:I0814 21:43:50.990000 137855 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6673280Z [rank0]:I0814 21:43:51.008000 137853 site-packages/torch/testing/_internal/common_distributed.py:1652] Setup complete 2025-08-14T21:43:52.6673886Z [rank0]:I0814 21:43:51.008000 137853 site-packages/torch/testing/_internal/common_distributed.py:1677] Terminating ... 2025-08-14T21:43:52.6674529Z I0814 21:43:51.712000 136401 site-packages/torch/testing/_internal/common_distributed.py:1759] Class LoweringTest finished 2025-08-14T21:43:52.6675104Z SKIPPED [0.0005s] (Fails with 'one_shot_all_reduce' not found in AOT graph, TODO: fix) [ 97%] 2025-08-14T21:43:52.6675765Z distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_memset32 SKIPPED [0.0021s] (skipIfRocm: test only runs on ('gfx942',)) [ 98%] 2025-08-14T21:43:52.6676621Z distributed/test_symmetric_memory.py::SymmMemSingleProcTest::test_stream_write_value32 SKIPPED [0.0018s] (skipIfRocm: test only runs on ('gfx942',)) [100%] 2025-08-14T21:43:52.6677115Z 2025-08-14T21:43:52.6677217Z ==================================== RERUNS ==================================== 2025-08-14T21:43:52.6677623Z _ SymmetricMemoryTest.test_fused_scaled_matmul_reduce_scatter_scatter_dim_1_rowwise_True _ 2025-08-14T21:43:52.6678004Z Traceback (most recent call last): 2025-08-14T21:43:52.6678497Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1797, in wrapper 2025-08-14T21:43:52.6678973Z raise rv 2025-08-14T21:43:52.6679168Z RuntimeError: Exception in worker process: 2025-08-14T21:43:52.6679417Z Traceback (most recent call last): 2025-08-14T21:43:52.6680033Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1664, in _worker_loop 2025-08-14T21:43:52.6680613Z cls._run_test_given_id(test_id) 2025-08-14T21:43:52.6681137Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1637, in _run_test_given_id 2025-08-14T21:43:52.6681646Z test_fn(**kwargs) 2025-08-14T21:43:52.6682087Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 1806, in wrapper 2025-08-14T21:43:52.6682553Z fn() 2025-08-14T21:43:52.6682960Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 3224, in wrapper 2025-08-14T21:43:52.6683424Z method(*args, **kwargs) 2025-08-14T21:43:52.6683895Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 553, in instantiated_test 2025-08-14T21:43:52.6684396Z test(self, **param_kwargs) 2025-08-14T21:43:52.6684843Z File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py", line 1915, in wrapper 2025-08-14T21:43:52.6685322Z raise unittest.SkipTest(reason) 2025-08-14T21:43:52.6685679Z unittest.case.SkipTest: skipIfRocm: test doesn't currently work on the ROCm stack 2025-08-14T21:43:52.6685958Z 2025-08-14T21:43:52.6686471Z - generated xml file: /var/lib/jenkins/pytorch/test/test-reports/python-pytest/distributed.test_symmetric_memory/distributed.test_symmetric_memory-fca985c300d83d85.xml - 2025-08-14T21:43:52.6687156Z ======================== 88 skipped, 1 rerun in 24.40s ========================= 2025-08-14T21:43:52.6687351Z 2025-08-14T21:43:52.6687750Z FINISHED PRINTING LOG FILE of distributed/test_symmetric_memory 1/1 (test/test-reports/distributed.test_symmetric_memory_1.1_add93377493fef3b_.log) 2025-08-14T21:43:52.6688217Z 2025-08-14T21:43:52.6688428Z Running distributed/checkpoint/e2e/test_fine_tuning 1/1 ... [2025-08-14 21:43:52.601373] 2025-08-14T21:43:52.6688809Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:43:52.6689640Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/e2e/test_fine_tuning.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:43:52.601991] 2025-08-14T21:44:04.6467362Z 2025-08-14T21:44:04.6469724Z distributed/checkpoint/e2e/test_fine_tuning 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.e2e.test_fine_tuning_1.1_b44d4b4e4fb131e7_.log 2025-08-14T21:44:04.6472254Z Running 1 items in this shard: test/distributed/checkpoint/e2e/test_fine_tuning.py::TestFineTuning::test_fine_tuning 2025-08-14T21:44:04.6473216Z 2025-08-14T21:44:04.6478225Z Running distributed/pipelining/test_stage 1/1 ... [2025-08-14 21:44:04.647287] 2025-08-14T21:44:04.6479292Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:44:04.6485047Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/pipelining/test_stage.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:44:04.647880] 2025-08-14T21:44:35.2840777Z 2025-08-14T21:44:35.2842510Z distributed/pipelining/test_stage 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.pipelining.test_stage_1.1_8c01bbf32ba77c5a_.log 2025-08-14T21:44:35.2850587Z Running 8 items in this shard: test/distributed/pipelining/test_stage.py::StageTest::test_custom_dw_with_fb_schedule, test/distributed/pipelining/test_stage.py::StageTest::test_manual, test/distributed/pipelining/test_stage.py::StageTest::test_output_chunks_memory_usage, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass0, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_ModelClass1, test/distributed/pipelining/test_stage.py::StageTest::test_tracer_kwargs_ModelClass0, test/distributed/pipelining/test_stage.py::StageNegativeTest::test_custom_dw_errors, test/distributed/pipelining/test_stage.py::StageNegativeTest::test_shape_prop_mismatch 2025-08-14T21:44:35.2855872Z 2025-08-14T21:44:35.2856401Z Running distributed/checkpoint/test_tp_checkpoint 1/1 ... [2025-08-14 21:44:35.284295] 2025-08-14T21:44:35.2857332Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:44:35.2859345Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_tp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:44:35.284839] 2025-08-14T21:44:54.6964615Z 2025-08-14T21:44:54.6966531Z distributed/checkpoint/test_tp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_tp_checkpoint_1.1_95c000b6a17a731c_.log 2025-08-14T21:44:54.6969756Z Running 2 items in this shard: test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint, test/distributed/checkpoint/test_tp_checkpoint.py::TestTpCheckpoint::test_tp_checkpoint_load_on_meta_device 2025-08-14T21:44:54.6971603Z 2025-08-14T21:44:54.6973470Z Running distributed/test_nvshmem 1/1 ... [2025-08-14 21:44:54.696653] 2025-08-14T21:44:54.6974489Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:44:54.6977663Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_nvshmem.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:44:54.697225] 2025-08-14T21:44:58.2717652Z 2025-08-14T21:44:58.2719151Z distributed/test_nvshmem 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_nvshmem_1.1_6edf0732fcd4042c_.log 2025-08-14T21:44:58.2728042Z Running 12 items in this shard: test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_alloc, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_nvshmem_get, test/distributed/test_nvshmem.py::NVSHMEMSymmetricMemoryTest::test_nvshmem_put, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_1, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_16, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_align_8, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_all_to_all_vdev_2d_offset, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_nvshmem_all_to_all, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_shuffle_combine_align_1, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_shuffle_combine_align_16, test/distributed/test_nvshmem.py::NVSHMEMAll2AllTest::test_shuffle_combine_align_8 2025-08-14T21:44:58.2736194Z 2025-08-14T21:44:58.2736711Z Running distributed/tensor/test_embedding_ops 1/1 ... [2025-08-14 21:44:58.272046] 2025-08-14T21:44:58.2737598Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:44:58.2739609Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_embedding_ops.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:44:58.272558] 2025-08-14T21:45:29.0590545Z 2025-08-14T21:45:29.0596986Z distributed/tensor/test_embedding_ops 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_embedding_ops_1.1_50988b58f9f44b85_.log 2025-08-14T21:45:29.0603485Z Running 4 items in this shard: test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_multiple_embeddings_rowwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_colwise_max_norm_errors, test/distributed/tensor/test_embedding_ops.py::TestEmbeddingOp::test_sharded_embedding_rowwise 2025-08-14T21:45:29.0607200Z 2025-08-14T21:45:29.0607843Z Running distributed/test_control_collectives 1/1 ... [2025-08-14 21:45:29.059398] 2025-08-14T21:45:29.0608973Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:45:29.0611502Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_control_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:45:29.059954] 2025-08-14T21:45:33.0863828Z 2025-08-14T21:45:33.0865062Z distributed/test_control_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_control_collectives_1.1_836de9111c372a70_.log 2025-08-14T21:45:33.0869895Z Running 13 items in this shard: test/distributed/test_control_collectives.py::TestCollectives::test_all_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum, test/distributed/test_control_collectives.py::TestCollectives::test_all_sum_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_barrier, test/distributed/test_control_collectives.py::TestCollectives::test_barrier_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast, test/distributed/test_control_collectives.py::TestCollectives::test_broadcast_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_gather, test/distributed/test_control_collectives.py::TestCollectives::test_gather_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_scatter, test/distributed/test_control_collectives.py::TestCollectives::test_scatter_timeout, test/distributed/test_control_collectives.py::TestCollectives::test_simple_user_func, test/distributed/test_control_collectives.py::TestCollectives::test_unique 2025-08-14T21:45:33.0874317Z 2025-08-14T21:45:33.0874609Z Running distributed/checkpoint/test_hf_safetensor_e2e 1/1 ... [2025-08-14 21:45:33.086832] 2025-08-14T21:45:33.0876082Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:45:33.0878365Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hf_safetensor_e2e.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:45:33.087410] 2025-08-14T21:46:17.3567919Z 2025-08-14T21:46:17.3575197Z distributed/checkpoint/test_hf_safetensor_e2e 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hf_safetensor_e2e_1.1_175dd776c0c1dd07_.log 2025-08-14T21:46:17.3585964Z Running 9 items in this shard: test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_load, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_load_into_empty_dict, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestSingleRankSaveLoad::test_save, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDistributedHFSafetensorsConsolidation::test_consolidate_to_one_file, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardPlacementChange::test_1d_to_1d_reshard_placement_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardPlacementChange::test_2d_to_2d_reshard_placement_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_1d_to_2d_reshard_mesh_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_2d_to_1d_reshard_mesh_change, test/distributed/checkpoint/test_hf_safetensor_e2e.py::TestDTensorReshardMeshChange::test_dtensor_checkpoint_resharding_with_empty_shard 2025-08-14T21:46:17.3595658Z 2025-08-14T21:46:17.3612125Z Running distributed/checkpoint/test_hsdp_checkpoint 1/1 ... [2025-08-14 21:46:17.357178] 2025-08-14T21:46:17.3613288Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:46:17.3615537Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_hsdp_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:46:17.357781] 2025-08-14T21:46:55.4108103Z 2025-08-14T21:46:55.4117122Z distributed/checkpoint/test_hsdp_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_hsdp_checkpoint_1.1_993f84e884195747_.log 2025-08-14T21:46:55.4123207Z Running 4 items in this shard: test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_checkpoint_is_even_sharded_model_True, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_False, test/distributed/checkpoint/test_hsdp_checkpoint.py::TestHSDPCheckpoint::test_hsdp_fsdp_checkpoint_conversion_is_even_sharded_model_True 2025-08-14T21:46:55.4127497Z 2025-08-14T21:46:55.4128157Z Running distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 ... [2025-08-14 21:46:55.411253] 2025-08-14T21:46:55.4129199Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:46:55.4131416Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:46:55.411846] 2025-08-14T21:47:07.5552249Z 2025-08-14T21:47:07.5554489Z distributed/checkpoint/test_fsdp_tp_checkpoint_conversion 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_fsdp_tp_checkpoint_conversion_1.1_c6487dcf1e7bfa5a_.log 2025-08-14T21:47:07.5557614Z Running 1 items in this shard: test/distributed/checkpoint/test_fsdp_tp_checkpoint_conversion.py::TestFsdpTpCheckpointConversion::test_fsdp_to_tp 2025-08-14T21:47:07.5574898Z 2025-08-14T21:47:07.5575474Z Running distributed/tensor/test_optimizers 1/1 ... [2025-08-14 21:47:07.555675] 2025-08-14T21:47:07.5576513Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:47:07.5578558Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_optimizers.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:47:07.556298] 2025-08-14T21:48:47.0084222Z 2025-08-14T21:48:47.0085984Z distributed/tensor/test_optimizers 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_optimizers_1.1_b5de8d131fe7520f_.log 2025-08-14T21:48:47.0097407Z Running 12 items in this shard: test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_RMSprop_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adadelta_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adagrad_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adamax_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_adamw_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_admaw_fused_across_meshes, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_asgd_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_nadam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_optimizer_foreach_supported_types_include_DTensor, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_radam_1d_sharding, test/distributed/tensor/test_optimizers.py::TestDTensorOptimizer::test_sgd_1d_sharding 2025-08-14T21:48:47.0106859Z 2025-08-14T21:48:47.0107480Z Running distributed/_composable/fsdp/test_fully_shard_overlap 1/1 ... [2025-08-14 21:48:47.008688] 2025-08-14T21:48:47.0108536Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:48:47.0110683Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_overlap.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:48:47.009269] 2025-08-14T21:48:50.8346290Z 2025-08-14T21:48:50.8348521Z distributed/_composable/fsdp/test_fully_shard_overlap 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_overlap_1.1_44b7b0794e9634d4_.log 2025-08-14T21:48:50.8352291Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_post_optim_event_overlap, test/distributed/_composable/fsdp/test_fully_shard_overlap.py::TestFullyShardOverlap::test_fully_shard_training_overlap 2025-08-14T21:48:50.8354509Z 2025-08-14T21:48:50.8355098Z Running distributed/tensor/parallel/test_tp_random_state 1/1 ... [2025-08-14 21:48:50.834557] 2025-08-14T21:48:50.8356083Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:48:50.8358191Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/parallel/test_tp_random_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:48:50.835128] 2025-08-14T21:49:02.5281499Z 2025-08-14T21:49:02.5283405Z distributed/tensor/parallel/test_tp_random_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.parallel.test_tp_random_state_1.1_36397181516ecfcf_.log 2025-08-14T21:49:02.5286252Z Running 1 items in this shard: test/distributed/tensor/parallel/test_tp_random_state.py::TensorParallelRandomStateTests::test_model_init 2025-08-14T21:49:02.5288052Z 2025-08-14T21:49:02.5289081Z Running distributed/_composable/test_composability/test_2d_composability 1/2 ... [2025-08-14 21:49:02.528399] 2025-08-14T21:49:02.5290211Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:49:02.5296229Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_composability/test_2d_composability.py', '--shard-id=1', '--num-shards=2', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:49:02.528952] 2025-08-14T21:50:35.3185528Z 2025-08-14T21:50:35.3187967Z distributed/_composable/test_composability/test_2d_composability 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_composability.test_2d_composability_1.2_f8bd0024a0183c20_.log 2025-08-14T21:50:35.3202086Z Running 9 items in this shard: test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_tp_with_fsdp_offloading, test/distributed/_composable/test_composability/test_2d_composability.py::TestFullyShard2DTraining::test_train_parity_2d_transformer, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelTraining::test_2d_e2e_training_use_orig_params, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_load_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_optim_state_dict_is_even_sharded_model_True, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_False, test/distributed/_composable/test_composability/test_2d_composability.py::TestNew2dParallelStateDict::test_2d_state_dict_is_even_sharded_model_True 2025-08-14T21:50:35.3213412Z 2025-08-14T21:50:35.3213953Z Running distributed/fsdp/test_fsdp_optim_state 1/1 ... [2025-08-14 21:50:35.319117] 2025-08-14T21:50:35.3214940Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:50:35.3217085Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_optim_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:50:35.319695] 2025-08-14T21:58:29.4401783Z 2025-08-14T21:58:29.4403646Z distributed/fsdp/test_fsdp_optim_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_optim_state_1.1_28efe7b7d88e7156_.log 2025-08-14T21:58:29.4483133Z Running 60 items in this shard: test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_compatible_with_trec, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_nested, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_flatten_sharded_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_keys, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_full_optim_state_dict_nested_invalid, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_interface_arguments, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_no_grad, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_input_warning, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type0_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_False_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_dict_nested_state_dict_type1_use_multiple_param_groups_True_rank0_only_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_optim_state_without_param_groups, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type0_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_ids_state_dict_type1_use_multiple_param_groups_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_rekey_optim_state_dict_to_names, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type0, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_save_load_without_0th_param_state_state_dict_type1, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_scatter_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_halve_world_size, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_False_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_False_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_nested_use_multiple_param_groups_True_wrap_alt_True_use_diff_optim_inputs_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_transformer, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type0_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_False, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_shard_full_optim_state_dict_unmanaged_params_state_dict_type1_add_to_fsdp_module_True, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_state_dict_with_none_tensor_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_use_orig_params, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_empty_optimizer_state, test/distributed/fsdp/test_fsdp_optim_state.py::TestFSDPOptimState::test_with_no_shard 2025-08-14T21:58:29.4550956Z 2025-08-14T21:58:29.4551577Z Running distributed/_composable/fsdp/test_fully_shard_state 1/1 ... [2025-08-14 21:58:29.446125] 2025-08-14T21:58:29.4552620Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:58:29.4554745Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_state.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:58:29.446749] 2025-08-14T21:58:33.7232869Z 2025-08-14T21:58:33.7235006Z distributed/_composable/fsdp/test_fully_shard_state 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_state_1.1_33d2792ab1527964_.log 2025-08-14T21:58:33.7241182Z Running 5 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_cls, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_deepcopy, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_reapply, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_state, test/distributed/_composable/fsdp/test_fully_shard_state.py::TestFullyShardState::test_fully_shard_unsupported_module_cls 2025-08-14T21:58:33.7245824Z 2025-08-14T21:58:33.7247126Z Running distributed/test_dist2 1/1 ... [2025-08-14 21:58:33.723567] 2025-08-14T21:58:33.7247988Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T21:58:33.7249983Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_dist2.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 21:58:33.724149] 2025-08-14T22:01:10.3950687Z 2025-08-14T22:01:10.3951557Z distributed/test_dist2 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_dist2_1.1_e688a3426e985d71_.log 2025-08-14T22:01:10.3964081Z Running 34 items in this shard: test/distributed/test_dist2.py::ProcessGroupTest::test_context_manager, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_allgather, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_allreduce, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_alltoall_base, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_barrier, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_broadcast, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_gather, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_group_split, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_reduce, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_reduce_scatter, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_remote_group_merge, test/distributed/test_dist2.py::Dist2MultiProcessTestCase::test_scatter, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_allgather, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_allreduce, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_alltoall_base, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_barrier, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_broadcast, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_gather, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_group_split, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_reduce, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_reduce_scatter, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_remote_group_merge, test/distributed/test_dist2.py::ProcessGroupGlooTest::test_scatter, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_allgather, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_allreduce, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_alltoall_base, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_barrier, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_broadcast, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_gather, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_group_split, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_reduce, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_reduce_scatter, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_remote_group_merge, test/distributed/test_dist2.py::ProcessGroupNCCLTest::test_scatter 2025-08-14T22:01:10.3988599Z 2025-08-14T22:01:10.3989057Z Running distributed/fsdp/test_fsdp_memory 1/1 ... [2025-08-14 22:01:10.395686] 2025-08-14T22:01:10.3989920Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:01:10.3991853Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:01:10.396270] 2025-08-14T22:01:45.1938834Z 2025-08-14T22:01:45.1941002Z distributed/fsdp/test_fsdp_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_memory_1.1_3c7a61a0baf160ae_.log 2025-08-14T22:01:45.1952800Z Running 2 items in this shard: test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_ckpt, test/distributed/fsdp/test_fsdp_memory.py::TestFSDPMemory::test_fsdp_memory_ckpt_no_ckpt 2025-08-14T22:01:45.1976724Z 2025-08-14T22:01:45.1977354Z Running distributed/_composable/test_replicate 1/1 ... [2025-08-14 22:01:45.194155] 2025-08-14T22:01:45.1978367Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:01:45.1980451Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/test_replicate.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:01:45.194729] 2025-08-14T22:03:03.0400441Z 2025-08-14T22:03:03.0402024Z distributed/_composable/test_replicate 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.test_replicate_1.1_e8b2aee5beaf8f62_.log 2025-08-14T22:03:03.0417816Z Running 17 items in this shard: test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_non_root_multiple_save_load, test/distributed/_composable/test_replicate.py::ReplicateStateDictTest::test_replicate_single_module_save_load, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateTest::test_replicate_wrong_device_id_type, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_device_id, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_fully_shard_init, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_ignore_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_move_args_kwargs_to_device, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_multi_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_single_module, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_with_kwargs, test/distributed/_composable/test_replicate.py::ReplicateFullyShardInit::test_replicate_wrong_device_id_type 2025-08-14T22:03:03.0432755Z 2025-08-14T22:03:03.0433481Z Running distributed/_composable/fsdp/test_fully_shard_init 1/1 ... [2025-08-14 22:03:03.040334] 2025-08-14T22:03:03.0434550Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:03:03.0436641Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:03:03.040914] 2025-08-14T22:03:23.6567088Z 2025-08-14T22:03:23.6568909Z distributed/_composable/fsdp/test_fully_shard_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_init_1.1_5d1f6cca2dde4d2d_.log 2025-08-14T22:03:23.6620697Z Running 42 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_ignored_param_device, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceTensor::test_move_states_to_device_tensor, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_invalid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardDeviceDTensor::test_move_states_to_device_dtensor_valid, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_2d_mesh_without_mesh_dim_names, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMeshArg::test_invalid_mesh_ndim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_duplicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_nested_fully_shard_and_replicate, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_modules_single, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_nested_fully_shard, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardManagedModulesAndStates::test_managed_states_shared_params_and_buffers, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_duplicates, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_list_of_mlps, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardParamModuleInfos::test_get_param_module_infos_shared_params, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_noncontiguous_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_raise_scalar_parameter, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterTensor::test_shard_tensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardedParameterDTensor::test_shard_dtensor_parameters, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_double_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_is_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_module_and_param_fqns, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_fully_shard_multi_module_root, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardLazyInit::test_reset_sharded_param_in_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_invalid_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_1d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_meta_device_2d_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMetaDeviceInit::test_rank0_broadcast_meta_device_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_1d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardProcessGroupInit::test_2d_process_group_init, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardHSDPBroadcast::test_hsdp_broadcast_across_replicas, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestHSDPWithCustomHook::test_custom_hook_custom_stream, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestHSDPWithCustomHook::test_custom_hsdp_all_reduce_hook, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_dim_neg1, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_transformer_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_1d_uneven_shard_largest_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_init_2d_transformer_shard_diff_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardShardPlacementFn::test_invalid_shard_dim, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardOldImport::test_old_import_training, test/distributed/_composable/fsdp/test_fully_shard_init.py::TestFullyShardMixedDtypeParam::test_mixed_dtypes_no_grad_param 2025-08-14T22:03:23.6662182Z 2025-08-14T22:03:23.6662687Z Running distributed/fsdp/test_fsdp_grad_acc 1/1 ... [2025-08-14 22:03:23.656872] 2025-08-14T22:03:23.6663582Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:03:23.6665612Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:03:23.657447] 2025-08-14T22:04:25.0164391Z 2025-08-14T22:04:25.0166447Z distributed/fsdp/test_fsdp_grad_acc 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_grad_acc_1.1_24bda261150c5023_.log 2025-08-14T22:04:25.0172911Z Running 6 items in this shard: test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs0_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_configs1_use_orig_params_True, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_False, test/distributed/fsdp/test_fsdp_grad_acc.py::TestGradAcc::test_grad_acc_cpu_offload_use_orig_params_True 2025-08-14T22:04:25.0177605Z 2025-08-14T22:04:25.0178074Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T22:04:25.0179814Z Uploading artifacts took 0.00 seconds 2025-08-14T22:04:25.0180599Z Running distributed/test_c10d_logger 1/1 ... [2025-08-14 22:04:25.017104] 2025-08-14T22:04:25.0181573Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:04:25.0183856Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_logger.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:04:25.017664] 2025-08-14T22:04:39.2677619Z 2025-08-14T22:04:39.2679668Z distributed/test_c10d_logger 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_logger_1.1_0aab9fff88f01f85_.log 2025-08-14T22:04:39.2682628Z Running 2 items in this shard: test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_exception_logger, test/distributed/test_c10d_logger.py::C10dErrorLoggerTest::test_get_or_create_logger 2025-08-14T22:04:39.2684171Z 2025-08-14T22:04:39.2686338Z Running distributed/test_c10d_object_collectives 1/1 ... [2025-08-14 22:04:39.268140] 2025-08-14T22:04:39.2687442Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:04:39.2694284Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_object_collectives.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:04:39.268742] 2025-08-14T22:05:44.4822753Z 2025-08-14T22:05:44.4824160Z distributed/test_c10d_object_collectives 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_object_collectives_1.1_3871b58ce75bc297_.log 2025-08-14T22:05:44.4833604Z Running 9 items in this shard: test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_all_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_broadcast_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_scatter_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_send_recv_object_list_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_all_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_broadcast_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_gather_object_cuda, test/distributed/test_c10d_object_collectives.py::TestObjectCollectivesCUDA::test_subpg_scatter_object_cuda 2025-08-14T22:05:44.4841340Z 2025-08-14T22:05:44.4841972Z Running distributed/_composable/fsdp/test_fully_shard_comm 1/1 ... [2025-08-14 22:05:44.482689] 2025-08-14T22:05:44.4842962Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:05:44.4845014Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:05:44.483243] 2025-08-14T22:09:36.2116789Z 2025-08-14T22:09:36.2124020Z distributed/_composable/fsdp/test_fully_shard_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_comm_1.1_506c911bb1fdb9ad_.log 2025-08-14T22:09:36.2146904Z Running 21 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_all_gather_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp16, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCollectiveOps::test_reduce_scatter_fp32, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_fully_shard_communication_count, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_manual_reshard_with_reshard_after_forward_false, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reduce_scatter_divide_factor, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardCommunication::test_set_reshard_after_forward, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_backward_misprefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_fully_shard_multi_module_unused_module, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_backward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardPrefetch::test_set_modules_to_forward_prefetch, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiProcess::test_unshard_async, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_no_param_group, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardUnshardMultiThread::test_unshard_without_lazy_init, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_exception_when_used_together_with_comm_hooks, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardAllocFromPG::test_fully_shard_alloc_from_pg, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_both_reductions, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardForceSumReduction::test_fully_shard_force_sum_reduce_scatter, test/distributed/_composable/fsdp/test_fully_shard_comm.py::TestFullyShardReduceOpWorldSize1::test_size1_reduceop 2025-08-14T22:09:36.2168331Z 2025-08-14T22:09:36.2168936Z Running distributed/checkpoint/test_file_system_checkpoint 1/1 ... [2025-08-14 22:09:36.216417] 2025-08-14T22:09:36.2169954Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:09:36.2176722Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:09:36.217012] 2025-08-14T22:10:32.2124042Z 2025-08-14T22:10:32.2125876Z distributed/checkpoint/test_file_system_checkpoint 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.checkpoint.test_file_system_checkpoint_1.1_c005275dac402ed6_.log 2025-08-14T22:10:32.2138885Z Running 9 items in this shard: test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoad::test_read_write_only_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions0, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions1, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithSharedTensor::test_read_write_shard_tensor_extensions2, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_rowwise_to_colwise, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_load_with_different_shard_plan, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_save_load_bytes, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedReshardOnLoad::test_switch_between_sharded_tensor_to_tensor, test/distributed/checkpoint/test_file_system_checkpoint.py::TestDistributedStateDictSaveLoadWithCaching::test_read_write_shard_tensor 2025-08-14T22:10:32.2150502Z 2025-08-14T22:10:32.2150982Z Running distributed/test_distributed_spawn 1/8 ... [2025-08-14 22:10:32.212955] 2025-08-14T22:10:32.2162549Z MPI not available -- MPI backend tests will be skipped 2025-08-14T22:10:32.2165924Z Running distributed tests for the test backend with env init_method 2025-08-14T22:10:32.2170656Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:10:32.2177486Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:32.217128] 2025-08-14T22:10:35.8640809Z 2025-08-14T22:10:35.8643181Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_dd92656ac8f99db3_.log 2025-08-14T22:10:35.8645023Z Running 0 items in this shard: 2025-08-14T22:10:35.8655790Z 2025-08-14T22:10:35.8656389Z Running distributed tests for the test backend with file init_method 2025-08-14T22:10:35.8658252Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:10:35.8664984Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:35.865916] 2025-08-14T22:10:39.5606347Z 2025-08-14T22:10:39.5608858Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_1382d9728cc1c4e9_.log 2025-08-14T22:10:39.5610593Z Running 0 items in this shard: 2025-08-14T22:10:39.5610975Z 2025-08-14T22:10:39.5620693Z Running distributed tests for the nccl backend with env init_method 2025-08-14T22:10:39.5623220Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:10:39.5631032Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:10:39.562417] 2025-08-14T22:16:37.5833660Z 2025-08-14T22:16:37.5835404Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_71cb450547a4c836_.log 2025-08-14T22:16:37.5865375Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:16:37.5896305Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-08-14T22:16:37.5897560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-08-14T22:16:37.5898693Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2025-08-14T22:16:37.5899856Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2025-08-14T22:16:37.5901092Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-08-14T22:16:37.5902313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-08-14T22:16:37.5903416Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2025-08-14T22:16:37.5904545Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-08-14T22:16:37.5905622Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-08-14T22:16:37.5906686Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2025-08-14T22:16:37.5908093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-08-14T22:16:37.5909346Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-08-14T22:16:37.5910549Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2025-08-14T22:16:37.5911704Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-08-14T22:16:37.5912849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-08-14T22:16:37.5914118Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2025-08-14T22:16:37.5915406Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2025-08-14T22:16:37.5916639Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2025-08-14T22:16:37.5917872Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-08-14T22:16:37.5919298Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-08-14T22:16:37.5920932Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2025-08-14T22:16:37.5922219Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2025-08-14T22:16:37.5923291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-08-14T22:16:37.5924369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-08-14T22:16:37.5925445Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-08-14T22:16:37.5926555Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-08-14T22:16:37.5927621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-08-14T22:16:37.5928756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2025-08-14T22:16:37.5929950Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param 2025-08-14T22:16:37.5931186Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-08-14T22:16:37.5932344Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2025-08-14T22:16:37.5933358Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-08-14T22:16:37.5934399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2025-08-14T22:16:37.5935489Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2025-08-14T22:16:37.5936896Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:16:37.5937571Z 2025-08-14T22:16:37.5937820Z Running distributed tests for the nccl backend with file init_method 2025-08-14T22:16:37.5938318Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:16:37.5939601Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:16:37.589896] 2025-08-14T22:21:51.0722630Z 2025-08-14T22:21:51.0724309Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_b8f208ea715e3916_.log 2025-08-14T22:21:51.0757799Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:21:51.0789253Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-08-14T22:21:51.0791333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-08-14T22:21:51.0793258Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2025-08-14T22:21:51.0795165Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2025-08-14T22:21:51.0797227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-08-14T22:21:51.0799292Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-08-14T22:21:51.0801831Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2025-08-14T22:21:51.0803797Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-08-14T22:21:51.0805635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-08-14T22:21:51.0807431Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2025-08-14T22:21:51.0809410Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-08-14T22:21:51.0811514Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-08-14T22:21:51.0813513Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2025-08-14T22:21:51.0815438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-08-14T22:21:51.0817333Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-08-14T22:21:51.0819438Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2025-08-14T22:21:51.0821562Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2025-08-14T22:21:51.0823760Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2025-08-14T22:21:51.0825948Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-08-14T22:21:51.0828000Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-08-14T22:21:51.0830399Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2025-08-14T22:21:51.0832866Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2025-08-14T22:21:51.0834649Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-08-14T22:21:51.0836437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-08-14T22:21:51.0838237Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-08-14T22:21:51.0840057Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-08-14T22:21:51.0841915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-08-14T22:21:51.0843785Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2025-08-14T22:21:51.0845762Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param 2025-08-14T22:21:51.0847809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-08-14T22:21:51.0849718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2025-08-14T22:21:51.0851714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-08-14T22:21:51.0853470Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2025-08-14T22:21:51.0855325Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2025-08-14T22:21:51.0857283Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:21:51.0858406Z 2025-08-14T22:21:51.0858798Z Running distributed tests for the gloo backend with env init_method 2025-08-14T22:21:51.0859599Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:21:51.0861693Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:21:51.078446] 2025-08-14T22:27:00.8395165Z 2025-08-14T22:27:00.8396886Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_de816a1de9c49b0e_.log 2025-08-14T22:27:00.8429338Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:27:00.8460612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-08-14T22:27:00.8462644Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-08-14T22:27:00.8464507Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2025-08-14T22:27:00.8466382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2025-08-14T22:27:00.8468447Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-08-14T22:27:00.8470468Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-08-14T22:27:00.8472320Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2025-08-14T22:27:00.8474152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-08-14T22:27:00.8476412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-08-14T22:27:00.8478228Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2025-08-14T22:27:00.8480493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-08-14T22:27:00.8482727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-08-14T22:27:00.8484731Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2025-08-14T22:27:00.8486640Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-08-14T22:27:00.8488504Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-08-14T22:27:00.8490577Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2025-08-14T22:27:00.8492664Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2025-08-14T22:27:00.8494705Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2025-08-14T22:27:00.8497085Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-08-14T22:27:00.8499141Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-08-14T22:27:00.8501556Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2025-08-14T22:27:00.8504036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2025-08-14T22:27:00.8505790Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-08-14T22:27:00.8507570Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-08-14T22:27:00.8509391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-08-14T22:27:00.8511214Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-08-14T22:27:00.8512992Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-08-14T22:27:00.8514857Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2025-08-14T22:27:00.8516828Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param 2025-08-14T22:27:00.8518864Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-08-14T22:27:00.8520898Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2025-08-14T22:27:00.8522766Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-08-14T22:27:00.8525005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2025-08-14T22:27:00.8526809Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2025-08-14T22:27:00.8528751Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:27:00.8530055Z 2025-08-14T22:27:00.8530528Z Running distributed tests for the gloo backend with file init_method 2025-08-14T22:27:00.8531463Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:27:00.8533911Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=1', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:27:00.844183] 2025-08-14T22:32:12.5485299Z 2025-08-14T22:32:12.5486825Z distributed/test_distributed_spawn 1/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_1.8_fa0c85b331988c96_.log 2025-08-14T22:32:12.5520920Z Running 35 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:32:12.5551299Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_accumulate_gradients_no_sync_grad_is_view 2025-08-14T22:32:12.5553347Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_complex 2025-08-14T22:32:12.5555248Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_coalesced_group 2025-08-14T22:32:12.5557136Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_min 2025-08-14T22:32:12.5559433Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_max_complex_unsupported 2025-08-14T22:32:12.5561598Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_complex 2025-08-14T22:32:12.5563424Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_async 2025-08-14T22:32:12.5565629Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda_complex 2025-08-14T22:32:12.5567511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_cuda 2025-08-14T22:32:12.5569273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_full_group_cuda 2025-08-14T22:32:12.5571198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_complex 2025-08-14T22:32:12.5573270Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_group_cuda 2025-08-14T22:32:12.5575560Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_full_group 2025-08-14T22:32:12.5577814Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_op_list_err 2025-08-14T22:32:12.5579756Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager_async 2025-08-14T22:32:12.5581812Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_compute_bucket_assignment_by_size_sparse_error_with_logger 2025-08-14T22:32:12.5583959Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_buffer_hook_allreduce 2025-08-14T22:32:12.5586373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_build_debug_param_to_name_mapping_requires_grad 2025-08-14T22:32:12.5588404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_grad_div_uneven_inputs 2025-08-14T22:32:12.5590792Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_multiple_nested_unused_params_err_ignore_params 2025-08-14T22:32:12.5593139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_no_set_grad_none 2025-08-14T22:32:12.5595315Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_uneven_inputs 2025-08-14T22:32:12.5597076Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_zero_output_features 2025-08-14T22:32:12.5598849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_destroy_full_group 2025-08-14T22:32:12.5600930Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_dump_DDP_relevant_env_vars 2025-08-14T22:32:12.5602732Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_invalid_static_graph 2025-08-14T22:32:12.5604457Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_isend_torch_profiler 2025-08-14T22:32:12.5606288Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_wait_all_ranks 2025-08-14T22:32:12.5608226Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_new_subgroups_with_group_param 2025-08-14T22:32:12.5610581Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_grad_is_view 2025-08-14T22:32:12.5612511Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_cuda 2025-08-14T22:32:12.5614331Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum_twice 2025-08-14T22:32:12.5616373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_api_cpu 2025-08-14T22:32:12.5618482Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_static_graph_multi_forward 2025-08-14T22:32:12.5620430Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_verify_model_across_rank_with_logger 2025-08-14T22:32:12.5621549Z 2025-08-14T22:32:12.5622011Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T22:32:12.5622869Z Uploading artifacts took 0.00 seconds 2025-08-14T22:32:12.5623717Z Running distributed/test_distributed_spawn 4/8 ... [2025-08-14 22:32:12.552855] 2025-08-14T22:32:12.5624901Z MPI not available -- MPI backend tests will be skipped 2025-08-14T22:32:12.5625956Z Running distributed tests for the test backend with env init_method 2025-08-14T22:32:12.5626827Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:32:12.5628914Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:32:12.556645] 2025-08-14T22:32:16.3064114Z 2025-08-14T22:32:16.3066043Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_0f17a7170ad86e32_.log 2025-08-14T22:32:16.3068027Z Running 0 items in this shard: 2025-08-14T22:32:16.3068447Z 2025-08-14T22:32:16.3078748Z Running distributed tests for the test backend with file init_method 2025-08-14T22:32:16.3081773Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:32:16.3090235Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:32:16.308273] 2025-08-14T22:32:20.0019951Z 2025-08-14T22:32:20.0021484Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_23592acc0692c398_.log 2025-08-14T22:32:20.0023136Z Running 0 items in this shard: 2025-08-14T22:32:20.0023503Z 2025-08-14T22:32:20.0032058Z Running distributed tests for the nccl backend with env init_method 2025-08-14T22:32:20.0032987Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:32:20.0043118Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:32:20.003570] 2025-08-14T22:35:55.8840671Z 2025-08-14T22:35:55.8842375Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_b1a1d8f40e7b9039_.log 2025-08-14T22:35:55.8869626Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:35:55.8896656Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2025-08-14T22:35:55.8898879Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2025-08-14T22:35:55.8901033Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2025-08-14T22:35:55.8903047Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-08-14T22:35:55.8904822Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-08-14T22:35:55.8906612Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-08-14T22:35:55.8908376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-08-14T22:35:55.8910459Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2025-08-14T22:35:55.8912240Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2025-08-14T22:35:55.8913994Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-08-14T22:35:55.8915712Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-08-14T22:35:55.8917648Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-08-14T22:35:55.8919776Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-08-14T22:35:55.8921875Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-08-14T22:35:55.8923838Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-08-14T22:35:55.8925710Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2025-08-14T22:35:55.8927476Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2025-08-14T22:35:55.8929369Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-08-14T22:35:55.8931119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2025-08-14T22:35:55.8932873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2025-08-14T22:35:55.8934829Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-08-14T22:35:55.8936703Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-08-14T22:35:55.8939189Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2025-08-14T22:35:55.8941428Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-08-14T22:35:55.8943297Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-08-14T22:35:55.8945037Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-08-14T22:35:55.8946721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2025-08-14T22:35:55.8948345Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2025-08-14T22:35:55.8950168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-08-14T22:35:55.8952119Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-08-14T22:35:55.8954067Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:35:55.8955166Z 2025-08-14T22:35:55.8955576Z Running distributed tests for the nccl backend with file init_method 2025-08-14T22:35:55.8956370Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:35:55.8958775Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:35:55.886829] 2025-08-14T22:39:31.9294816Z 2025-08-14T22:39:31.9296852Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_f6a17c53aa05448a_.log 2025-08-14T22:39:31.9324039Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:39:31.9359404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2025-08-14T22:39:31.9361939Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2025-08-14T22:39:31.9364061Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2025-08-14T22:39:31.9366062Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-08-14T22:39:31.9367807Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-08-14T22:39:31.9369574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-08-14T22:39:31.9371328Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-08-14T22:39:31.9373028Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2025-08-14T22:39:31.9374708Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2025-08-14T22:39:31.9376412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-08-14T22:39:31.9378112Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-08-14T22:39:31.9380010Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-08-14T22:39:31.9382127Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-08-14T22:39:31.9384093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-08-14T22:39:31.9386051Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-08-14T22:39:31.9387902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2025-08-14T22:39:31.9390024Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2025-08-14T22:39:31.9391902Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-08-14T22:39:31.9393641Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2025-08-14T22:39:31.9395458Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2025-08-14T22:39:31.9397437Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-08-14T22:39:31.9399303Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-08-14T22:39:31.9401750Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2025-08-14T22:39:31.9404001Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-08-14T22:39:31.9405823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-08-14T22:39:31.9407533Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-08-14T22:39:31.9409202Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2025-08-14T22:39:31.9411293Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2025-08-14T22:39:31.9413124Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-08-14T22:39:31.9415070Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-08-14T22:39:31.9417003Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:39:31.9418097Z 2025-08-14T22:39:31.9418489Z Running distributed tests for the gloo backend with env init_method 2025-08-14T22:39:31.9419270Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:39:31.9421366Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:39:31.932765] 2025-08-14T22:43:39.8439690Z 2025-08-14T22:43:39.8447405Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_306b930065fa3797_.log 2025-08-14T22:43:39.8476365Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:43:39.8504123Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2025-08-14T22:43:39.8506087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2025-08-14T22:43:39.8508152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2025-08-14T22:43:39.8510146Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-08-14T22:43:39.8511921Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-08-14T22:43:39.8513672Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-08-14T22:43:39.8515407Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-08-14T22:43:39.8517091Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2025-08-14T22:43:39.8518777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2025-08-14T22:43:39.8520781Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-08-14T22:43:39.8522667Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-08-14T22:43:39.8525116Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-08-14T22:43:39.8527242Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-08-14T22:43:39.8529198Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-08-14T22:43:39.8531176Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-08-14T22:43:39.8533108Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2025-08-14T22:43:39.8534875Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2025-08-14T22:43:39.8536791Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-08-14T22:43:39.8538563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2025-08-14T22:43:39.8540643Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2025-08-14T22:43:39.8542694Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-08-14T22:43:39.8544859Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-08-14T22:43:39.8546982Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2025-08-14T22:43:39.8549231Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-08-14T22:43:39.8551064Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-08-14T22:43:39.8552777Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-08-14T22:43:39.8554472Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2025-08-14T22:43:39.8556087Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2025-08-14T22:43:39.8557900Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-08-14T22:43:39.8559823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-08-14T22:43:39.8561967Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:43:39.8563264Z 2025-08-14T22:43:39.8563730Z Running distributed tests for the gloo backend with file init_method 2025-08-14T22:43:39.8564530Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:43:39.8566599Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=4', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:43:39.847575] 2025-08-14T22:47:49.7397275Z 2025-08-14T22:47:49.7399023Z distributed/test_distributed_spawn 4/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_4.8_4cce4994cc5179b9_.log 2025-08-14T22:47:49.7436415Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:47:49.7464603Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_3_level_hierarchical_model_averager 2025-08-14T22:47:49.7466617Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU 2025-08-14T22:47:49.7468721Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_No_Affine 2025-08-14T22:47:49.7470746Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_complex 2025-08-14T22:47:49.7472519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_gather_full_group 2025-08-14T22:47:49.7474983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_max 2025-08-14T22:47:49.7476743Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_sum 2025-08-14T22:47:49.7478441Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_min 2025-08-14T22:47:49.7480597Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_product 2025-08-14T22:47:49.7482379Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_sum_cuda 2025-08-14T22:47:49.7484083Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group 2025-08-14T22:47:49.7486027Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_full_group_cuda 2025-08-14T22:47:49.7488148Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_group 2025-08-14T22:47:49.7490100Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_batch_isend_irecv_gloo_tags 2025-08-14T22:47:49.7492065Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_control_flow_different_across_ranks 2025-08-14T22:47:49.7494268Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_create_graph 2025-08-14T22:47:49.7496055Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_hook_pickling_powerSGD 2025-08-14T22:47:49.7497942Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_profiling_autograd_profiler 2025-08-14T22:47:49.7499687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_gather 2025-08-14T22:47:49.7501736Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_monitored_barrier_allreduce_hang 2025-08-14T22:47:49.7503849Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_dict_module 2025-08-14T22:47:49.7505714Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager 2025-08-14T22:47:49.7507815Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view 2025-08-14T22:47:49.7510056Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_post_localSGD_optimizer_step_reload 2025-08-14T22:47:49.7511913Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_sum 2025-08-14T22:47:49.7513621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_scatter_v_cuda 2025-08-14T22:47:49.7515291Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_sum 2025-08-14T22:47:49.7516915Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter_checks 2025-08-14T22:47:49.7518736Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_any_source_torch_profiler 2025-08-14T22:47:49.7520806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_nccl_torch_profiler 2025-08-14T22:47:49.7523122Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_with_tag_autograd_profiler 2025-08-14T22:47:49.7524225Z 2025-08-14T22:47:49.7524683Z Running distributed/test_distributed_spawn 7/8 ... [2025-08-14 22:47:49.742823] 2025-08-14T22:47:49.7525639Z MPI not available -- MPI backend tests will be skipped 2025-08-14T22:47:49.7526509Z Running distributed tests for the test backend with env init_method 2025-08-14T22:47:49.7527297Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:47:49.7529423Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:47:49.746516] 2025-08-14T22:47:53.5048146Z 2025-08-14T22:47:53.5049772Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_7f4a99ffc70913c7_.log 2025-08-14T22:47:53.5051544Z Running 0 items in this shard: 2025-08-14T22:47:53.5051919Z 2025-08-14T22:47:53.5061961Z Running distributed tests for the test backend with file init_method 2025-08-14T22:47:53.5063154Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:47:53.5070576Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:47:53.506376] 2025-08-14T22:47:57.2311486Z 2025-08-14T22:47:57.2314411Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_0d85d319d0746da1_.log 2025-08-14T22:47:57.2316113Z Running 0 items in this shard: 2025-08-14T22:47:57.2316502Z 2025-08-14T22:47:57.2324104Z Running distributed tests for the nccl backend with env init_method 2025-08-14T22:47:57.2326975Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:47:57.2334117Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:47:57.232872] 2025-08-14T22:51:46.5965838Z 2025-08-14T22:51:46.5968038Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_5039708cd7b4b1a7_.log 2025-08-14T22:51:46.5996210Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:51:46.6046025Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-08-14T22:51:46.6048227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-08-14T22:51:46.6050493Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2025-08-14T22:51:46.6052617Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-08-14T22:51:46.6054681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-08-14T22:51:46.6056671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2025-08-14T22:51:46.6058675Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-08-14T22:51:46.6060687Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-08-14T22:51:46.6062574Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-08-14T22:51:46.6064354Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2025-08-14T22:51:46.6066250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-08-14T22:51:46.6068340Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2025-08-14T22:51:46.6070983Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-08-14T22:51:46.6072892Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-08-14T22:51:46.6074671Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-08-14T22:51:46.6076463Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-08-14T22:51:46.6078282Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-08-14T22:51:46.6080058Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-08-14T22:51:46.6082216Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-08-14T22:51:46.6084150Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-08-14T22:51:46.6086375Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:51:46.6088868Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:51:46.6091417Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-08-14T22:51:46.6093449Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-08-14T22:51:46.6095486Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2025-08-14T22:51:46.6097509Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2025-08-14T22:51:46.6099414Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-08-14T22:51:46.6101126Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-08-14T22:51:46.6102727Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-08-14T22:51:46.6104339Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2025-08-14T22:51:46.6106093Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:51:46.6107142Z 2025-08-14T22:51:46.6107542Z Running distributed tests for the nccl backend with file init_method 2025-08-14T22:51:46.6108352Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:51:46.6110458Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:51:46.599660] 2025-08-14T22:55:34.8711035Z 2025-08-14T22:55:34.8712907Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_565bef4261123afa_.log 2025-08-14T22:55:34.8742482Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:55:34.8771681Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-08-14T22:55:34.8773754Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-08-14T22:55:34.8776030Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2025-08-14T22:55:34.8778262Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-08-14T22:55:34.8780391Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-08-14T22:55:34.8782561Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2025-08-14T22:55:34.8785071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-08-14T22:55:34.8787238Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-08-14T22:55:34.8789250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-08-14T22:55:34.8791152Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2025-08-14T22:55:34.8793174Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-08-14T22:55:34.8795363Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2025-08-14T22:55:34.8797578Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-08-14T22:55:34.8799563Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-08-14T22:55:34.8801753Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-08-14T22:55:34.8803623Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-08-14T22:55:34.8805863Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-08-14T22:55:34.8807755Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-08-14T22:55:34.8809635Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-08-14T22:55:34.8811674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-08-14T22:55:34.8814011Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:55:34.8816606Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:55:34.8818934Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-08-14T22:55:34.8821035Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-08-14T22:55:34.8823168Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2025-08-14T22:55:34.8825276Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2025-08-14T22:55:34.8827281Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-08-14T22:55:34.8829074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-08-14T22:55:34.8830767Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-08-14T22:55:34.8832434Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2025-08-14T22:55:34.8834250Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:55:34.8835684Z 2025-08-14T22:55:34.8836106Z Running distributed tests for the gloo backend with env init_method 2025-08-14T22:55:34.8836929Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:55:34.8839116Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:55:34.874361] 2025-08-14T22:59:53.7790224Z 2025-08-14T22:59:53.7791860Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_f809debafcf0e15a_.log 2025-08-14T22:59:53.7821068Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:59:53.7848784Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-08-14T22:59:53.7850824Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-08-14T22:59:53.7853005Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2025-08-14T22:59:53.7855132Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-08-14T22:59:53.7857158Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-08-14T22:59:53.7859155Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2025-08-14T22:59:53.7861388Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-08-14T22:59:53.7863390Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-08-14T22:59:53.7865273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-08-14T22:59:53.7867071Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2025-08-14T22:59:53.7869273Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-08-14T22:59:53.7871382Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2025-08-14T22:59:53.7873490Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-08-14T22:59:53.7875359Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-08-14T22:59:53.7877074Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-08-14T22:59:53.7878823Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-08-14T22:59:53.7880873Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-08-14T22:59:53.7882693Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-08-14T22:59:53.7884718Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-08-14T22:59:53.7886617Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-08-14T22:59:53.7888793Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:59:53.7891227Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-08-14T22:59:53.7893412Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-08-14T22:59:53.7895373Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-08-14T22:59:53.7897726Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2025-08-14T22:59:53.7899758Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2025-08-14T22:59:53.7901847Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-08-14T22:59:53.7903526Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-08-14T22:59:53.7905120Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-08-14T22:59:53.7906692Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2025-08-14T22:59:53.7908404Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T22:59:53.7909433Z 2025-08-14T22:59:53.7909834Z Running distributed tests for the gloo backend with file init_method 2025-08-14T22:59:53.7910624Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T22:59:53.7912695Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '--shard-id=7', '--num-shards=8', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 22:59:53.782363] 2025-08-14T23:04:13.0189360Z 2025-08-14T23:04:13.0191463Z distributed/test_distributed_spawn 7/8 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_distributed_spawn_7.8_23fbb35510c53d3a_.log 2025-08-14T23:04:13.0207919Z Running 31 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv, test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T23:04:13.0224139Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_Backend_enum_class 2025-08-14T23:04:13.0225313Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallelCPU_grad_is_view 2025-08-14T23:04:13.0226619Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_DistributedDataParallel_SyncBatchNorm_2D_Input 2025-08-14T23:04:13.0228498Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_max 2025-08-14T23:04:13.0230880Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_full_group_sum 2025-08-14T23:04:13.0232867Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_max 2025-08-14T23:04:13.0234839Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_coalesced_group_product 2025-08-14T23:04:13.0236820Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_complex_unsupported_ops 2025-08-14T23:04:13.0238702Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_reduce_group_min 2025-08-14T23:04:13.0240621Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_group_cuda 2025-08-14T23:04:13.0242518Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda 2025-08-14T23:04:13.0244596Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_equal_split_cuda_complex 2025-08-14T23:04:13.0246674Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_all_to_all_single_unequal_split_cuda 2025-08-14T23:04:13.0248806Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_average_parameters 2025-08-14T23:04:13.0250895Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_full_group 2025-08-14T23:04:13.0252996Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_barrier_timeout_global 2025-08-14T23:04:13.0254103Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_broadcast_object_list 2025-08-14T23:04:13.0254986Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_coalescing_manager 2025-08-14T23:04:13.0256066Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_compile_static_graph 2025-08-14T23:04:13.0257036Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_model_diff_shape_across_ranks 2025-08-14T23:04:13.0258149Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_grad_as_bucket_view_set_grad_to_none 2025-08-14T23:04:13.0259400Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_ddp_native_mixed_precision_no_grad_as_bucket_view_set_grad_to_none 2025-08-14T23:04:13.0260525Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_detect_ddp_is_actually_static 2025-08-14T23:04:13.0261527Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_grads_same_across_ranks_with_no_sync 2025-08-14T23:04:13.0262544Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_output_unused_in_loss_tuple_module 2025-08-14T23:04:13.0263553Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_periodic_model_averager_param_group 2025-08-14T23:04:13.0264519Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_group_product 2025-08-14T23:04:13.0265376Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_reduce_max 2025-08-14T23:04:13.0266348Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_scatter 2025-08-14T23:04:13.0267160Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv 2025-08-14T23:04:13.0268043Z Running 1 items in this shard: test/distributed/test_distributed_spawn.py::TestDistBackendWithSpawn::test_send_recv_torch_profiler 2025-08-14T23:04:13.0268560Z 2025-08-14T23:04:13.0268810Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T23:04:13.0269238Z Uploading artifacts took 0.00 seconds 2025-08-14T23:04:13.0269686Z Running distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 ... [2025-08-14 23:04:13.020304] 2025-08-14T23:04:13.0270152Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:04:13.0271197Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:04:13.020697] 2025-08-14T23:06:15.0724382Z 2025-08-14T23:06:15.0726497Z distributed/fsdp/test_fsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_dtensor_state_dict_1.1_01fb0c59e9ec2b65_.log 2025-08-14T23:06:15.0751358Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_False_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_fsdp_init_with_device_mesh_is_even_sharded_model_True_cuda, test/distributed/fsdp/test_fsdp_dtensor_state_dict.py::TestFSDPWithDeviceMeshAndDTensorCUDA::test_raises_warning_or_errors_cuda 2025-08-14T23:06:15.0773926Z 2025-08-14T23:06:15.0774413Z Running distributed/fsdp/test_fsdp_fine_tune 1/1 ... [2025-08-14 23:06:15.073231] 2025-08-14T23:06:15.0775314Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:06:15.0777289Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_fine_tune.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:06:15.073634] 2025-08-14T23:07:00.0468118Z 2025-08-14T23:07:00.0470321Z distributed/fsdp/test_fsdp_fine_tune 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_fine_tune_1.1_23ef23b67d344019_.log 2025-08-14T23:07:00.0476002Z Running 4 items in this shard: test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_backward_reshard_hooks_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_hooks_multi_traversal_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_ddp_cuda, test/distributed/fsdp/test_fsdp_fine_tune.py::TestFSDPFineTuneCUDA::test_parity_with_non_frozen_fsdp_cuda 2025-08-14T23:07:00.0479382Z 2025-08-14T23:07:00.0479890Z Running distributed/fsdp/test_fsdp_comm_hooks 1/1 ... [2025-08-14 23:07:00.047288] 2025-08-14T23:07:00.0480949Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:07:00.0484918Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:07:00.047872] 2025-08-14T23:10:17.9793131Z 2025-08-14T23:10:17.9794963Z distributed/fsdp/test_fsdp_comm_hooks 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_hooks_1.1_b4aba19a8ba9150e_.log 2025-08-14T23:10:17.9826523Z Running 28 items in this shard: test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_bf16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_behavior_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_False_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_fp16_hook_has_wrapping_True_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_hybrid_strategy, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_non_root_sharding_strategy2, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy0, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy1, test/distributed/fsdp/test_fsdp_comm_hooks.py::TestCommunicationHooks::test_registering_hook_submodules_sharding_strategy2 2025-08-14T23:10:17.9855117Z 2025-08-14T23:10:17.9855565Z Running distributed/fsdp/test_fsdp_meta 1/1 ... [2025-08-14 23:10:17.980213] 2025-08-14T23:10:17.9856438Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:10:17.9858414Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_meta.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:10:17.980833] 2025-08-14T23:11:17.4366510Z 2025-08-14T23:11:17.4384864Z distributed/fsdp/test_fsdp_meta 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_meta_1.1_7326e4debbf0f8c1_.log 2025-08-14T23:11:17.4399793Z Running 15 items in this shard: test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_meta, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_bad_arg_torchdistx, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_meta_device_with_mixed_precision, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_meta_device_reset_params_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_default_init_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_False, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_nested_model_with_torchdistX_init_fn_auto_wrap_True, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_meta_device_reset_params, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_default_init, test/distributed/fsdp/test_fsdp_meta.py::TestFSDPWithMetaDevice::test_simple_model_with_torchdistX_init_fn 2025-08-14T23:11:17.4413456Z 2025-08-14T23:11:17.4414042Z Running distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 ... [2025-08-14 23:11:17.436925] 2025-08-14T23:11:17.4415009Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:11:17.4417085Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_hsdp_dtensor_state_dict.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:11:17.437537] 2025-08-14T23:12:43.9066280Z 2025-08-14T23:12:43.9068891Z distributed/fsdp/test_hsdp_dtensor_state_dict 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_hsdp_dtensor_state_dict_1.1_1645da5de6c2c3d3_.log 2025-08-14T23:12:43.9082422Z Running 8 items in this shard: test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_model_load_state_dict_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_optim_load_state_dict_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_False_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_dtensor_sharded_tensor_state_dict_identical_offload_to_cpu_True_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_hsdp_init_with_device_mesh_cuda, test/distributed/fsdp/test_hsdp_dtensor_state_dict.py::TestHSDPWithDeviceMeshAndDTensorCUDA::test_root_module_is_not_FSDP_cuda 2025-08-14T23:12:43.9117341Z 2025-08-14T23:12:43.9117918Z Running distributed/fsdp/test_fsdp_comm 1/1 ... [2025-08-14 23:12:43.907184] 2025-08-14T23:12:43.9118857Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:12:43.9121042Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:12:43.907780] 2025-08-14T23:14:08.6233943Z 2025-08-14T23:14:08.6236301Z distributed/fsdp/test_fsdp_comm 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_comm_1.1_6008646662b5c2df_.log 2025-08-14T23:14:08.6248826Z Running 10 items in this shard: test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_False_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_False_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy0_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestCommunicationCUDA::test_communication_nested_model_True_use_no_sync_True_sharding_strategy1_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_False_cuda, test/distributed/fsdp/test_fsdp_comm.py::TestExplicitUnshardCUDA::test_unshard_async_use_orig_params_True_cuda 2025-08-14T23:14:08.6259660Z 2025-08-14T23:14:08.6260106Z Running distributed/test_data_parallel 1/1 ... [2025-08-14 23:14:08.623925] 2025-08-14T23:14:08.6260967Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:14:08.6262932Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_data_parallel.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:14:08.624581] 2025-08-14T23:14:39.8172976Z 2025-08-14T23:14:39.8174593Z distributed/test_data_parallel 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_data_parallel_1.1_3c48cd8b0c99d690_.log 2025-08-14T23:14:39.8214963Z Running 46 items in this shard: test/distributed/test_data_parallel.py::TestDataParallel::test_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_buffers_requiring_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_complex, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_device_args, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_function_deletion, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_lazy_linear, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_device, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_model_no_refcycles, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_module_zero_inputs, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_multiple_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_input, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_nested_output, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_no_grad, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_rnn, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_small_back, test/distributed/test_data_parallel.py::TestDataParallel::test_data_parallel_sparse, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_different_len_dicts, test/distributed/test_data_parallel.py::TestDataParallel::test_gather_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_autocast, test/distributed/test_data_parallel.py::TestDataParallel::test_parallel_apply_passes_exception, test/distributed/test_data_parallel.py::TestDataParallel::test_parameter_list_dict_replica, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate, test/distributed/test_data_parallel.py::TestDataParallel::test_replicate_buffers, test/distributed/test_data_parallel.py::TestDataParallel::test_save_replica_module, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_cpu, test/distributed/test_data_parallel.py::TestDataParallel::test_scatter_gpu, test/distributed/test_data_parallel.py::TestDataParallel::test_strided_grad_layout, test/distributed/test_data_parallel.py::TestDataParallel::test_zero_grad, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_dict_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_list_cuda_float64, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float16, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float32, test/distributed/test_data_parallel.py::TestDataParallelDeviceTypeCUDA::test_data_parallel_module_kwargs_only_empty_tuple_cuda_float64 2025-08-14T23:14:39.8252445Z 2025-08-14T23:14:39.8253155Z Running distributed/_shard/sharding_plan/test_sharding_plan 1/1 ... [2025-08-14 23:14:39.817710] 2025-08-14T23:14:39.8254455Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:14:39.8256907Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharding_plan/test_sharding_plan.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:14:39.818317] 2025-08-14T23:15:04.0409645Z 2025-08-14T23:15:04.0411966Z distributed/_shard/sharding_plan/test_sharding_plan 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharding_plan.test_sharding_plan_1.1_dd22f186bea7d1c4_.log 2025-08-14T23:15:04.0416381Z Running 3 items in this shard: test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_custom_sharding_planner, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_shard_module_sub_process_group, test/distributed/_shard/sharding_plan/test_sharding_plan.py::TestShardingPlan::test_sharding_plan_errors 2025-08-14T23:15:04.0419149Z 2025-08-14T23:15:04.0419675Z Running distributed/nn/jit/test_instantiator 1/1 ... [2025-08-14 23:15:04.041437] 2025-08-14T23:15:04.0420611Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:15:04.0423242Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/nn/jit/test_instantiator.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:15:04.041838] 2025-08-14T23:15:07.9169634Z 2025-08-14T23:15:07.9171318Z distributed/nn/jit/test_instantiator 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.nn.jit.test_instantiator_1.1_d7b32df01954ac6c_.log 2025-08-14T23:15:07.9175522Z Running 3 items in this shard: test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_get_arg_return_types_from_interface, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_non_scripted_remote_module_template, test/distributed/nn/jit/test_instantiator.py::TestInstantiator::test_instantiate_scripted_remote_module_template 2025-08-14T23:15:07.9179450Z 2025-08-14T23:15:07.9179934Z Running distributed/test_pg_wrapper 1/1 ... [2025-08-14 23:15:07.917160] 2025-08-14T23:15:07.9180797Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:15:07.9182944Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_pg_wrapper.py', '--shard-id=1', '--num-shards=1', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:15:07.917732] 2025-08-14T23:17:47.3520132Z 2025-08-14T23:17:47.3521867Z distributed/test_pg_wrapper 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_pg_wrapper_1.1_d77ef80aaa7d1b50_.log 2025-08-14T23:17:47.3537206Z Running 17 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode, test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2025-08-14T23:17:47.3553376Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_coalescing_manager_debug_mode_detail 2025-08-14T23:17:47.3555394Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_hang 2025-08-14T23:17:47.3557255Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_detail 2025-08-14T23:17:47.3559318Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collective_shape_mismatch_debug_mode_off 2025-08-14T23:17:47.3561358Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch 2025-08-14T23:17:47.3563281Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_collectives_op_mismatch_debug_mode 2025-08-14T23:17:47.3565163Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_debug_level_detail_no_gloo 2025-08-14T23:17:47.3566887Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupNCCLWrapperTest::test_new_group_no_gloo 2025-08-14T23:17:47.3568509Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_hang 2025-08-14T23:17:47.3570720Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda 2025-08-14T23:17:47.3572798Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_cuda_debug_mode 2025-08-14T23:17:47.3574832Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode 2025-08-14T23:17:47.3576834Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collective_shape_mismatch_debug_mode_off 2025-08-14T23:17:47.3578735Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch 2025-08-14T23:17:47.3580516Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda 2025-08-14T23:17:47.3582451Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_cuda_debug_mode 2025-08-14T23:17:47.3584495Z Running 1 items in this shard: test/distributed/test_pg_wrapper.py::ProcessGroupGlooWrapperTest::test_collectives_op_mismatch_debug_mode 2025-08-14T23:17:47.3585565Z 2025-08-14T23:17:47.3585998Z Running distributed/fsdp/test_fsdp_core 1/3 ... [2025-08-14 23:17:47.353240] 2025-08-14T23:17:47.3586825Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:17:47.3588761Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_core.py', '--shard-id=1', '--num-shards=3', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:17:47.353838] 2025-08-14T23:24:35.4440878Z 2025-08-14T23:24:35.4442589Z distributed/fsdp/test_fsdp_core 1/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_core_1.3_d068581dc45cdee1_.log 2025-08-14T23:24:35.4469084Z Running 19 items in this shard: test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_after_state_dict_cuda, test/distributed/fsdp/test_fsdp_core.py::TestHooksCUDA::test_pre_backward_hook_registration_cuda_first_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_optim_step_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_delayed_reduce_scatter_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_mixture_of_experts_with_delay_before_free_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_always_wrap_model_offload_true_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_offload_true_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_no_shard_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_nested_wrapped_model_single_iteration_mixed_precision_offload_false_none_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParityWithDDPCUDA::test_transformer_offload_false_shard_grad_op_cuda, test/distributed/fsdp/test_fsdp_core.py::TestNoGradCUDA::test_transformer_no_grad_mixed_precision_True_cuda, test/distributed/fsdp/test_fsdp_core.py::TestParamInitCUDA::test_param_change_after_init_mixed_precision_False_cuda 2025-08-14T23:24:35.4487631Z 2025-08-14T23:24:35.4488176Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T23:24:35.4489061Z Uploading artifacts took 0.00 seconds 2025-08-14T23:24:35.4490898Z Running distributed/_shard/sharded_optim/test_sharded_optim 1/1 ... [2025-08-14 23:24:35.448616] 2025-08-14T23:24:35.4491914Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:24:35.4499347Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_optim/test_sharded_optim.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:24:35.449274] 2025-08-14T23:24:52.4055372Z 2025-08-14T23:24:52.4057466Z distributed/_shard/sharded_optim/test_sharded_optim 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_optim.test_sharded_optim_1.1_95ab1427cd5d0b21_.log 2025-08-14T23:24:52.4061099Z Running 2 items in this shard: test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_named_params_with_sharded_tensor, test/distributed/_shard/sharded_optim/test_sharded_optim.py::TestShardedOptimizer::test_sharded_optim 2025-08-14T23:24:52.4063102Z 2025-08-14T23:24:52.4065709Z Running distributed/fsdp/test_fsdp_multiple_forward 1/1 ... [2025-08-14 23:24:52.406045] 2025-08-14T23:24:52.4066696Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:24:52.4073643Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/fsdp/test_fsdp_multiple_forward.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:24:52.406673] 2025-08-14T23:24:56.2545920Z 2025-08-14T23:24:56.2547649Z distributed/fsdp/test_fsdp_multiple_forward 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.fsdp.test_fsdp_multiple_forward_1.1_ea776cf330027201_.log 2025-08-14T23:24:56.2550311Z Running 0 items in this shard: 2025-08-14T23:24:56.2550719Z 2025-08-14T23:24:56.2553477Z Running distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 ... [2025-08-14 23:24:56.254724] 2025-08-14T23:24:56.2554596Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:24:56.2559133Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_binary_cmp.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:24:56.255302] 2025-08-14T23:25:27.0955271Z 2025-08-14T23:25:27.0961828Z distributed/_shard/sharded_tensor/ops/test_binary_cmp 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_binary_cmp_1.1_c3d1a3c1a8e80936_.log 2025-08-14T23:25:27.0966182Z Running 4 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_allclose_tensor_specs, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal, test/distributed/_shard/sharded_tensor/ops/test_binary_cmp.py::TestShardedTensorBinaryOps::test_torch_equal_tensor_specs 2025-08-14T23:25:27.0967774Z 2025-08-14T23:25:27.0968041Z Running distributed/_composable/fsdp/test_fully_shard_memory 1/1 ... [2025-08-14 23:25:27.096139] 2025-08-14T23:25:27.0968452Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:25:27.0983722Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_composable/fsdp/test_fully_shard_memory.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:25:27.096703] 2025-08-14T23:25:49.4630248Z 2025-08-14T23:25:49.4632826Z distributed/_composable/fsdp/test_fully_shard_memory 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._composable.fsdp.test_fully_shard_memory_1.1_f4ddb39d1c90850d_.log 2025-08-14T23:25:49.4639334Z Running 2 items in this shard: test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_del_memory, test/distributed/_composable/fsdp/test_fully_shard_memory.py::TestFullyShardMemory::test_fully_shard_training_memory 2025-08-14T23:25:49.4641836Z 2025-08-14T23:25:49.4644750Z Running distributed/_shard/sharded_tensor/ops/test_init 1/1 ... [2025-08-14 23:25:49.463209] 2025-08-14T23:25:49.4645971Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:25:49.4648668Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_init.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:25:49.463837] 2025-08-14T23:26:13.8379384Z 2025-08-14T23:26:13.8381578Z distributed/_shard/sharded_tensor/ops/test_init 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_init_1.1_f8121bfafbc0a7d6_.log 2025-08-14T23:26:13.8388396Z Running 3 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_kaiming_uniform, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_normal, test/distributed/_shard/sharded_tensor/ops/test_init.py::TestShardedTensorNNInit::test_init_sharded_tensor_with_uniform 2025-08-14T23:26:13.8391732Z 2025-08-14T23:26:13.8392435Z Running distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 ... [2025-08-14 23:26:13.838339] 2025-08-14T23:26:13.8395245Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:26:13.8397510Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:26:13.838925] 2025-08-14T23:26:30.6932400Z 2025-08-14T23:26:30.6934982Z distributed/_shard/sharded_tensor/test_sharded_tensor_reshard 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_reshard_1.1_27f4eb2a458f265f_.log 2025-08-14T23:26:30.6939046Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard, test/distributed/_shard/sharded_tensor/test_sharded_tensor_reshard.py::TestReshard::test_sharded_tensor_reshard_errors 2025-08-14T23:26:30.6941217Z 2025-08-14T23:26:30.6941885Z Running distributed/test_c10d_gloo 1/2 ... [2025-08-14 23:26:30.693675] 2025-08-14T23:26:30.6942704Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:26:30.6949384Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_gloo.py', '--shard-id=1', '--num-shards=2', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:26:30.694294] 2025-08-14T23:42:53.9276631Z 2025-08-14T23:42:53.9277540Z distributed/test_c10d_gloo 1/2 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_gloo_1.2_cce6a06ad5ed460d_.log 2025-08-14T23:42:53.9353146Z Running 126 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousTCPTest::test_tcp_init, test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_overall_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view, test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward, test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_into_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_checks_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_op_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_overall_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_block_current_stream_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_empty_tensors, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_noncontiguous_input, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter_tensor, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_basics, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_coalesced_async, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_coalesced_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_inference_mode, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_op_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_overall_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_block_current_stream_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_stress, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_checks, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor_coalesced, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_send_recv_all_to_all, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_set_gloo_pg_timeout, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_json, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics_cuda, test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_checks, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu, test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda, test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo, test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group, test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced, test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2025-08-14T23:42:53.9444795Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::RendezvousTCPTest::test_tcp_init 2025-08-14T23:42:53.9446394Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::TimeoutTest::test_default_store_timeout_gloo 2025-08-14T23:42:53.9448030Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_async 2025-08-14T23:42:53.9449722Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_coalesced_checks 2025-08-14T23:42:53.9451621Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_into_tensor_coalesced 2025-08-14T23:42:53.9453689Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress 2025-08-14T23:42:53.9455301Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allgather_stress_cuda 2025-08-14T23:42:53.9456892Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics 2025-08-14T23:42:53.9458445Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_basics_cuda 2025-08-14T23:42:53.9459997Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_checks 2025-08-14T23:42:53.9461586Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_basics 2025-08-14T23:42:53.9463308Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_coalesced_checks_cuda 2025-08-14T23:42:53.9465043Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_overall_timeout 2025-08-14T23:42:53.9466619Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_allreduce_stress 2025-08-14T23:42:53.9468184Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_barrier_implies_wait 2025-08-14T23:42:53.9469723Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics 2025-08-14T23:42:53.9471252Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_basics_cuda 2025-08-14T23:42:53.9472771Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress 2025-08-14T23:42:53.9474305Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_broadcast_stress_cuda 2025-08-14T23:42:53.9475832Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_empty_tensors 2025-08-14T23:42:53.9477338Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_gather_stress_cuda 2025-08-14T23:42:53.9478898Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_multi_device_constructor 2025-08-14T23:42:53.9481280Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_basics 2025-08-14T23:42:53.9482897Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress 2025-08-14T23:42:53.9484490Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_reduce_stress_cuda 2025-08-14T23:42:53.9486045Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics 2025-08-14T23:42:53.9487615Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_basics_cuda 2025-08-14T23:42:53.9489177Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_scatter_stress 2025-08-14T23:42:53.9490733Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_send_recv_all_to_all 2025-08-14T23:42:53.9492322Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_set_gloo_pg_timeout 2025-08-14T23:42:53.9493931Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics 2025-08-14T23:42:53.9495609Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooTest::test_sparse_allreduce_basics_cuda 2025-08-14T23:42:53.9497330Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_dataclass_output 2025-08-14T23:42:53.9499506Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_module 2025-08-14T23:42:53.9501591Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_dynamic_weight_sharing 2025-08-14T23:42:53.9503712Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_False 2025-08-14T23:42:53.9505815Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2025-08-14T23:42:53.9508003Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2025-08-14T23:42:53.9510210Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_False 2025-08-14T23:42:53.9512315Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2025-08-14T23:42:53.9514464Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2025-08-14T23:42:53.9516689Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2025-08-14T23:42:53.9518762Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_cpu 2025-08-14T23:42:53.9520870Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_future_passing_gpu_gloo 2025-08-14T23:42:53.9522860Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ddp_comm_hook_sparse_gradients 2025-08-14T23:42:53.9524795Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad 2025-08-14T23:42:53.9526887Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_global_local_unused_params_grad_with_grad_is_view 2025-08-14T23:42:53.9529275Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module 2025-08-14T23:42:53.9531176Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_gloo_backend_cpu_module_grad_is_view 2025-08-14T23:42:53.9533009Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_output 2025-08-14T23:42:53.9534722Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_ignored_sharded_tensor 2025-08-14T23:42:53.9536441Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients 2025-08-14T23:42:53.9538225Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sparse_gradients_grad_is_view 2025-08-14T23:42:53.9540086Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2025-08-14T23:42:53.9541744Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_forward_backward 2025-08-14T23:42:53.9543226Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_multi_dtype_single_bucket 2025-08-14T23:42:53.9544744Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ReducerTest::test_single_dtype_single_bucket 2025-08-14T23:42:53.9546335Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_checks 2025-08-14T23:42:53.9548110Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_coalesced_checks 2025-08-14T23:42:53.9550274Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_inference_mode 2025-08-14T23:42:53.9552153Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_into_tensor_coalesced 2025-08-14T23:42:53.9553967Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allgather_stress 2025-08-14T23:42:53.9555679Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_basics_cuda 2025-08-14T23:42:53.9557376Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_checks 2025-08-14T23:42:53.9559117Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_async 2025-08-14T23:42:53.9561076Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_checks 2025-08-14T23:42:53.9562965Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_checks_cuda 2025-08-14T23:42:53.9564853Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_coalesced_stress 2025-08-14T23:42:53.9566655Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_op_timeout 2025-08-14T23:42:53.9568437Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_overall_timeout 2025-08-14T23:42:53.9570180Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_stress 2025-08-14T23:42:53.9571880Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_allreduce_stress_cuda 2025-08-14T23:42:53.9573674Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_block_current_stream_cuda 2025-08-14T23:42:53.9575416Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_basics 2025-08-14T23:42:53.9577435Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_stress 2025-08-14T23:42:53.9579135Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_broadcast_stress_cuda 2025-08-14T23:42:53.9580839Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_empty_tensors 2025-08-14T23:42:53.9582481Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_basics 2025-08-14T23:42:53.9584088Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_checks 2025-08-14T23:42:53.9585826Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_noncontiguous_input 2025-08-14T23:42:53.9587604Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_gather_stress_cuda 2025-08-14T23:42:53.9589321Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter_tensor 2025-08-14T23:42:53.9591151Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_scatter_tensor_coalesced 2025-08-14T23:42:53.9592917Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_reduce_stress 2025-08-14T23:42:53.9594527Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_basics 2025-08-14T23:42:53.9596142Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_scatter_checks 2025-08-14T23:42:53.9609843Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_send_recv_all_to_all 2025-08-14T23:42:53.9611762Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_set_gloo_pg_timeout 2025-08-14T23:42:53.9613521Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooLazyInitTest::test_sparse_allreduce_basics 2025-08-14T23:42:53.9614941Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_basics 2025-08-14T23:42:53.9616287Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_basics_cuda 2025-08-14T23:42:53.9617686Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_coalesced_async 2025-08-14T23:42:53.9619113Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_coalesced_checks 2025-08-14T23:42:53.9620528Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_inference_mode 2025-08-14T23:42:53.9621899Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_stress 2025-08-14T23:42:53.9623249Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allgather_stress_cuda 2025-08-14T23:42:53.9624635Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_basics_cuda 2025-08-14T23:42:53.9626038Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_coalesced_stress 2025-08-14T23:42:53.9627432Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_op_timeout 2025-08-14T23:42:53.9628828Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_overall_timeout 2025-08-14T23:42:53.9630197Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_allreduce_stress 2025-08-14T23:42:53.9631549Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_block_current_stream_cuda 2025-08-14T23:42:53.9633237Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_stress 2025-08-14T23:42:53.9634578Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_broadcast_stress_cuda 2025-08-14T23:42:53.9635926Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_basics_cuda 2025-08-14T23:42:53.9637229Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_checks 2025-08-14T23:42:53.9638522Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_gather_stress_cuda 2025-08-14T23:42:53.9639816Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_checks 2025-08-14T23:42:53.9641180Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter 2025-08-14T23:42:53.9642606Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_reduce_scatter_tensor_coalesced 2025-08-14T23:42:53.9644049Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_basics_cuda 2025-08-14T23:42:53.9645397Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_scatter_stress_cuda 2025-08-14T23:42:53.9646734Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_send_recv_all_to_all 2025-08-14T23:42:53.9648052Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_set_gloo_pg_timeout 2025-08-14T23:42:53.9649652Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_short_json 2025-08-14T23:42:53.9651038Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_basics_cuda 2025-08-14T23:42:53.9652481Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::ProcessGroupGlooFRTest::test_sparse_allreduce_checks 2025-08-14T23:42:53.9653802Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cpu 2025-08-14T23:42:53.9655065Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_broadcast_coalesced_gloo_cuda 2025-08-14T23:42:53.9656309Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_gloo_rank_membership 2025-08-14T23:42:53.9657564Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_default_pg_gloo 2025-08-14T23:42:53.9658899Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_sequence_num_set_gloo_new_group 2025-08-14T23:42:53.9660174Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::CommTest::test_tensor_dtype_complex 2025-08-14T23:42:53.9661694Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_allgather_coalesced 2025-08-14T23:42:53.9663632Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::GlooProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2025-08-14T23:42:53.9665344Z Running 1 items in this shard: test/distributed/test_c10d_gloo.py::LargeCommTest::test_new_group_local_sync_duplicate_pg 2025-08-14T23:42:53.9666136Z 2025-08-14T23:42:53.9666647Z Running distributed/_shard/sharded_tensor/ops/test_embedding 1/1 ... [2025-08-14 23:42:53.934921] 2025-08-14T23:42:53.9667511Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:42:53.9669384Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:42:53.935518] 2025-08-14T23:43:12.0427448Z 2025-08-14T23:43:12.0429442Z distributed/_shard/sharded_tensor/ops/test_embedding 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.ops.test_embedding_1.1_7c832c57bec3609c_.log 2025-08-14T23:43:12.0433108Z Running 2 items in this shard: test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_colwise, test/distributed/_shard/sharded_tensor/ops/test_embedding.py::TestShardedEmbedding::test_sharded_embedding_rowwise 2025-08-14T23:43:12.0435201Z 2025-08-14T23:43:12.0436429Z Running distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 ... [2025-08-14 23:43:12.043190] 2025-08-14T23:43:12.0437488Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:43:12.0444496Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_sharded_tensor.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:43:12.043790] 2025-08-14T23:49:47.2937316Z 2025-08-14T23:49:47.2939297Z distributed/_shard/sharded_tensor/test_sharded_tensor 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed._shard.sharded_tensor.test_sharded_tensor_1.1_84dacbe7f22a66ec_.log 2025-08-14T23:49:47.3013890Z Running 74 items in this shard: test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorMetadata::test_serialize_and_deserialize, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorFromParams::test_empty, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardParameter::test_shard_parameter_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardTensor::test_shard_tensor_with_empty_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_collect_local_shard, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestModuleHookApi::test_reshard_output, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestLocalTensor::test_local_tensor_error, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_cleanup, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_complete_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_like, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_full, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_rand, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_create_sharded_tensor_with_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_insufficient_sharding_dims, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_pg_rpc_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_invalid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_load_state_dict_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharded_tensor_sizes, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_sharding_columns, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorChunked::test_state_dict_no_sharded_tensors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_create_sharded_tensor_with_ones, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_even, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_gather_uneven, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_grid_sharding, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_multiple_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_partial_world_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_device, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cpu, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_cuda, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_sharded_tensor_to_test, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_uneven_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorEnumerable::test_with_rpc_names, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalTensor::test_init_from_local_tensor_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_invalid_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_with_all_zeros, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_and_global_metadata_with_local_view, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_pin_memory, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_property_cross_ranks, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_gaps, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_invalid_shards_overlap, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_new_group, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_init_from_local_shards_with_different_glb_size, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_local_shards, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_non_rw_sharded_recalc_for_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_recalc_for_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorFromLocalShards::test_st_base_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_errors, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorCustomOps::test_custom_op_override, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_create_shard_with_no_placement, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardMetadata::test_shard_metadata_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_placement_validation, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestShardedTensorSubGroupInit::test_sub_process_group_sharded_tensor_init, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_init_from_local_shards_and_global_metadata, test/distributed/_shard/sharded_tensor/test_sharded_tensor.py::TestCreateTensorNoProcessGroupMode::test_non_contiguous_local_shards 2025-08-14T23:49:47.3083875Z 2025-08-14T23:49:47.3084349Z GITHUB_RUN_ID, GITHUB_RUN_ATTEMPT, or ARTIFACTS_FILE_SUFFIX not set, not uploading 2025-08-14T23:49:47.3085216Z Uploading artifacts took 0.00 seconds 2025-08-14T23:49:47.3085971Z Running distributed/test_c10d_nccl 3/3 ... [2025-08-14 23:49:47.297334] 2025-08-14T23:49:47.3086753Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-14T23:49:47.3088771Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/test_c10d_nccl.py', '--shard-id=3', '--num-shards=3', '-v', '--subprocess', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-14 23:49:47.297953] 2025-08-15T00:00:50.8172589Z 2025-08-15T00:00:50.8179061Z distributed/test_c10d_nccl 3/3 was successful, full logs can be found in artifacts with path test/test-reports/distributed.test_c10d_nccl_3.3_f2fe68b15865d620_.log 2025-08-15T00:00:50.8227978Z Running 64 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context_sync_ops, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl, test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input, test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier, test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking, test/distributed/test_c10d_nccl.py::NcclRegistrationTest::test_nccl_user_buffer_registration, test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail, test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info, test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex, test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch, test/distributed/test_c10d_nccl.py::CommTest::test_time_estimate_nccl, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced, test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends, test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg, test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check, test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True, test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False, test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False, test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2025-08-15T00:00:50.8275307Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLInitTest::test_init_wo_backend_str 2025-08-15T00:00:50.8277384Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_abort_in_destroy_mixed_empty_pgs 2025-08-15T00:00:50.8279163Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_comm_eager_init_subgroup 2025-08-15T00:00:50.8281233Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context 2025-08-15T00:00:50.8283085Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_extra_cuda_context_sync_ops 2025-08-15T00:00:50.8284787Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_get_uid 2025-08-15T00:00:50.8286271Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nan_assert_float32 2025-08-15T00:00:50.8287868Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_nccl_dist_backend_error 2025-08-15T00:00:50.8289569Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_non_blocking_with_eager_init 2025-08-15T00:00:50.8291244Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_restart_pg 2025-08-15T00:00:50.8292913Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_set_nccl_pg_timeout_backend_nccl 2025-08-15T00:00:50.8294700Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::ProcessGroupNCCLGroupTest::test_subgroup_p2p_eager_init_False 2025-08-15T00:00:50.8296579Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_arbitrary_forward_return_value_grad_is_view 2025-08-15T00:00:50.8298461Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_builtin_ddp_comm_hooks_nccl 2025-08-15T00:00:50.8300441Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_channels_last_contig 2025-08-15T00:00:50.8302156Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_dataclass_output_unused_param 2025-08-15T00:00:50.8304406Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_once_use_reentrant_True 2025-08-15T00:00:50.8306474Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_False 2025-08-15T00:00:50.8308630Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_static_graph_use_reentrant_True 2025-08-15T00:00:50.8310706Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_use_reentrant_True 2025-08-15T00:00:50.8312662Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_twice_weight_sharing 2025-08-15T00:00:50.8314678Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_unused_params_use_reentrant_True 2025-08-15T00:00:50.8316767Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_checkpointing_weight_sharing_use_reentrant_True 2025-08-15T00:00:50.8318715Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_comm_hook_allreduce_hook_nccl 2025-08-15T00:00:50.8320536Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_ddp_packed_sequence 2025-08-15T00:00:50.8322168Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_failure_recovery 2025-08-15T00:00:50.8324313Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_find_unused_parameters_kwarg_grad_is_view_debug_info 2025-08-15T00:00:50.8326300Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_fp16_compress_wrapper_is_view 2025-08-15T00:00:50.8328246Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_multiple_outputs_multiple_backward_grad_is_view 2025-08-15T00:00:50.8330269Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_nccl_backend_multi_device_ids_not_allowed 2025-08-15T00:00:50.8332122Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_empty_input 2025-08-15T00:00:50.8333931Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::DistributedDataParallelTest::test_sync_batch_norm_only_empty_input 2025-08-15T00:00:50.8335696Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_blocking_wait_with_barrier 2025-08-15T00:00:50.8337308Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclErrorHandlingTest::test_nccl_errors_blocking 2025-08-15T00:00:50.8338925Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclRegistrationTest::test_nccl_user_buffer_registration 2025-08-15T00:00:50.8340495Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_intra_node_comm_all_reduce 2025-08-15T00:00:50.8341956Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_detail 2025-08-15T00:00:50.8343463Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_nccl_warn_not_in_group_debug_info 2025-08-15T00:00:50.8344954Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_sequence_num_set_default_pg_nccl 2025-08-15T00:00:50.8346378Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_complex 2025-08-15T00:00:50.8347736Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_tensor_dtype_mismatch 2025-08-15T00:00:50.8349094Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::CommTest::test_time_estimate_nccl 2025-08-15T00:00:50.8351106Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_all_to_all_single 2025-08-15T00:00:50.8353197Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allgather_float8_float8_e4m3fn 2025-08-15T00:00:50.8355324Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_allreduce_coalesced 2025-08-15T00:00:50.8357482Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NcclProcessGroupWithDispatchedCollectivesTests::test_init_process_group_for_all_backends 2025-08-15T00:00:50.8359623Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_broadcast_object_list_subgroup_set_device1_group_rank_False 2025-08-15T00:00:50.8361518Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_gather_object_subgroup_group_rank_True 2025-08-15T00:00:50.8363447Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_duplicated_pg 2025-08-15T00:00:50.8365083Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_new_group_local_sync_sanity_check 2025-08-15T00:00:50.8366626Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_reduce_subgroup_group_rank_True 2025-08-15T00:00:50.8368169Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_scatter_subgroup_group_rank_False 2025-08-15T00:00:50.8370232Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::LargeCommTest::test_send_recv_subgroup_group_rank_False_async_op_True 2025-08-15T00:00:50.8371989Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_allgather_uneven_timing_enabled_True 2025-08-15T00:00:50.8373536Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_barrier_profiling 2025-08-15T00:00:50.8375267Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_batched_send_recv_op_sizes_per_coalesce0_timing_enabled_True 2025-08-15T00:00:50.8377176Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_False 2025-08-15T00:00:50.8379036Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_coalescing_manager_collective_timing_enabled_True 2025-08-15T00:00:50.8381095Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_individual_send_recv_op_sizes1_timing_enabled_False 2025-08-15T00:00:50.8382983Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_False_include_collectives_False 2025-08-15T00:00:50.8384864Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_short_json_timing_enabled_True_include_collectives_True 2025-08-15T00:00:50.8386762Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_active_timing_enabled_False_only_active_True 2025-08-15T00:00:50.8388534Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_False 2025-08-15T00:00:50.8390185Z Running 1 items in this shard: test/distributed/test_c10d_nccl.py::NCCLTraceTest::test_trace_while_stuck_timing_enabled_True 2025-08-15T00:00:50.8391099Z 2025-08-15T00:00:50.8391663Z Running distributed/tensor/test_dtensor_dispatch_overhead 1/1 ... [2025-08-15 00:00:50.820232] 2025-08-15T00:00:50.8392646Z SCRIBE_GRAPHQL_ACCESS_TOKEN is NOT set 2025-08-15T00:00:50.8394771Z Executing ['/opt/conda/envs/py_3.10/bin/python', '-bb', 'distributed/tensor/test_dtensor_dispatch_overhead.py', '--shard-id=1', '--num-shards=1', '-v', '-vv', '-rfEX', '-p', 'no:xdist', '--use-pytest', '-x', '--reruns=2', '--import-slow-tests', '--import-disabled-tests'] ... [2025-08-15 00:00:50.820832] 2025-08-15T00:01:00.9865264Z 2025-08-15T00:01:00.9868132Z distributed/tensor/test_dtensor_dispatch_overhead 1/1 was successful, full logs can be found in artifacts with path test/test-reports/distributed.tensor.test_dtensor_dispatch_overhead_1.1_1f9d8c229f870556_.log 2025-08-15T00:01:00.9871095Z Running 1 items in this shard: test/distributed/tensor/test_dtensor_dispatch_overhead.py::DistOpDispatchOverHead::test_dtensor_add_op_dispatch_overhead 2025-08-15T00:01:00.9872395Z 2025-08-15T00:01:04.0835659Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.0838554Z import pkg_resources 2025-08-15T00:01:04.2272382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.2275147Z import pkg_resources 2025-08-15T00:01:04.2729904Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.2732659Z import pkg_resources 2025-08-15T00:01:04.2864382Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.2867152Z import pkg_resources 2025-08-15T00:01:04.3071151Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.3073812Z import pkg_resources 2025-08-15T00:01:04.3076378Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.3078942Z import pkg_resources 2025-08-15T00:01:04.3081576Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.3084124Z import pkg_resources 2025-08-15T00:01:04.3114189Z /opt/conda/envs/py_3.10/lib/python3.10/site-packages/hypothesis/entry_points.py:23: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-15T00:01:04.3116922Z import pkg_resources 2025-08-15T00:01:06.1578127Z Running test batch 'tests to run' cost 9505.74 seconds 2025-08-15T00:01:07.1004081Z 2025-08-15T00:01:07.1004797Z real 158m31.842s 2025-08-15T00:01:07.1005469Z user 418m56.329s 2025-08-15T00:01:07.1005881Z sys 497m15.460s 2025-08-15T00:01:07.1007308Z + assert_git_not_dirty 2025-08-15T00:01:07.1007832Z + [[ linux-jammy-rocm-py3.10 != *rocm* ]] 2025-08-15T00:01:07.1008465Z + [[ linux-jammy-rocm-py3.10 == *cuda* ]] 2025-08-15T00:01:07.1009038Z + [[ linux-jammy-rocm-py3.10 == *rocm* ]] 2025-08-15T00:01:07.1009561Z + [[ 2 == 1 ]] 2025-08-15T00:01:07.1009940Z + [[ 2 == 1 ]] 2025-08-15T00:01:07.1010325Z + sccache_epilogue 2025-08-15T00:01:07.1010814Z + echo '::group::Sccache Compilation Log' 2025-08-15T00:01:07.1011941Z ##[group]Sccache Compilation Log 2025-08-15T00:01:07.1012593Z + echo '=================== sccache compilation log ===================' 2025-08-15T00:01:07.1013339Z =================== sccache compilation log =================== 2025-08-15T00:01:07.1014457Z + python /var/lib/jenkins/pytorch/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-08-15T00:01:07.1290674Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-08-15T00:01:07.1292219Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-08-15T00:01:07.1293219Z + sccache --show-stats 2025-08-15T00:01:07.1352863Z Compile requests 596 2025-08-15T00:01:07.1353674Z Compile requests executed 18 2025-08-15T00:01:07.1354254Z Cache hits 0 2025-08-15T00:01:07.1354798Z Cache misses 18 2025-08-15T00:01:07.1355347Z Cache misses (C/C++) 18 2025-08-15T00:01:07.1355899Z Cache hits rate 0.00 % 2025-08-15T00:01:07.1356463Z Cache hits rate (C/C++) 0.00 % 2025-08-15T00:01:07.1356992Z Cache timeouts 0 2025-08-15T00:01:07.1357506Z Cache read errors 0 2025-08-15T00:01:07.1358570Z Forced recaches 0 2025-08-15T00:01:07.1359158Z Cache write errors 0 2025-08-15T00:01:07.1359685Z Cache errors 0 2025-08-15T00:01:07.1360362Z Compilations 18 2025-08-15T00:01:07.1360911Z Compilation failures 0 2025-08-15T00:01:07.1361446Z Non-cacheable compilations 0 2025-08-15T00:01:07.1361982Z Non-cacheable calls 16 2025-08-15T00:01:07.1362571Z Non-compilation calls 562 2025-08-15T00:01:07.1363204Z Unsupported compiler calls 0 2025-08-15T00:01:07.1363835Z Average cache write 0.000 s 2025-08-15T00:01:07.1364415Z Average compiler 1.507 s 2025-08-15T00:01:07.1364962Z Average cache read hit 0.000 s 2025-08-15T00:01:07.1365515Z Failed distributed compilations 0 2025-08-15T00:01:07.1365895Z 2025-08-15T00:01:07.1366074Z Non-cacheable reasons: 2025-08-15T00:01:07.1366550Z unknown source language 9 2025-08-15T00:01:07.1367093Z -E 7 2025-08-15T00:01:07.1367457Z 2025-08-15T00:01:07.1367845Z Cache location Local disk: "/var/lib/jenkins/.cache/sccache" 2025-08-15T00:01:07.1368795Z Use direct/preprocessor mode? yes 2025-08-15T00:01:07.1369486Z Version (client) 0.10.0 2025-08-15T00:01:07.1370122Z Cache size 1020 KiB 2025-08-15T00:01:07.1370696Z Max cache size 10 GiB 2025-08-15T00:01:07.1371245Z + sccache --stop-server 2025-08-15T00:01:07.1425146Z Stopping sccache server... 2025-08-15T00:01:07.1430663Z Compile requests 596 2025-08-15T00:01:07.1431386Z Compile requests executed 18 2025-08-15T00:01:07.1431964Z Cache hits 0 2025-08-15T00:01:07.1432509Z Cache misses 18 2025-08-15T00:01:07.1433036Z Cache misses (C/C++) 18 2025-08-15T00:01:07.1433573Z Cache hits rate 0.00 % 2025-08-15T00:01:07.1459809Z Cache hits rate (C/C++) 0.00 % 2025-08-15T00:01:07.1460563Z Cache timeouts 0 2025-08-15T00:01:07.1461162Z Cache read errors 0 2025-08-15T00:01:07.1461708Z Forced recaches 0 2025-08-15T00:01:07.1462726Z Cache write errors 0 2025-08-15T00:01:07.1463247Z Cache errors 0 2025-08-15T00:01:07.1463761Z Compilations 18 2025-08-15T00:01:07.1464295Z Compilation failures 0 2025-08-15T00:01:07.1464845Z Non-cacheable compilations 0 2025-08-15T00:01:07.1465381Z Non-cacheable calls 16 2025-08-15T00:01:07.1465916Z Non-compilation calls 562 2025-08-15T00:01:07.1466461Z Unsupported compiler calls 0 2025-08-15T00:01:07.1467014Z Average cache write 0.000 s 2025-08-15T00:01:07.1467594Z Average compiler 1.507 s 2025-08-15T00:01:07.1468150Z Average cache read hit 0.000 s 2025-08-15T00:01:07.1468730Z Failed distributed compilations 0 2025-08-15T00:01:07.1469128Z 2025-08-15T00:01:07.1469349Z Non-cacheable reasons: 2025-08-15T00:01:07.1469837Z unknown source language 9 2025-08-15T00:01:07.1470405Z -E 7 2025-08-15T00:01:07.1470760Z 2025-08-15T00:01:07.1471133Z Cache location Local disk: "/var/lib/jenkins/.cache/sccache" 2025-08-15T00:01:07.1471913Z Use direct/preprocessor mode? yes 2025-08-15T00:01:07.1472467Z Version (client) 0.10.0 2025-08-15T00:01:07.1473029Z Cache size 1020 KiB 2025-08-15T00:01:07.1473587Z Max cache size 10 GiB 2025-08-15T00:01:07.1474153Z + echo ::endgroup:: 2025-08-15T00:01:07.1474929Z ##[endgroup] 2025-08-15T00:01:07.1619333Z ##[group]Run # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2025-08-15T00:01:07.1621007Z # copy test results back to the mounted workspace, needed sudo, resulting permissions were correct 2025-08-15T00:01:07.1623027Z docker exec -t "aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8" sh -c "cd ../pytorch && sudo cp -R test/test-reports ../workspace/test" 2025-08-15T00:01:07.1678239Z shell: /usr/bin/bash -e {0} 2025-08-15T00:01:07.1678772Z env: 2025-08-15T00:01:07.1679198Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:07.1680044Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:07.1681464Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:07.1682597Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:07.1684565Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:07.1686296Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:07.1686841Z AWS_REGION: us-east-1 2025-08-15T00:01:07.1687439Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:07.1688167Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:07.1700511Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:07.1701368Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:07.1702287Z ##[endgroup] 2025-08-15T00:01:07.3881215Z ##[group]Run cat test/**/*_toprint.log || true 2025-08-15T00:01:07.3882090Z cat test/**/*_toprint.log || true 2025-08-15T00:01:07.3936646Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:07.3937431Z env: 2025-08-15T00:01:07.3937910Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:07.3938837Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:07.3940094Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:07.3941157Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:07.3942945Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:07.3944999Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:07.3945515Z AWS_REGION: us-east-1 2025-08-15T00:01:07.3946151Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:07.3946838Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:07.3957712Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:07.3958522Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:07.3959396Z ##[endgroup] 2025-08-15T00:01:07.4265148Z cat: 'test/**/*_toprint.log': No such file or directory 2025-08-15T00:01:07.4502523Z Prepare all required actions 2025-08-15T00:01:07.4503518Z Getting action download info 2025-08-15T00:01:07.7675777Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-15T00:01:09.0417994Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-08-15T00:01:10.8772457Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-08-15T00:01:10.8772767Z with: 2025-08-15T00:01:10.8772967Z use-gha: true 2025-08-15T00:01:10.8773276Z file-suffix: test-distributed-2-3-linux.rocm.gpu.4_48128182520 2025-08-15T00:01:10.8773640Z s3-bucket: gha-artifacts 2025-08-15T00:01:10.8773862Z env: 2025-08-15T00:01:10.8774041Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:10.8774405Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:10.8774965Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:10.8775521Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:10.8776394Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:10.8777170Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:10.8777415Z AWS_REGION: us-east-1 2025-08-15T00:01:10.8777686Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:10.8778020Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:10.8783441Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:10.8783901Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:10.8784400Z ##[endgroup] 2025-08-15T00:01:10.8847829Z ##[group]Run actions/upload-artifact@v4 2025-08-15T00:01:10.8848131Z with: 2025-08-15T00:01:10.8848516Z name: test-jsons-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip 2025-08-15T00:01:10.8848982Z retention-days: 14 2025-08-15T00:01:10.8849207Z if-no-files-found: warn 2025-08-15T00:01:10.8849441Z path: test/**/*.json 2025-08-15T00:01:10.8849664Z compression-level: 6 2025-08-15T00:01:10.8849876Z overwrite: false 2025-08-15T00:01:10.8850092Z include-hidden-files: false 2025-08-15T00:01:10.8850326Z env: 2025-08-15T00:01:10.8850509Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:10.8850882Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:10.8851443Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:10.8851972Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:10.8852842Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:10.8853631Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:10.8853879Z AWS_REGION: us-east-1 2025-08-15T00:01:10.8854153Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:10.8854485Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:10.8859802Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:10.8860192Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:10.8860613Z ##[endgroup] 2025-08-15T00:01:11.7878746Z With the provided path, there will be 6 files uploaded 2025-08-15T00:01:11.7884819Z Artifact name is valid! 2025-08-15T00:01:11.7885891Z Root directory input is valid! 2025-08-15T00:01:12.0703955Z Beginning upload of artifact content to blob storage 2025-08-15T00:01:12.4745954Z Uploaded bytes 43828 2025-08-15T00:01:12.5639337Z Finished uploading artifact content to blob storage! 2025-08-15T00:01:12.5644444Z SHA256 digest of uploaded artifact zip is 63f8dcabc1f07f25c44ccc1f8629b58966bca10b6414ef01b0172013eca5de1b 2025-08-15T00:01:12.5646477Z Finalizing artifact upload 2025-08-15T00:01:12.6962462Z Artifact test-jsons-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip.zip successfully finalized. Artifact ID 3770296459 2025-08-15T00:01:12.6964934Z Artifact test-jsons-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip has been successfully uploaded! Final size is 43828 bytes. Artifact ID is 3770296459 2025-08-15T00:01:12.6974116Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770296459 2025-08-15T00:01:12.7335252Z ##[group]Run actions/upload-artifact@v4 2025-08-15T00:01:12.7335558Z with: 2025-08-15T00:01:12.7335991Z name: test-reports-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip 2025-08-15T00:01:12.7336482Z retention-days: 14 2025-08-15T00:01:12.7336725Z if-no-files-found: ignore 2025-08-15T00:01:12.7336995Z path: test/**/*.xml test/**/*.csv 2025-08-15T00:01:12.7337272Z compression-level: 6 2025-08-15T00:01:12.7337501Z overwrite: false 2025-08-15T00:01:12.7337733Z include-hidden-files: false 2025-08-15T00:01:12.7337990Z env: 2025-08-15T00:01:12.7338211Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:12.7338613Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:12.7339200Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:12.7339764Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:12.7341459Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:12.7343309Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:12.7343817Z AWS_REGION: us-east-1 2025-08-15T00:01:12.7344391Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:12.7345077Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:12.7355584Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:12.7356387Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:12.7357256Z ##[endgroup] 2025-08-15T00:01:13.6756554Z With the provided path, there will be 664 files uploaded 2025-08-15T00:01:13.6761468Z Artifact name is valid! 2025-08-15T00:01:13.6762265Z Root directory input is valid! 2025-08-15T00:01:13.9209714Z Beginning upload of artifact content to blob storage 2025-08-15T00:01:14.8979636Z Uploaded bytes 527252 2025-08-15T00:01:14.9759530Z Finished uploading artifact content to blob storage! 2025-08-15T00:01:14.9765428Z SHA256 digest of uploaded artifact zip is 8e406af6fc49c60a7a25ccc52685272f009113834b7447a1b6bef63230205472 2025-08-15T00:01:14.9767848Z Finalizing artifact upload 2025-08-15T00:01:15.1029413Z Artifact test-reports-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip.zip successfully finalized. Artifact ID 3770296635 2025-08-15T00:01:15.1031940Z Artifact test-reports-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip has been successfully uploaded! Final size is 527252 bytes. Artifact ID is 3770296635 2025-08-15T00:01:15.1043160Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770296635 2025-08-15T00:01:15.1502919Z ##[group]Run actions/upload-artifact@v4 2025-08-15T00:01:15.1503543Z with: 2025-08-15T00:01:15.1504313Z name: logs-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip 2025-08-15T00:01:15.1505229Z retention-days: 14 2025-08-15T00:01:15.1505738Z if-no-files-found: ignore 2025-08-15T00:01:15.1506293Z path: usage_log.txt test/**/*.log 2025-08-15T00:01:15.1506876Z compression-level: 6 2025-08-15T00:01:15.1507920Z overwrite: false 2025-08-15T00:01:15.1508407Z include-hidden-files: false 2025-08-15T00:01:15.1508923Z env: 2025-08-15T00:01:15.1509330Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:15.1510160Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:15.1511360Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:15.1512901Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:15.1514840Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:15.1516515Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:15.1517066Z AWS_REGION: us-east-1 2025-08-15T00:01:15.1517668Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:15.1518411Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:15.1529656Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:15.1530521Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:15.1531415Z ##[endgroup] 2025-08-15T00:01:16.0684308Z Multiple search paths detected. Calculating the least common ancestor of all paths 2025-08-15T00:01:16.0686675Z The least common ancestor is /var/home/pytorchci/actions-runner/_work/pytorch/pytorch. This will be the root directory of the artifact 2025-08-15T00:01:16.0688152Z With the provided path, there will be 88 files uploaded 2025-08-15T00:01:16.0691732Z Artifact name is valid! 2025-08-15T00:01:16.0693175Z Root directory input is valid! 2025-08-15T00:01:16.3022211Z Beginning upload of artifact content to blob storage 2025-08-15T00:01:17.0154447Z Uploaded bytes 329416 2025-08-15T00:01:17.0902335Z Finished uploading artifact content to blob storage! 2025-08-15T00:01:17.0907841Z SHA256 digest of uploaded artifact zip is 533423ca6e738449b594e0b5b5ef5a87df4f642bb705713d384a1b89db897bfc 2025-08-15T00:01:17.0910666Z Finalizing artifact upload 2025-08-15T00:01:17.2067714Z Artifact logs-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip.zip successfully finalized. Artifact ID 3770296780 2025-08-15T00:01:17.2070095Z Artifact logs-runattempt1-test-distributed-2-3-linux.rocm.gpu.4_48128182520.zip has been successfully uploaded! Final size is 329416 bytes. Artifact ID is 3770296780 2025-08-15T00:01:17.2081644Z Artifact download URL: https://github.com/pytorch/pytorch/actions/runs/16976370303/artifacts/3770296780 2025-08-15T00:01:17.2461447Z ##[group]Run # shellcheck disable=SC2156 2025-08-15T00:01:17.2462363Z # shellcheck disable=SC2156 2025-08-15T00:01:17.2463584Z find . -iname "core.[1-9]*" -exec docker exec "${CONTAINER_NAME}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-08-15T00:01:17.2510089Z shell: /usr/bin/bash -e {0} 2025-08-15T00:01:17.2510633Z env: 2025-08-15T00:01:17.2511077Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:17.2511906Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:17.2513154Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:17.2514280Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:17.2516141Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:17.2517864Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:17.2518423Z AWS_REGION: us-east-1 2025-08-15T00:01:17.2519032Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:17.2519770Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:17.2531663Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:17.2532682Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:17.2533773Z ##[endgroup] 2025-08-15T00:01:17.6545570Z ##[group]Run aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722 2025-08-15T00:01:17.6547132Z with: 2025-08-15T00:01:17.6547956Z role-to-assume: arn:aws:iam::308535385114:role/gha_workflow_upload-benchmark-results 2025-08-15T00:01:17.6548933Z role-duration-seconds: 18000 2025-08-15T00:01:17.6549531Z aws-region: us-east-1 2025-08-15T00:01:17.6550087Z audience: sts.amazonaws.com 2025-08-15T00:01:17.6550636Z env: 2025-08-15T00:01:17.6551078Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:17.6551945Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:17.6553204Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:17.6554374Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:17.6556393Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:17.6558131Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:17.6558720Z AWS_REGION: us-east-1 2025-08-15T00:01:17.6559366Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:17.6560138Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:17.6573045Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:17.6573961Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:17.6574940Z ##[endgroup] 2025-08-15T00:01:18.0613662Z Assuming role with OIDC 2025-08-15T00:01:18.3594253Z Authenticated as assumedRoleId AROAUPVRELQNA5GQHA6IA:GitHubActions 2025-08-15T00:01:18.4637536Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-08-15T00:01:18.4638533Z with: 2025-08-15T00:01:18.4639094Z benchmark-results-dir: test/test-reports 2025-08-15T00:01:18.4639758Z dry-run: false 2025-08-15T00:01:18.4640659Z schema-version: v3 2025-08-15T00:01:18.4641596Z github-token: *** 2025-08-15T00:01:18.4642117Z env: 2025-08-15T00:01:18.4642594Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:18.4643506Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:18.4644768Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:18.4645939Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:18.4647738Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:18.4649296Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:18.4649831Z AWS_REGION: us-east-1 2025-08-15T00:01:18.4650374Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:18.4651048Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:18.4660657Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:18.4661437Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:18.4662285Z ##[endgroup] 2025-08-15T00:01:18.4695314Z ##[group]Run set -eux 2025-08-15T00:01:18.4695847Z set -eux 2025-08-15T00:01:18.4696508Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-15T00:01:18.4697256Z  2025-08-15T00:01:18.4697660Z DEVICE_NAME="" 2025-08-15T00:01:18.4698143Z DEVICE_TYPE="" 2025-08-15T00:01:18.4698586Z  2025-08-15T00:01:18.4699027Z if command -v nvidia-smi; then 2025-08-15T00:01:18.4699826Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-08-15T00:01:18.4700811Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-08-15T00:01:18.4701720Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-08-15T00:01:18.4702450Z  python3 -mpip install torch==2.7.1 2025-08-15T00:01:18.4703042Z elif command -v rocminfo; then 2025-08-15T00:01:18.4703796Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-08-15T00:01:18.4705237Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-08-15T00:01:18.4706180Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-08-15T00:01:18.4707014Z  # GPU device name coming from rocminfo instead 2025-08-15T00:01:18.4707640Z  DEVICE_NAME=rocm 2025-08-15T00:01:18.4708469Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-08-15T00:01:18.4709321Z fi 2025-08-15T00:01:18.4709701Z  2025-08-15T00:01:18.4710185Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-08-15T00:01:18.4710893Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-08-15T00:01:18.4758808Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:18.4759467Z env: 2025-08-15T00:01:18.4759875Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:18.4760920Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:18.4762060Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:18.4763082Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:18.4764759Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:18.4766251Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:18.4767179Z AWS_REGION: us-east-1 2025-08-15T00:01:18.4767786Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:18.4768459Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:18.4778071Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:18.4778859Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:18.4779684Z ##[endgroup] 2025-08-15T00:01:18.4869852Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-15T00:01:18.7736271Z Defaulting to user installation because normal site-packages is not writeable 2025-08-15T00:01:18.9434784Z Requirement already satisfied: boto3==1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (1.35.33) 2025-08-15T00:01:18.9438447Z Requirement already satisfied: psutil==7.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (7.0.0) 2025-08-15T00:01:18.9444440Z Requirement already satisfied: pynvml==12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (12.0.0) 2025-08-15T00:01:18.9478093Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3/dist-packages (from boto3==1.35.33) (0.10.0) 2025-08-15T00:01:18.9484103Z Requirement already satisfied: botocore<1.36.0,>=1.35.33 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (1.35.99) 2025-08-15T00:01:18.9487878Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from boto3==1.35.33) (0.10.4) 2025-08-15T00:01:18.9641060Z Requirement already satisfied: nvidia-ml-py<13.0.0a0,>=12.0.0 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from pynvml==12.0.0) (12.575.51) 2025-08-15T00:01:18.9690208Z Requirement already satisfied: urllib3!=2.2.0,<3,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.26.5) 2025-08-15T00:01:18.9694688Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /var/home/pytorchci/.local/lib/python3.10/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.9.0.post0) 2025-08-15T00:01:18.9737686Z Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.16.0) 2025-08-15T00:01:19.2551149Z + DEVICE_NAME= 2025-08-15T00:01:19.2551706Z + DEVICE_TYPE= 2025-08-15T00:01:19.2552272Z + command -v nvidia-smi 2025-08-15T00:01:19.2552830Z + command -v rocminfo 2025-08-15T00:01:19.2553339Z + DEVICE_NAME=rocm 2025-08-15T00:01:19.2553873Z /opt/rocm/bin/rocminfo 2025-08-15T00:01:19.2568336Z ++ rocminfo 2025-08-15T00:01:19.2572347Z ++ grep 'Marketing Name' 2025-08-15T00:01:19.2575626Z ++ tail -n1 2025-08-15T00:01:19.2580965Z ++ awk -F: '{print $2}' 2025-08-15T00:01:19.2585049Z ++ xargs 2025-08-15T00:01:19.4155862Z + DEVICE_TYPE='AMD Instinct MI250X/MI250' 2025-08-15T00:01:19.4156612Z + echo DEVICE_NAME=rocm 2025-08-15T00:01:19.4161105Z + echo 'DEVICE_TYPE=AMD Instinct MI250X/MI250' 2025-08-15T00:01:19.4211936Z ##[group]Run set -eux 2025-08-15T00:01:19.4212548Z set -eux 2025-08-15T00:01:19.4213094Z  2025-08-15T00:01:19.4213637Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-08-15T00:01:19.4214391Z  echo "Missing github-token input" 2025-08-15T00:01:19.4215076Z  exit 1 2025-08-15T00:01:19.4215590Z fi 2025-08-15T00:01:19.4271039Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:19.4271862Z env: 2025-08-15T00:01:19.4272356Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:19.4273296Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:19.4274616Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:19.4275833Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:19.4277845Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:19.4280101Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:19.4281163Z AWS_REGION: us-east-1 2025-08-15T00:01:19.4281851Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:19.4282698Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:19.4295069Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:19.4295945Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:19.4296888Z DEVICE_NAME: rocm 2025-08-15T00:01:19.4297452Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:19.4298376Z GITHUB_TOKEN: *** 2025-08-15T00:01:19.4298943Z ##[endgroup] 2025-08-15T00:01:19.4392170Z + [[ -z *** ]] 2025-08-15T00:01:19.4482113Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-08-15T00:01:19.4483141Z with: 2025-08-15T00:01:19.4483970Z github-token: *** 2025-08-15T00:01:19.4484469Z env: 2025-08-15T00:01:19.4484938Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:19.4485799Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:19.4487086Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:19.4488216Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:19.4490145Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:19.4491867Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:19.4492477Z AWS_REGION: us-east-1 2025-08-15T00:01:19.4493086Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:19.4493847Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:19.4505062Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:19.4505949Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:19.4506888Z DEVICE_NAME: rocm 2025-08-15T00:01:19.4507423Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:19.4508036Z ##[endgroup] 2025-08-15T00:01:19.4539757Z ##[group]Run set -eux 2025-08-15T00:01:19.4540324Z set -eux 2025-08-15T00:01:19.4540774Z  2025-08-15T00:01:19.4541857Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-15T00:01:19.4594413Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:19.4595184Z env: 2025-08-15T00:01:19.4595644Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:19.4596495Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:19.4598239Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:19.4599441Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:19.4601746Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:19.4603813Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:19.4604433Z AWS_REGION: us-east-1 2025-08-15T00:01:19.4605052Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:19.4605830Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:19.4615882Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:19.4616801Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:19.4617636Z DEVICE_NAME: rocm 2025-08-15T00:01:19.4618107Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:19.4618981Z GITHUB_TOKEN: *** 2025-08-15T00:01:19.4619414Z ##[endgroup] 2025-08-15T00:01:19.4704158Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16976370303 gpu9498 2025-08-15T00:01:21.1326761Z setting job-id=48128182520 2025-08-15T00:01:21.1328031Z setting job-name=linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-15T00:01:21.1572895Z ##[group]Run set -eux 2025-08-15T00:01:21.1573495Z set -eux 2025-08-15T00:01:21.1573969Z  2025-08-15T00:01:21.1574737Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-08-15T00:01:21.1575743Z  --schema-version "${SCHEMA_VERSION}" \ 2025-08-15T00:01:21.1576438Z  --repo "${REPO}" \ 2025-08-15T00:01:21.1577048Z  --head-branch "${HEAD_BRANCH}" \ 2025-08-15T00:01:21.1577709Z  --head-sha "${HEAD_SHA}" \ 2025-08-15T00:01:21.1578402Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-08-15T00:01:21.1579094Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-08-15T00:01:21.1579742Z  --job-id "${JOB_ID}" \ 2025-08-15T00:01:21.1580338Z  --job-name "${JOB_NAME}" 2025-08-15T00:01:21.1631373Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:21.1632162Z env: 2025-08-15T00:01:21.1632628Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:21.1633504Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:21.1634743Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:21.1635919Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:21.1637824Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:21.1639538Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:21.1640129Z AWS_REGION: us-east-1 2025-08-15T00:01:21.1640879Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:21.1641627Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:21.1652416Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:21.1653294Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:21.1654223Z DEVICE_NAME: rocm 2025-08-15T00:01:21.1654755Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:21.1655386Z SCHEMA_VERSION: v3 2025-08-15T00:01:21.1655901Z REPO: pytorch/pytorch 2025-08-15T00:01:21.1656448Z HEAD_BRANCH: refs/heads/main 2025-08-15T00:01:21.1657082Z HEAD_SHA: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-15T00:01:21.1657769Z WORKFLOW_RUN_ID: 16976370303 2025-08-15T00:01:21.1658300Z RUN_ATTEMPT: 1 2025-08-15T00:01:21.1658776Z JOB_ID: 48128182520 2025-08-15T00:01:21.1659771Z JOB_NAME: linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed) 2025-08-15T00:01:21.1661307Z ##[endgroup] 2025-08-15T00:01:21.1756171Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 1fc683cf17c8c673044538d10266c00f92987be2 --workflow-id 16976370303 --run-attempt 1 --job-id 48128182520 --job-name 'linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)' 2025-08-15T00:01:21.2389108Z ##[group]Run set -eux 2025-08-15T00:01:21.2389648Z set -eux 2025-08-15T00:01:21.2390110Z  2025-08-15T00:01:21.2390875Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-08-15T00:01:21.2438022Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:21.2438794Z env: 2025-08-15T00:01:21.2439243Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:21.2440137Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:21.2441739Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:21.2442895Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:21.2445152Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:21.2446848Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:21.2447432Z AWS_REGION: us-east-1 2025-08-15T00:01:21.2448056Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:21.2448838Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:21.2460383Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:21.2461418Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:21.2462535Z DEVICE_NAME: rocm 2025-08-15T00:01:21.2463186Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:21.2463889Z ##[endgroup] 2025-08-15T00:01:21.2549142Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-08-15T00:01:22.0231405Z /var/home/pytorchci/.local/lib/python3.10/site-packages/torch/_subclasses/functional_tensor.py:276: UserWarning: Failed to initialize NumPy: No module named 'numpy' (Triggered internally at /pytorch/torch/csrc/utils/tensor_numpy.cpp:81.) 2025-08-15T00:01:22.0233760Z cpu = _conversion_method_template(device=torch.device("cpu")) 2025-08-15T00:01:22.6378935Z ##[group]Run set -eux 2025-08-15T00:01:22.6379528Z set -eux 2025-08-15T00:01:22.6380011Z  2025-08-15T00:01:22.6380547Z # TODO (huydhn): Implement this part 2025-08-15T00:01:22.6381359Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-08-15T00:01:22.6434404Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:22.6435213Z env: 2025-08-15T00:01:22.6435705Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:22.6436566Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:22.6437809Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:22.6438983Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:22.6441235Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:22.6442945Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:22.6443556Z AWS_REGION: us-east-1 2025-08-15T00:01:22.6444224Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:22.6444965Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:22.6456874Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:22.6457774Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:22.6459253Z DEVICE_NAME: rocm 2025-08-15T00:01:22.6459794Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:22.6460424Z ##[endgroup] 2025-08-15T00:01:22.6554729Z + echo 'dependencies={}' 2025-08-15T00:01:22.6604045Z ##[group]Run set -eux 2025-08-15T00:01:22.6604650Z set -eux 2025-08-15T00:01:22.6605111Z  2025-08-15T00:01:22.6605671Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-08-15T00:01:22.6606617Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-08-15T00:01:22.6607607Z  # We don't want the job to fail if the directory doesn't exist 2025-08-15T00:01:22.6608389Z  exit 0 2025-08-15T00:01:22.6608853Z fi 2025-08-15T00:01:22.6609307Z  2025-08-15T00:01:22.6609793Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-08-15T00:01:22.6610761Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-15T00:01:22.6611879Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-15T00:01:22.6612710Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-15T00:01:22.6613421Z  --runners "${RUNNER_INFO}" \ 2025-08-15T00:01:22.6614135Z  --dependencies "${DEPENDENCIES}" \ 2025-08-15T00:01:22.6614790Z  --dry-run 2025-08-15T00:01:22.6615316Z else 2025-08-15T00:01:22.6616058Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-15T00:01:22.6617556Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-15T00:01:22.6618423Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-15T00:01:22.6619127Z  --runners "${RUNNER_INFO}" \ 2025-08-15T00:01:22.6619828Z  --dependencies "${DEPENDENCIES}" 2025-08-15T00:01:22.6620475Z fi 2025-08-15T00:01:22.6671348Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:22.6672161Z env: 2025-08-15T00:01:22.6672647Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:22.6673516Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:22.6674776Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:22.6675948Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:22.6677858Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:22.6679585Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:22.6680346Z AWS_REGION: us-east-1 2025-08-15T00:01:22.6681009Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:22.6681785Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:22.6693190Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:22.6694289Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:22.6695432Z DEVICE_NAME: rocm 2025-08-15T00:01:22.6696071Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:22.6696869Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-08-15T00:01:22.6697504Z DRY_RUN: false 2025-08-15T00:01:22.6700185Z BENCHMARK_METADATA: {"timestamp": 1755216081, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182520} 2025-08-15T00:01:22.6704101Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpu9498.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}] 2025-08-15T00:01:22.6705725Z DEPENDENCIES: {} 2025-08-15T00:01:22.6706212Z ##[endgroup] 2025-08-15T00:01:22.6801512Z + [[ ! -d test/test-reports ]] 2025-08-15T00:01:22.6802659Z + [[ false == \t\r\u\e ]] 2025-08-15T00:01:22.6807667Z + python3 /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1755216081, "schema_version": "v3", "name": "linux-jammy-rocm-py3.10 / test (distributed, 2, 3, linux.rocm.gpu.4, module:rocm, oncall:distributed)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976370303, "run_attempt": 1, "job_id": 48128182520}' --runners '[{"cpu_info": "x86_64", "cpu_count": 128, "avail_mem_in_gb": 1007, "extra_info": {"hostname": "gpu9498.jax.cs.cpe.ice.amd.com"}, "name": "rocm", "type": "AMD Instinct MI250X/MI250"}]' --dependencies '{}' 2025-08-15T00:01:22.8492932Z /var/home/pytorchci/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py:236: UserWarning: {'included': [{'test_file': 'distributed/fsdp/test_fsdp_apply'}, {'test_file': 'distributed/fsdp/test_distributed_checkpoint'}, {'test_file': 'distributed/test_store'}, {'test_file': 'distributed/test_dynamo_distributed'}, {'test_file': 'distributed/test_inductor_collectives'}, {'test_file': 'distributed/tensor/test_dtensor_compile'}, {'test_file': 'distributed/test_c10d_functional_native'}, {'test_file': 'distributed/tensor/parallel/test_micro_pipeline_tp'}, {'test_file': 'distributed/tensor/debug/test_op_coverage'}, {'test_file': 'distributed/_tools/test_mod_tracker'}, {'test_file': 'distributed/checkpoint/test_nested_dict'}, {'test_file': 'distributed/flight_recorder/test_fr_analysis'}, {'test_file': 'distributed/test_compute_comm_reordering'}, {'test_file': 'distributed/checkpoint/test_consolidate_hf_safetensors'}, {'test_file': 'distributed/_tools/test_runtime_estimator'}, {'test_file': 'distributed/_tools/test_sac_ilp'}, {'test_file': 'distributed/pipelining/test_schedule_multiproc'}, {'test_file': 'distributed/_composable/test_contract'}, {'test_file': 'distributed/_shard/sharded_tensor/test_logger'}, {'test_file': 'distributed/pipelining/test_transformer'}, {'test_file': 'distributed/tensor/test_dtensor'}, {'test_file': 'distributed/tensor/test_attention'}, {'test_file': 'distributed/pipelining/test_pipe'}, {'test_file': 'distributed/_tools/test_fake_collectives'}, {'test_file': 'distributed/pipelining/test_backward'}, {'test_file': 'distributed/checkpoint/_experimental/test_barriers'}, {'test_file': 'distributed/checkpoint/_experimental/test_types'}, {'test_file': 'distributed/_tools/test_sac_estimator'}, {'test_file': 'distributed/tensor/test_xla_integration'}, {'test_file': 'distributed/checkpoint/test_dedup_tensors'}, {'test_file': 'distributed/pipelining/test_unflatten'}, {'test_file': 'distributed/checkpoint/test_format_utils'}, {'test_file': 'distributed/tensor/test_dtensor_ops'}, {'test_file': 'distributed/pipelining/test_microbatch'}, {'test_file': 'distributed/tensor/test_api'}, {'test_file': 'distributed/fsdp/test_fsdp_fx'}, {'test_file': 'distributed/test_nvshmem_triton'}, {'test_file': 'distributed/tensor/test_experimental_ops'}, {'test_file': 'distributed/checkpoint/test_hf_storage'}, {'test_file': 'distributed/tensor/test_common_rules'}, {'test_file': 'distributed/tensor/test_convolution_ops'}, {'test_file': 'distributed/tensor/test_matrix_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_reader'}, {'test_file': 'distributed/_composable/test_composability/test_pp_composability'}, {'test_file': 'distributed/tensor/test_tensor_ops'}, {'test_file': 'distributed/tensor/debug/test_comm_mode_features'}, {'test_file': 'distributed/tensor/experimental/test_register_sharding'}, {'test_file': 'distributed/checkpoint/_experimental/test_builder'}, {'test_file': 'distributed/tensor/test_init'}, {'test_file': 'distributed/optim/test_named_optimizer'}, {'test_file': 'distributed/_composable/test_replicate_with_fsdp'}, {'test_file': 'distributed/tensor/parallel/test_tp_style'}, {'test_file': 'distributed/_tools/test_memory_tracker'}, {'test_file': 'distributed/_composable/test_checkpoint'}, {'test_file': 'distributed/checkpoint/_experimental/test_staging'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_writer'}, {'test_file': 'distributed/tensor/test_pointwise_ops'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpointer'}, {'test_file': 'distributed/checkpoint/fsdp/test_fsdp_dsd'}, {'test_file': 'distributed/tensor/experimental/test_local_map'}, {'test_file': 'distributed/checkpoint/e2e/test_fsdp_ep'}, {'test_file': 'distributed/test_functional_api'}, {'test_file': 'distributed/tensor/test_random_ops'}, {'test_file': 'distributed/_tools/test_fsdp2_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_traverse'}, {'test_file': 'distributed/checkpoint/test_fsspec'}, {'test_file': 'distributed/tensor/test_redistribute'}, {'test_file': 'distributed/checkpoint/test_async_process_executor'}, {'test_file': 'distributed/tensor/test_math_ops'}, {'test_file': 'distributed/test_backends'}, {'test_file': 'distributed/tensor/experimental/test_tp_transform'}, {'test_file': 'distributed/test_device_mesh'}, {'test_file': 'distributed/checkpoint/test_fsdp_optim_state'}, {'test_file': 'distributed/checkpoint/test_dtensor_resharding'}, {'test_file': 'distributed/test_symmetric_memory'}, {'test_file': 'distributed/checkpoint/e2e/test_fine_tuning'}, {'test_file': 'distributed/pipelining/test_stage'}, {'test_file': 'distributed/_tools/test_mem_tracker'}, {'test_file': 'distributed/checkpoint/test_tp_checkpoint'}, {'test_file': 'distributed/pipelining/test_schedule'}, {'test_file': 'distributed/test_nvshmem'}, {'test_file': 'distributed/tensor/test_embedding_ops'}, {'test_file': 'distributed/checkpoint/test_save_load_api'}, {'test_file': 'distributed/test_control_collectives'}, {'test_file': 'distributed/checkpoint/test_hf_safetensor_e2e'}, {'test_file': 'distributed/checkpoint/e2e/test_e2e_save_and_load'}, {'test_file': 'distributed/checkpoint/test_planner'}, {'test_file': 'distributed/launcher/test_run'}, {'test_file': 'distributed/checkpoint/test_hsdp_checkpoint'}, {'test_file': 'distributed/checkpoint/test_dtensor_checkpoint'}, {'test_file': 'distributed/tensor/test_view_ops'}, {'test_file': 'distributed/checkpoint/test_fsdp_tp_checkpoint_conversion'}, {'test_file': 'distributed/tensor/test_optimizers'}, {'test_file': 'distributed/test_p2p_ipc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_ignore_params'}, {'test_file': 'distributed/checkpoint/test_compatibility'}, {'test_file': 'distributed/tensor/parallel/test_tp_examples'}, {'test_file': 'distributed/tensor/test_utils'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_overlap'}, {'test_file': 'distributed/tensor/parallel/test_tp_random_state'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_logging'}, {'test_file': 'distributed/optim/test_apply_optimizer_in_backward'}, {'test_file': 'distributed/_composable/test_composability/test_2d_composability'}, {'test_file': 'distributed/_composable/test_replicate_with_compiler'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_compile'}, {'test_file': 'distributed/checkpoint/test_fsdp_model_state'}, {'test_file': 'distributed/fsdp/test_shard_utils'}, {'test_file': 'distributed/elastic/test_control_plane'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_grad_scaler'}, {'test_file': 'distributed/checkpoint/_experimental/test_checkpoint_process'}, {'test_file': 'distributed/tensor/parallel/test_parallelize_api'}, {'test_file': 'distributed/fsdp/test_fsdp_optim_state'}, {'test_file': 'distributed/tensor/test_op_strategy'}, {'test_file': 'distributed/checkpoint/test_checkpoint'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_clip_grad_norm_'}, {'test_file': 'distributed/checkpoint/test_pg_transport'}, {'test_file': 'distributed/test_collective_utils'}, {'test_file': 'distributed/tensor/debug/test_comm_mode'}, {'test_file': 'distributed/fsdp/test_fsdp_pure_fp16'}, {'test_file': 'distributed/checkpoint/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_state_dict'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state'}, {'test_file': 'distributed/test_dist2'}, {'test_file': 'distributed/fsdp/test_fsdp_memory'}, {'test_file': 'distributed/_composable/test_replicate'}, {'test_file': 'distributed/test_fake_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_backward_prefetch'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_autograd'}, {'test_file': 'distributed/checkpoint/test_state_dict_stager'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_frozen'}, {'test_file': 'distributed/fsdp/test_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_overlap'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_init'}, {'test_file': 'distributed/fsdp/test_checkpoint_wrapper'}, {'test_file': 'distributed/fsdp/test_fsdp_grad_acc'}, {'test_file': 'distributed/fsdp/test_fsdp_flatten_params'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_mixed_precision'}, {'test_file': 'distributed/test_c10d_logger'}, {'test_file': 'distributed/test_c10d_object_collectives'}, {'test_file': 'distributed/test_serialization'}, {'test_file': 'distributed/checkpoint/test_state_dict'}, {'test_file': 'distributed/optim/test_zero_redundancy_optimizer'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_comm'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint_cpu'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_extensions'}, {'test_file': 'distributed/fsdp/test_fsdp_ignored_modules'}, {'test_file': 'distributed/fsdp/test_fsdp_freezing_weights'}, {'test_file': 'distributed/checkpoint/test_file_system_checkpoint'}, {'test_file': 'distributed/test_nccl'}, {'test_file': 'distributed/fsdp/test_fsdp_checkpoint'}, {'test_file': 'distributed/test_distributed_spawn'}, {'test_file': 'distributed/fsdp/test_fsdp_unshard_params'}, {'test_file': 'distributed/test_multi_threaded_pg'}, {'test_file': 'distributed/fsdp/test_fsdp_tp_integration'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_training'}, {'test_file': 'distributed/fsdp/test_fsdp_dtensor_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_fine_tune'}, {'test_file': 'distributed/fsdp/test_fsdp_hybrid_shard'}, {'test_file': 'distributed/fsdp/test_fsdp_comm_hooks'}, {'test_file': 'distributed/fsdp/test_fsdp_exec_order'}, {'test_file': 'distributed/fsdp/test_fsdp_misc'}, {'test_file': 'distributed/fsdp/test_wrap'}, {'test_file': 'distributed/fsdp/test_fsdp_clip_grad_norm'}, {'test_file': 'distributed/fsdp/test_fsdp_meta'}, {'test_file': 'distributed/_shard/sharding_spec/test_sharding_spec'}, {'test_file': 'distributed/fsdp/test_hsdp_dtensor_state_dict'}, {'test_file': 'distributed/checkpoint/test_state_dict_utils'}, {'test_file': 'distributed/fsdp/test_fsdp_uneven'}, {'test_file': 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks'}, {'test_file': 'distributed/algorithms/quantization/test_quantization'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_state_dict'}, {'test_file': 'distributed/fsdp/test_fsdp_comm'}, {'test_file': 'distributed/fsdp/test_fsdp_input'}, {'test_file': 'distributed/fsdp/test_fsdp_traversal'}, {'test_file': 'distributed/fsdp/test_fsdp_sharded_grad_scaler'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_wrapping'}, {'test_file': 'distributed/_shard/test_sharder'}, {'test_file': 'distributed/algorithms/test_join'}, {'test_file': 'distributed/test_data_parallel'}, {'test_file': 'distributed/_shard/sharding_plan/test_sharding_plan'}, {'test_file': 'distributed/nn/jit/test_instantiator'}, {'test_file': 'distributed/test_pg_wrapper'}, {'test_file': 'distributed/test_c10d_pypg'}, {'test_file': 'distributed/fsdp/test_fsdp_use_orig_params'}, {'test_file': 'distributed/test_composability'}, {'test_file': 'distributed/fsdp/test_fsdp_core'}, {'test_file': 'distributed/_shard/sharded_optim/test_sharded_optim'}, {'test_file': 'distributed/fsdp/test_fsdp_multiple_forward'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_tensor_ops'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_binary_cmp'}, {'test_file': 'distributed/fsdp/test_fsdp_mixed_precision'}, {'test_file': 'distributed/test_c10d_ucc'}, {'test_file': 'distributed/_composable/fsdp/test_fully_shard_memory'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_init'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor_reshard'}, {'test_file': 'distributed/test_c10d_gloo'}, {'test_file': 'distributed/test_c10d_common'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding'}, {'test_file': 'distributed/_shard/sharded_tensor/ops/test_embedding_bag'}, {'test_file': 'distributed/_shard/sharded_tensor/test_sharded_tensor'}, {'test_file': 'distributed/test_c10d_spawn_gloo'}, {'test_file': 'distributed/test_c10d_spawn_nccl'}, {'test_file': 'distributed/test_launcher'}, {'test_file': 'distributed/test_c10d_spawn_ucc'}, {'test_file': 'distributed/test_c10d_nccl'}, {'test_file': 'distributed/test_c10d_ops_nccl'}, {'test_file': 'distributed/elastic/events/lib_test'}, {'test_file': 'distributed/elastic/metrics/api_test'}, {'test_file': 'distributed/elastic/multiprocessing/api_test'}, {'test_file': 'distributed/elastic/timer/api_test'}, {'test_file': 'distributed/elastic/timer/local_timer_example'}, {'test_file': 'distributed/elastic/timer/local_timer_test'}, {'test_file': 'distributed/elastic/utils/distributed_test'}, {'test_file': 'distributed/elastic/utils/logging_test'}, {'test_file': 'distributed/elastic/utils/util_test'}, {'test_file': 'distributed/tensor/test_dtensor_dispatch_overhead'}], 'excluded': []} from test/test-reports/td_exclusions-55b0c4c4fc868780ec0f.json is not a benchmark record, skipping 2025-08-15T00:01:22.8581236Z warn(f"{result} from {filepath} is not a benchmark record, skipping") 2025-08-15T00:01:22.8653453Z Prepare all required actions 2025-08-15T00:01:22.8654301Z Getting action download info 2025-08-15T00:01:22.8709104Z ##[group]Run ./.github/actions/teardown-rocm 2025-08-15T00:01:22.8709768Z env: 2025-08-15T00:01:22.8710262Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:22.8711130Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:22.8712371Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:22.8713507Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:22.8715411Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:22.8717095Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:22.8717659Z AWS_REGION: us-east-1 2025-08-15T00:01:22.8718294Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:22.8719094Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:22.8730180Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:22.8731062Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:22.8732023Z DEVICE_NAME: rocm 2025-08-15T00:01:22.8732537Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:22.8733160Z ##[endgroup] 2025-08-15T00:01:22.8764545Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-15T00:01:22.8765626Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-15T00:01:22.8766483Z # shellcheck disable=SC2046 2025-08-15T00:01:22.8767182Z docker stop $(docker ps -q) || true 2025-08-15T00:01:22.8767897Z # Prune all stopped containers. 2025-08-15T00:01:22.8768570Z docker container prune -f 2025-08-15T00:01:22.8816044Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:22.8816744Z env: 2025-08-15T00:01:22.8817157Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:22.8817907Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:22.8818994Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:22.8820428Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:22.8822072Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:22.8823567Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:22.8824096Z AWS_REGION: us-east-1 2025-08-15T00:01:22.8824639Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:22.8825305Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:22.8834808Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:22.8835602Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:22.8836437Z DEVICE_NAME: rocm 2025-08-15T00:01:22.8836933Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:22.8837489Z ##[endgroup] 2025-08-15T00:01:33.6737739Z aa60442279b3 2025-08-15T00:01:41.1590714Z Deleted Containers: 2025-08-15T00:01:41.1591620Z aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:41.1592476Z 2025-08-15T00:01:41.1592734Z Total reclaimed space: 10.41GB 2025-08-15T00:01:41.1708423Z Prepare all required actions 2025-08-15T00:01:41.1770708Z ##[group]Run ./.github/actions/diskspace-cleanup 2025-08-15T00:01:41.1771525Z with: 2025-08-15T00:01:41.1772076Z diskspace-cutoff: 70 2025-08-15T00:01:41.1772682Z env: 2025-08-15T00:01:41.1773199Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:41.1774063Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:41.1775301Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:41.1776464Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:41.1778912Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:41.1780824Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:41.1781508Z AWS_REGION: us-east-1 2025-08-15T00:01:41.1782276Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:41.1783076Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:41.1794015Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:41.1794909Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:41.1795889Z DEVICE_NAME: rocm 2025-08-15T00:01:41.1796442Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:41.1797065Z ##[endgroup] 2025-08-15T00:01:41.1829679Z ##[group]Run set -ex 2025-08-15T00:01:41.1830258Z set -ex 2025-08-15T00:01:41.1830768Z diskspace_cutoff=70 2025-08-15T00:01:41.1831545Z docker_root_dir=$(docker info -f '{{.DockerRootDir}}') 2025-08-15T00:01:41.1832391Z if [ ! -d "$docker_root_dir" ]; then 2025-08-15T00:01:41.1833431Z  echo "Docker root directory ($docker_root_dir) does not exist. Skipping disk space check." 2025-08-15T00:01:41.1834438Z  exit 0 2025-08-15T00:01:41.1834922Z fi 2025-08-15T00:01:41.1835768Z diskspace=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2025-08-15T00:01:41.1837523Z msg="Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified" 2025-08-15T00:01:41.1839020Z if [[ "$diskspace" -ge "$diskspace_cutoff" ]] ; then 2025-08-15T00:01:41.1839777Z  docker system prune -af 2025-08-15T00:01:41.1840941Z  diskspace_new=$(df -H --output=pcent ${docker_root_dir} | sed -n 2p | sed 's/%//' | sed 's/ //') 2025-08-15T00:01:41.1842082Z  if [[ "$diskspace_new" -gt "$diskspace_cutoff" ]] ; then 2025-08-15T00:01:41.1843292Z  echo "Error: Available diskspace is less than $diskspace_cutoff percent. Not enough diskspace." 2025-08-15T00:01:41.1844366Z  echo "$msg" 2025-08-15T00:01:41.1845489Z  exit 1 2025-08-15T00:01:41.1845989Z  else 2025-08-15T00:01:41.1846593Z  difference=$((diskspace - diskspace_new)) 2025-08-15T00:01:41.1847413Z  echo "Diskspace saved: $difference percent" 2025-08-15T00:01:41.1848088Z  fi 2025-08-15T00:01:41.1848549Z fi 2025-08-15T00:01:41.1900712Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-15T00:01:41.1901547Z env: 2025-08-15T00:01:41.1901970Z GIT_DEFAULT_BRANCH: main 2025-08-15T00:01:41.1902751Z RUNNER_ARTIFACT_DIR: /var/home/pytorchci/actions-runner/_work/_temp/artifacts 2025-08-15T00:01:41.1903870Z RUNNER_TEST_RESULTS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/test-results 2025-08-15T00:01:41.1904961Z RUNNER_DOCS_DIR: /var/home/pytorchci/actions-runner/_work/_temp/docs 2025-08-15T00:01:41.1906679Z GPU_FLAG: --device=/dev/mem --device=/dev/kfd --device /dev/dri --group-add video --group-add 110 --group-add daemon --group-add bin --cap-add=SYS_PTRACE --security-opt seccomp=unconfined --network=host 2025-08-15T00:01:41.1908443Z AWS_DEFAULT_REGION: us-east-1 2025-08-15T00:01:41.1909066Z AWS_REGION: us-east-1 2025-08-15T00:01:41.1933460Z AWS_ACCESS_KEY_ID: *** 2025-08-15T00:01:41.1934211Z AWS_SECRET_ACCESS_KEY: *** 2025-08-15T00:01:41.1943889Z AWS_SESSION_TOKEN: *** 2025-08-15T00:01:41.1944694Z CONTAINER_NAME: aa60442279b381246b5e5dca2faffc1fb627684cc5fbe89197f864d1a55625a8 2025-08-15T00:01:41.1945533Z DEVICE_NAME: rocm 2025-08-15T00:01:41.1946020Z DEVICE_TYPE: AMD Instinct MI250X/MI250 2025-08-15T00:01:41.1946585Z ##[endgroup] 2025-08-15T00:01:41.2030116Z + diskspace_cutoff=70 2025-08-15T00:01:41.2038649Z ++ docker info -f '{{.DockerRootDir}}' 2025-08-15T00:01:41.3025863Z + docker_root_dir=/media/4TB/docker-rootless 2025-08-15T00:01:41.3026748Z + '[' '!' -d /media/4TB/docker-rootless ']' 2025-08-15T00:01:41.3042603Z ++ df -H --output=pcent /media/4TB/docker-rootless 2025-08-15T00:01:41.3043843Z ++ sed -n 2p 2025-08-15T00:01:41.3048737Z ++ sed s/%// 2025-08-15T00:01:41.3050504Z ++ sed 's/ //' 2025-08-15T00:01:41.3099561Z + diskspace=35 2025-08-15T00:01:41.3100993Z + msg='Please file an issue on pytorch/pytorch reporting the faulty runner. Include a link to the runner logs so the runner can be identified' 2025-08-15T00:01:41.3102417Z + [[ 35 -ge 70 ]] 2025-08-15T00:01:41.3196707Z Post job cleanup. 2025-08-15T00:01:41.3294438Z Post job cleanup. 2025-08-15T00:01:41.4788872Z Post job cleanup. 2025-08-15T00:01:41.5442563Z Logging out of registry 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-15T00:01:41.6023483Z Post job cleanup. 2025-08-15T00:01:41.7453669Z Post job cleanup. 2025-08-15T00:01:41.7538350Z Post job cleanup. 2025-08-15T00:01:41.8816810Z [command]/usr/bin/git version 2025-08-15T00:01:41.8873180Z git version 2.34.1 2025-08-15T00:01:41.8907054Z Copying '/var/home/pytorchci/.gitconfig' to '/var/home/pytorchci/actions-runner/_work/_temp/4c421d02-a88e-4465-83bb-f37ec1a1d11c/.gitconfig' 2025-08-15T00:01:41.8919590Z Temporarily overriding HOME='/var/home/pytorchci/actions-runner/_work/_temp/4c421d02-a88e-4465-83bb-f37ec1a1d11c' before making global git config changes 2025-08-15T00:01:41.8921736Z Adding repository directory to the temporary git global config as a safe directory 2025-08-15T00:01:41.8923343Z [command]/usr/bin/git config --global --add safe.directory /var/home/pytorchci/actions-runner/_work/pytorch/pytorch 2025-08-15T00:01:41.9013727Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-15T00:01:41.9083849Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-15T00:01:41.9791381Z Entering 'android/libs/fbjni' 2025-08-15T00:01:41.9923253Z Entering 'third_party/FP16' 2025-08-15T00:01:42.0049709Z Entering 'third_party/FXdiv' 2025-08-15T00:01:42.0175297Z Entering 'third_party/NNPACK' 2025-08-15T00:01:42.0301540Z Entering 'third_party/NVTX' 2025-08-15T00:01:42.0430931Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-15T00:01:42.0558120Z Entering 'third_party/XNNPACK' 2025-08-15T00:01:42.0722495Z Entering 'third_party/aiter' 2025-08-15T00:01:42.0849437Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-15T00:01:42.0996652Z Entering 'third_party/benchmark' 2025-08-15T00:01:42.1124261Z Entering 'third_party/composable_kernel' 2025-08-15T00:01:42.1272052Z Entering 'third_party/cpp-httplib' 2025-08-15T00:01:42.1398616Z Entering 'third_party/cpuinfo' 2025-08-15T00:01:42.1523296Z Entering 'third_party/cudnn_frontend' 2025-08-15T00:01:42.1647170Z Entering 'third_party/cutlass' 2025-08-15T00:01:42.1792665Z Entering 'third_party/fbgemm' 2025-08-15T00:01:42.1916721Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-15T00:01:42.2031087Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-15T00:01:42.2161740Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-15T00:01:42.2278928Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-15T00:01:42.2417190Z Entering 'third_party/fbgemm/external/googletest' 2025-08-15T00:01:42.2530919Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-15T00:01:42.2645069Z Entering 'third_party/fbgemm/external/json' 2025-08-15T00:01:42.2771809Z Entering 'third_party/flash-attention' 2025-08-15T00:01:42.2893110Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-15T00:01:42.3020671Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-15T00:01:42.3159556Z Entering 'third_party/flatbuffers' 2025-08-15T00:01:42.3289802Z Entering 'third_party/fmt' 2025-08-15T00:01:42.3409609Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-15T00:01:42.3535439Z Entering 'third_party/gloo' 2025-08-15T00:01:42.3660358Z Entering 'third_party/googletest' 2025-08-15T00:01:42.3785081Z Entering 'third_party/ideep' 2025-08-15T00:01:42.3902661Z Entering 'third_party/ideep/mkl-dnn' 2025-08-15T00:01:42.4038073Z Entering 'third_party/ittapi' 2025-08-15T00:01:42.4161175Z Entering 'third_party/kineto' 2025-08-15T00:01:42.4280895Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-15T00:01:42.4389117Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-15T00:01:42.4509430Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-15T00:01:42.4628071Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-15T00:01:42.4742540Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-15T00:01:42.4857749Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-15T00:01:42.4978407Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-15T00:01:42.5096629Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-15T00:01:42.5215596Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-15T00:01:42.5335255Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-15T00:01:42.5455247Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-15T00:01:42.5571286Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-15T00:01:42.5695656Z Entering 'third_party/kleidiai' 2025-08-15T00:01:42.5823322Z Entering 'third_party/mimalloc' 2025-08-15T00:01:42.5947699Z Entering 'third_party/nlohmann' 2025-08-15T00:01:42.6079178Z Entering 'third_party/onnx' 2025-08-15T00:01:42.6241869Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-15T00:01:42.6374064Z Entering 'third_party/opentelemetry-cpp' 2025-08-15T00:01:42.6503999Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-15T00:01:42.6626982Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-15T00:01:42.6749155Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-15T00:01:42.6864469Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-15T00:01:42.6983168Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-15T00:01:42.7098642Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-15T00:01:42.7212009Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-15T00:01:42.7321767Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-15T00:01:42.7444442Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-15T00:01:42.7567375Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-15T00:01:42.7730561Z Entering 'third_party/pocketfft' 2025-08-15T00:01:42.7854237Z Entering 'third_party/protobuf' 2025-08-15T00:01:42.7982126Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-15T00:01:42.8097779Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-15T00:01:42.8222510Z Entering 'third_party/psimd' 2025-08-15T00:01:42.8350207Z Entering 'third_party/pthreadpool' 2025-08-15T00:01:42.8474295Z Entering 'third_party/pybind11' 2025-08-15T00:01:42.8601231Z Entering 'third_party/python-peachpy' 2025-08-15T00:01:42.8729741Z Entering 'third_party/sleef' 2025-08-15T00:01:42.8852357Z Entering 'third_party/tensorpipe' 2025-08-15T00:01:42.8980744Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-15T00:01:42.9099227Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-15T00:01:42.9214845Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-15T00:01:42.9333615Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-15T00:01:42.9442566Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-15T00:01:42.9613468Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-15T00:01:42.9673501Z http.https://github.com/.extraheader 2025-08-15T00:01:42.9697354Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-08-15T00:01:42.9784859Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-15T00:01:43.0476669Z Entering 'android/libs/fbjni' 2025-08-15T00:01:43.0547076Z http.https://github.com/.extraheader 2025-08-15T00:01:43.0649503Z Entering 'third_party/FP16' 2025-08-15T00:01:43.0717271Z http.https://github.com/.extraheader 2025-08-15T00:01:43.0819629Z Entering 'third_party/FXdiv' 2025-08-15T00:01:43.0887560Z http.https://github.com/.extraheader 2025-08-15T00:01:43.0986432Z Entering 'third_party/NNPACK' 2025-08-15T00:01:43.1057863Z http.https://github.com/.extraheader 2025-08-15T00:01:43.1155900Z Entering 'third_party/NVTX' 2025-08-15T00:01:43.1222977Z http.https://github.com/.extraheader 2025-08-15T00:01:43.1321047Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-15T00:01:43.1386709Z http.https://github.com/.extraheader 2025-08-15T00:01:43.1482237Z Entering 'third_party/XNNPACK' 2025-08-15T00:01:43.1548299Z http.https://github.com/.extraheader 2025-08-15T00:01:43.1678855Z Entering 'third_party/aiter' 2025-08-15T00:01:43.1743500Z http.https://github.com/.extraheader 2025-08-15T00:01:43.1837012Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-15T00:01:43.1897977Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2013532Z Entering 'third_party/benchmark' 2025-08-15T00:01:43.2079448Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2172353Z Entering 'third_party/composable_kernel' 2025-08-15T00:01:43.2237555Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2351417Z Entering 'third_party/cpp-httplib' 2025-08-15T00:01:43.2417741Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2510777Z Entering 'third_party/cpuinfo' 2025-08-15T00:01:43.2574339Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2669308Z Entering 'third_party/cudnn_frontend' 2025-08-15T00:01:43.2736001Z http.https://github.com/.extraheader 2025-08-15T00:01:43.2831038Z Entering 'third_party/cutlass' 2025-08-15T00:01:43.2896869Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3018972Z Entering 'third_party/fbgemm' 2025-08-15T00:01:43.3088347Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3187435Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-15T00:01:43.3249736Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3341964Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-15T00:01:43.3403503Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3509787Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-15T00:01:43.3571816Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3664156Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-15T00:01:43.3727005Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3848337Z Entering 'third_party/fbgemm/external/googletest' 2025-08-15T00:01:43.3906773Z http.https://github.com/.extraheader 2025-08-15T00:01:43.3997070Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-15T00:01:43.4058600Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4147032Z Entering 'third_party/fbgemm/external/json' 2025-08-15T00:01:43.4206729Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4308942Z Entering 'third_party/flash-attention' 2025-08-15T00:01:43.4372451Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4464784Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-15T00:01:43.4528420Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4631279Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-15T00:01:43.4693735Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4806958Z Entering 'third_party/flatbuffers' 2025-08-15T00:01:43.4876022Z http.https://github.com/.extraheader 2025-08-15T00:01:43.4984302Z Entering 'third_party/fmt' 2025-08-15T00:01:43.5048231Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5142683Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-15T00:01:43.5207871Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5304721Z Entering 'third_party/gloo' 2025-08-15T00:01:43.5369530Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5464008Z Entering 'third_party/googletest' 2025-08-15T00:01:43.5530034Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5624007Z Entering 'third_party/ideep' 2025-08-15T00:01:43.5690543Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5778449Z Entering 'third_party/ideep/mkl-dnn' 2025-08-15T00:01:43.5840560Z http.https://github.com/.extraheader 2025-08-15T00:01:43.5955287Z Entering 'third_party/ittapi' 2025-08-15T00:01:43.6022034Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6116256Z Entering 'third_party/kineto' 2025-08-15T00:01:43.6182383Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6276755Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-15T00:01:43.6338906Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6432028Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-15T00:01:43.6497110Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6595358Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-15T00:01:43.6656494Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6746739Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-15T00:01:43.6808528Z http.https://github.com/.extraheader 2025-08-15T00:01:43.6902563Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-15T00:01:43.6966926Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7054788Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-15T00:01:43.7115519Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7216586Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-15T00:01:43.7276234Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7369085Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-15T00:01:43.7428317Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7518414Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-15T00:01:43.7580048Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7676553Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-15T00:01:43.7740765Z http.https://github.com/.extraheader 2025-08-15T00:01:43.7846142Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-15T00:01:43.7908480Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8001115Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-15T00:01:43.8062352Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8163657Z Entering 'third_party/kleidiai' 2025-08-15T00:01:43.8228777Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8325421Z Entering 'third_party/mimalloc' 2025-08-15T00:01:43.8388736Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8486407Z Entering 'third_party/nlohmann' 2025-08-15T00:01:43.8550255Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8650691Z Entering 'third_party/onnx' 2025-08-15T00:01:43.8716009Z http.https://github.com/.extraheader 2025-08-15T00:01:43.8848813Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-15T00:01:43.8918116Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9025940Z Entering 'third_party/opentelemetry-cpp' 2025-08-15T00:01:43.9093533Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9188604Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-15T00:01:43.9252912Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9347801Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-15T00:01:43.9411198Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9505466Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-15T00:01:43.9567949Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9662604Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-15T00:01:43.9724357Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9820032Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-15T00:01:43.9884148Z http.https://github.com/.extraheader 2025-08-15T00:01:43.9977559Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-15T00:01:44.0038968Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0132034Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-15T00:01:44.0193010Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0284167Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-15T00:01:44.0347376Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0445345Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-15T00:01:44.0507424Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0609377Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-15T00:01:44.0671342Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0812485Z Entering 'third_party/pocketfft' 2025-08-15T00:01:44.0879700Z http.https://github.com/.extraheader 2025-08-15T00:01:44.0974368Z Entering 'third_party/protobuf' 2025-08-15T00:01:44.1039525Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1138187Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-15T00:01:44.1200411Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1291371Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-15T00:01:44.1355565Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1454840Z Entering 'third_party/psimd' 2025-08-15T00:01:44.1520975Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1614641Z Entering 'third_party/pthreadpool' 2025-08-15T00:01:44.1679696Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1772968Z Entering 'third_party/pybind11' 2025-08-15T00:01:44.1838617Z http.https://github.com/.extraheader 2025-08-15T00:01:44.1932294Z Entering 'third_party/python-peachpy' 2025-08-15T00:01:44.2001059Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2098443Z Entering 'third_party/sleef' 2025-08-15T00:01:44.2164180Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2257744Z Entering 'third_party/tensorpipe' 2025-08-15T00:01:44.2324594Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2414253Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-15T00:01:44.2477239Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2570655Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-15T00:01:44.2633401Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2722738Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-15T00:01:44.2786331Z http.https://github.com/.extraheader 2025-08-15T00:01:44.2879793Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-15T00:01:44.2942808Z http.https://github.com/.extraheader 2025-08-15T00:01:44.3031348Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-15T00:01:44.3095514Z http.https://github.com/.extraheader 2025-08-15T00:01:44.3530977Z Cleaning up orphan processes